Building Agents: Designing for Evaluability

Why this matters

Thursday morning, a customer emails Pip: "Your chat assistant promised me a full refund yesterday. Where is it?" Pip forwards it to Dev with one line: "Did it?" Dev opens the database. Orders, yes. Refund requests, none. Chat logs? The app printed replies to the console, and the console was cleared at the last deploy. Dev cannot see what the customer asked, what the model saw, which tools ran, or what Sprout said. Dev cannot prove Sprout promised nothing, cannot reproduce the conversation, and cannot fix a failure nobody can look at. This lesson makes sure that the next time someone asks "did it?", the answer takes thirty seconds and starts with opening a trace.

1. If you did not record it, it did not happen

Instrumentation is the code that records what your agent does: every model call, every tool call, every decision, in a structured form you can query later. Observability is the property you get from it: the ability to answer "what happened, and why?" from the records alone, without rerunning anything.

The rule of this lesson is short. Instrument before traffic. The first real user must produce the first complete trace, because the first real user is often the first surprise.

The sampling rate is the fraction of conversations you record in full. Before launch and during early traffic, it is 100%. Forty conversations a day cost nothing to store, and every one is a lesson. At 20,000 a day you may sample a share and keep every trace with an error or a permission denial. Not yet.

2. The trace data model

A trace is the full record of one conversation. A span is one unit of work inside it. Spans nest: a turn (one customer message and everything it caused) contains model calls, and a model call contains the tool calls it requested. A span inside another span is a nested span, and it carries the parent_id of the outer one.

Every span has the same eight fields.

The input of each model call above is a count ("messages": 2) to keep the example short. In a real trace, store the complete messages array. Section 5 explains why.

3. What to record

Three kinds of spans carry the important facts.

Model calls: model id, prompt hash, parameters (temperature, max tokens), token counts in and out, latency, and the full input and output. Tokens and latency feed the cost work in L9. The model id and prompt hash say which version of Sprout produced this behavior.

Tool calls: tool name, arguments, result, and error, if any. Store the whole result, not a summary. When the model invents a date, you need to see that the tool never returned one.

Permission decisions: every time canCall runs, record allowed or denied, the role, the tool, and the reason. A denial is not an error. It is the harness doing its job, and you will want to count them.

On the trace itself, record the session id (which conversation), the user id (who), the environment (dev, staging, prod), and the agent version.

Prompt hashes

A prompt hash is a short fingerprint of the exact system prompt text, computed with a hash function such as SHA-256 and truncated: sha256(system_prompt)[:8]. Change one character in the prompt and the hash changes.

Why a hash instead of a version number? Nobody forgets to compute a hash, and version numbers get skipped. The hash comes from the text itself, so it is always right.

4. Langfuse and ClickHouse

You could write traces to a file. Reading them is the problem. Langfuse is an open-source tool that stores traces, shows each one in a web UI as a nested tree, and exposes an API for querying and scoring them.

Under Langfuse sit two databases. Postgres holds metadata: projects, users, API keys, prompt versions. ClickHouse holds the spans. It is a column store: each column of a table is kept together on disk, so "average latency of tool_call:lookup_order spans this month" reads one column across millions of rows instead of every row in full. That is what makes analytics over a large trace set fast.

Here is a self-hosted setup as a sketch. The real compose file in the Langfuse docs also runs a worker, Redis, and object storage, and it will have changed by the time you read this. Copy the shape, not the values.

# Sketch only. Use the current compose file from the Langfuse docs for real values.
services:
  langfuse:
    image: langfuse/langfuse:3
    ports: ["3000:3000"]
    environment:
      DATABASE_URL: postgresql://pg:pg@postgres:5432/langfuse
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_USER: ch
      CLICKHOUSE_PASSWORD: ch
      NEXTAUTH_SECRET: change-me
      SALT: change-me
    depends_on: [postgres, clickhouse]
  postgres:
    image: postgres:16
    environment: { POSTGRES_USER: pg, POSTGRES_PASSWORD: pg, POSTGRES_DB: langfuse }
  clickhouse:
    image: clickhouse/clickhouse-server:24
    environment: { CLICKHOUSE_USER: ch, CLICKHOUSE_PASSWORD: ch }

Your agent needs three environment variables: a public key, a secret key, and the base URL of your Langfuse instance. Create the keys in the Langfuse UI after it starts.

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=http://localhost:3000

Here is the shape of an SDK call that creates a trace with a nested span. It is a sketch: method names differ between SDK versions.

// Sketch only. Check the Langfuse SDK docs for the exact method names in your version.
import { Langfuse } from "langfuse";
const langfuse = new Langfuse(); // reads the three env vars above

const trace = langfuse.trace({ name: "turn", sessionId: "sess_alex_01", userId: "cust_alex",
  metadata: { env: "dev", agent_version: "0.3.1", prompt_hash: "a1b2c3d4" } });
const gen = trace.generation({ name: "model_call", model: "claude-opus-5", input: messages });
gen.end({ output: response.content, usage: { input: 812, output: 41 } });
const tool = gen.span({ name: "tool_call:lookup_order", input: { order_id: "1042" } });
tool.end({ output: { status: "shipped" }, metadata: { tier: "T0", permission: "allowed" } });
await langfuse.flushAsync(); // send before the process exits

5. Designing for evaluability

Evaluability means a trace contains enough to judge whether the agent behaved correctly, and to run the same step again. Four habits get you there.

Store complete inputs so spans can be replayed. If a model-call span holds the full messages array, the exact tools list, and the parameters, you can send that request again and compare answers. If it holds "messages: 4," you cannot.

Use stable ids. The same customer, order, and session carry the same ids across traces, so "show me every conversation about order #1042" returns all of them.

Tag synthetic traces with their scenario id. A scenario is a scripted test conversation, and L3 builds hundreds of them. When Sprout runs a scenario, put scenario_id in the trace metadata, so a failing test leads straight to the trace that shows why.

Version everything. Agent version, prompt hash, tool schema version, and the version of the fictional world the scenarios run against. When behavior changes, one of these changed. If they are all recorded, you can find which.

6. Reading a trace

Reading a trace means walking the spans in order and finding the first failure: the earliest span where the agent's behavior stopped matching the spec. Not the span where the customer noticed. The first one.

Three questions, in order, every time you read a trace:

1. What did the customer ask, and what did they get? That is the turn span.
2. Walk each span. Does its output follow from its input and the spec?
3. Stop at the first span where the answer is no. Write one sentence about it.

That one sentence is the raw material of L4, where you collect many of them and turn them into named failure modes.

7. Checklist: instrumentation review before traffic

Run through this list before the first real user, and again before every launch.

Summary

• Instrument before the first real user and record 100% of conversations until volume forces sampling.
• A trace is one conversation; a span is one unit of work; spans nest through parent_id and share eight fields.
• Record model calls (model, prompt hash, params, tokens, latency), tool calls (name, args, result, error), and permission decisions (allowed or denied, with a reason).
• Langfuse is the UI and API, ClickHouse stores spans for fast analytics, and Postgres holds metadata.
• A trace is evaluable when you can replay any span, group by stable ids and scenario ids, and name every version. Read traces top down and mark the first failure.