Error Analysis: Finding Failures

Why this matters

The smoke report from L3 showed 11 fails out of 50. Dev wants the full picture fast, so Dev sends all 500 scenario traces to a model with one instruction: "Rate this conversation 1 to 10." The model comes back cheerful: 96% scored 8 or higher. Dev forwards the number to Pip. Maya, who approves every refund at the shop, asks for ten trace links and reads them over coffee. Four of the ten contain Sprout promising a refund on an order more than 30 days old. One contains an invented delivery date. The grader gave all five a 9. It was not broken. It was uninformed, because nobody had told it what "bad" looks like at Pip's Plant Shop. This lesson is about finding that out, and the only way to find it out is to read.

1. Humans first

Error analysis is the practice of reading traces to discover and name the ways an agent fails. It is the Analyze verb at its most literal. You look, you write down what you see, and only then do you build anything.

The reason you cannot skip to automation is blunt: you cannot automate the detection of a failure you have not named. A grader needs a definition. The definition comes from reading. Ask a model "is this good?" and it answers from its general sense of a pleasant conversation. Ask it "did Sprout promise a refund on an order delivered more than 30 days ago?" and it can check. The second question exists only because Maya read ten traces.

One person owns the taxonomy

A taxonomy is a named list of failure modes with a rule for what belongs in each. Taxonomies written by committee come out vague, because every disagreement gets settled by making the category wider. So one person owns it: the benevolent dictator. Everyone can read traces and write notes. One person merges the notes, names the modes, and decides the boundaries.

The owner should be whoever knows best what "correct" means for this agent and has time to read. At Pip's that is Maya, who knows the refund policy in her sleep. Dev builds the tooling and reads alongside her. If you are the only person on your project, the owner is you.

2. The review loop and interface

The review loop is a cycle: sample some traces, read them, note what went wrong, group the notes, then sample more. You go around it until new traces stop producing new notes.

The interface can be a spreadsheet. Three columns are enough to start.

Keep it fast. Two to three minutes per trace. A link that opens the trace in one click. No dropdowns yet, because dropdowns mean categories, and categories come later.

Sampling matters more than it looks. Pull traces at random across tags, using the L3 seed so the sample is reproducible, and include traces the smoke report marked as passes. The smoke check was mechanical; a trace can call the right tools and still be wrong.

// review-sample.ts (sketch; the Langfuse call is illustrative)
const traces = await langfuse.fetchTraces({ tags: ["smoke-2026-09-24"], limit: 500 });
const sample = shuffle(traces, seedrandom("review-1")).slice(0, 20);
writeCsv("review.csv", sample.map((t) => ({
  trace_id: t.id,
  link: `${LANGFUSE_URL}/trace/${t.id}`,
  first_failure_note: "",
  severity: "",
})));

3. Open coding

Open coding is reading each trace whole and writing what went wrong, in your own words, before any categories exist. The name comes from qualitative research, where "open" means you are not yet sorting, only noticing.

Four rules.

1. Read the whole trace. Tool calls, tool results, and the final reply. Most failures are visible only when you compare the reply with what the tools returned.
2. Note the first failure only. After the first mistake, everything downstream is contaminated. Later errors are usually consequences.
3. Plain language. Write what happened, not a label. "Tool said not_found, Sprout invented Thursday" beats "hallucination".
4. No categories yet. If you start sorting on trace three, every later trace gets squeezed into the bins you already have.

Notice what the notes are not. They are not labels. They are not scores. Each one points at evidence in the trace, and a second reader could check it in a minute.

4. Axial coding

Axial coding is grouping the open-coding notes into named failure modes. A failure mode is a specific, binary kind of mistake: a trace either has it or does not. The process is mechanical. Print the notes. Put similar ones next to each other. Name each pile with a short verb phrase. Then write three things for each pile: a definition, an example trace, and a boundary that says what counts and what does not.

Binary is the important word. "Wrong tone: 3 out of 5" cannot be acted on, and two reviewers will not agree on it. "Wrong tone: yes, because the reply mocked a frustrated customer" can be checked, counted, and later automated.

Two things happen to these modes later. In L5, each gets an evaluator, and every trace gets labeled for every mode, not only its first failure. The first-failure rule is for discovery; measurement labels everything.

5. Saturation

Saturation is the point where reading more traces stops producing new failure modes. It is how you know when to stop the loop, for now.

Track it with a running count. After each batch of 20, write down how many modes exist.

Saturated does not mean finished. It means this sample, from this scenario set, has been mined. Real traffic will bring new modes, and the weekly review in L6 sends them back here. Two practical notes: when a mode is rare, sample by tag as well as at random so it shows up sooner, and when a batch adds one new mode, read one more batch before stopping.

6. Published taxonomies last

There are published lists of agent failure types: hallucination, tool misuse, instruction drift, and in L7 the OWASP categories for agentic applications. They are useful. Read them after you build your own.

The reason is anchoring. If you read a list first, you see its categories in every trace and miss what is specific to Sprout. "Refund outside policy" appears on no generic list, and it is the one that costs Pip money. Once your taxonomy exists, use the published list as a gap check: for each item, ask whether your traces contain it and whether your scenarios could even produce it.

7. The failure report

The failure report is the deliverable of this lesson. It has counts per mode, one example trace id per mode, a severity, and a priority order. It feeds L5, where each mode gets an evaluator, and L6, where the worst ones become regression tests.

The report is one page, and it changes. When L5 gives you a validated evaluator per mode, the counts become prevalence estimates with ranges. When L6 runs the suite on every pull request, the report becomes a dashboard. But it starts as a table that one person wrote after reading 60 traces.

Summary

• A grader can only find failures someone has named, and naming requires humans reading whole traces.
• The review loop is sample, read, note, group, sample more, run from a three-column spreadsheet by one owner.
• Open coding writes the first failure of each trace in plain language with no categories; axial coding groups those notes into binary failure modes with a definition, an example, and a boundary.
• Saturation is reached when a new batch of traces adds no new modes; rare modes arrive late, so read at least three batches and sample by tag.
• Compare with published taxonomies only after building your own, then ship a one-page failure report with counts, examples, severity, and priority.