Error Analysis: Measuring with Evaluators
Turn each failure mode into a binary evaluator, prove it works with TPR and TNR on labeled traces, and report prevalence with a confidence interval you can defend.
Before this: L4 Finding failures
- Build one binary evaluator per failure mode.
- Choose code checks for objective failures and LLM judges only when interpretation is needed.
- Write a judge prompt with a definition, boundary examples, and a strict output format.
- Split labeled data into train, dev, and test, report TPR and TNR, and read the test set once.
- Estimate prevalence with a bootstrap confidence interval, correct it for judge error, and apply the same discipline to retrieval, grounding, and handoff.
Why this matters
The failure report from L4 says "Refund outside policy: 7 of 60 traces." Pip reads it and asks the questions an owner asks. "So 12%? Of every conversation? How sure are you?" Dev hesitates. Maya read 60 traces; 440 were never read. Sixty is small, and a different 60 might have shown 4 or 11. Pip has one more question: "When the shop opens, can it keep watching for this without Maya reading every chat?" Dev needs three things: a detector for each failure mode, proof that the detector works, and a number with an honest range around it.
1. One binary evaluator per failure mode
An eval is a measurement of agent behavior against a written definition of correct. An evaluator does the measuring: a function that takes one trace and answers one question about it. A binary evaluator answers yes or no, and every failure mode in the L4 taxonomy gets one.
Why not a 1 to 5 score? Three reasons. Agreement: two reviewers give a 3 and a 4 and both feel right, while yes or no disagrees only at the boundary the taxonomy already wrote down. Actionability: "tone averages 3.4" points nowhere, while "Wrong tone in 8 of 60" points at eight traces to fix. Validation: you can count how often a yes-or-no grader agrees with a human; a scale has no clean way to do that.
Each mode gets one evaluator, and the first decision is which kind. Code checks: refund amount exceeds order total (two numbers from the trace), missed escalation on legal words (a word list and a tool-call list), unknown order ids in the reply (compare with orders.yaml). Judges: Refund outside policy ("you'll see the money by Friday" is a promise in many wordings), Wrong tone, and claims not supported by tool output.
2. Code checks first
A code check is a deterministic function over the trace: the user messages, the tool calls with their arguments and results, and the reply. It runs in milliseconds, costs nothing, and gives the same answer every time. When a failure can be stated in terms of trace fields, write code.
// checks.ts: each returns true when the failure is present
export function refundExceedsTotal(t: Trace): boolean {
const refund = t.calls.find((c) => c.name === "issue_refund");
const order = t.calls.find((c) => c.name === "lookup_order");
if (!refund || !order) return false;
return refund.input.amount > order.output.total;
}
const LEGAL = /\b(lawyer|attorney|sue|legal action|small claims)\b/i;
export function missedLegalEscalation(t: Trace): boolean {
const legalWords = t.userMessages.some((m) => LEGAL.test(m));
const escalated = t.calls.some((c) => c.name === "escalate_to_human");
return legalWords && !escalated;
}
A third check, "every order id in the reply exists in the world", pulls the four-digit numbers out of the reply and looks each up in orders.yaml. On trace tr-2c4, where the customer writes "my attorney will be in touch about the dead fern" and Sprout replies with watering tips, missedLegalEscalation returns true and the other two return false.
3. LLM judges when interpretation is needed
An LLM judge is a model given one trace and one failure definition, asked to answer fail or pass with a reason. Use it when the failure is about meaning: wrong tone, a claim the tool output does not support, a refund promised in words a regex would never catch.
A judge prompt has five parts, in this order: a role (strict reviewer, checks one thing), the failure definition (the taxonomy entry with its boundary), boundary examples (at least one FAIL and one PASS from labeled traces), the trace, and an output format ({ "fail": true or false, "reason": "..." } and nothing else).
The examples are the part people skip, and they do the work. They come from your labeled traces (next section) and sit on the boundary: the PASS example is a firm reply that a lazy judge would flag. Running the judge is one client.messages.create call with the filled-in prompt as the user message, then JSON.parse on the text that comes back.
For Refund outside policy, the same skeleton needs two more things: where to find the dates (delivered_on from the lookup_order result, today's date from the trace metadata) and what the policy says (30 days, damaged on arrival). You will write that prompt in the exercise.
4. Validating the judge
A judge is a guess until you check it against humans. Labeled data is a set of traces where a person wrote the answer for this mode: fail or pass. Maya's L4 sheet is the start; for each mode, every trace gets a yes or no, not only its first-failure note.
Split the labeled traces three ways. Train: pick the boundary examples for the prompt from here. Dev: run the judge, measure, change the prompt, repeat. Test: the final score, frozen, read once. The split keeps the judge from being scored on traces it was shown. With 60 labeled traces, 20/20/20 works.
Two numbers tell you whether the judge works.
- True positive rate (TPR): of the traces that really fail, the fraction the judge flags. How much does it catch?
- True negative rate (TNR): of the traces that really pass, the fraction the judge passes. How often does it leave good traces alone?
Why accuracy alone misleads
Accuracy is the share of all traces the judge got right. It sounds natural, and it lies when failures are rare.
Tuning on dev is a loop: run, read the misses and false alarms, change the prompt, run again. Dev's Wrong tone judge went v1 (definition only): TPR 0.60, TNR 0.90; v2 (added "does not count: brief or formal replies" and one FAIL example): 0.80, 0.90; v3 (a second FAIL example): 0.90, 0.85. Which wins depends on which mistake costs more: for a CI gate, a false alarm wastes an afternoon, so TNR matters more. Rule of thumb: do not trust a judge under 0.80 on either number.
5. Freeze and read test once
The test split is the exam; the dev split is the practice exam. Every time you look at a test result and then change the prompt, you have turned the test set into more dev data, and its score stops meaning anything. "Freeze and read test once" means: choose the winner on dev, run it on test once, report that number, stop.
6. Prevalence with bootstrap confidence intervals
Prevalence is the share of traces that contain a failure. Maya found Refund outside policy in 7 of 60: prevalence 11.7%. That single number is a point estimate, and at n = 60 it wobbles; a different random 60 could have shown 4 or 11. Pip deserves a range.
A confidence interval says how far the estimate could move with a different sample of the same size. The bootstrap computes one without formulas: resample your labels with replacement, many times, and see how the estimate varies.
Correcting for judge error
Once the judge is validated, you run it on all 500 traces, and the flag rate is your observed prevalence. But the judge misses some failures (TPR below 1) and raises false alarms (TNR below 1), so that rate is biased. Corrected prevalence undoes the bias:
p_true = (p_observed − (1 − TNR)) / (TPR − (1 − TNR))
For a range on the corrected number, bootstrap the judge's flags and correct each resample. Then report one line: "Refund outside policy: about 9% of conversations (95% CI 4% to 14%), judge TPR 0.80 and TNR 0.90 on a frozen test set."
7. Subsystems
Sprout is not one thing. Retrieval, grounding, and handoff are subsystems, and each gets the same treatment: a binary question, human labels, a validated evaluator.
| Subsystem | Binary question | Evaluator |
|---|---|---|
| Retrieval | Did the right care guide come back in the top k? | Code check, reported as recall@k, labels from L3 ground truth |
| Grounding | Is every claim in the reply supported by a tool result? | LLM judge, labels from Maya |
| Handoff | Did the escalation summary carry what Maya needs? | Code check for the order id, judge for completeness |
Retrieval is what search_care_guide does: it returns the top k articles for a query. Recall@k is the share of scenarios where the correct article appears among those k.
Grounding asks whether the reply is supported by what the tools returned. "Delivered September 5" with delivered_on: 2026-09-05 in the result is grounded; "arrives Thursday" after not_found is not. "Supported" needs reading, so it is a judge.
Handoff asks whether escalate_to_human carried the context. Compare escalate_to_human("customer upset") with escalate_to_human("Order 1058, delivered Aug 1, 42 days; wants refund for dead plant; window closed; angry"). Maya can act on the second in ten seconds. A code check confirms an order id; a judge checks for the request and policy status.
8. Homework 1
Homework 1 is where this module becomes yours. You will label at least 60 support traces, build a taxonomy of 5 to 8 binary failure modes with definitions, examples, and boundaries, and write a one-page failure report with a bootstrap interval on your top mode. The full assignment and rubric are in the homework box below the quiz.
Summary
- Every failure mode gets one binary evaluator; yes or no beats 1 to 5 on agreement, actionability, and validation.
- Code checks for failures stated in trace fields; LLM judges only when meaning must be interpreted, built from a role, a definition, boundary examples, the trace, and strict JSON output.
- Validate every judge on labeled traces split into train, dev, and test; report TPR and TNR, never accuracy alone, and read the test set once.
- Prevalence is a share of traces; report it with a bootstrap confidence interval, corrected for the judge's TPR and TNR.
- Retrieval (recall@k), grounding, and handoff get the same discipline: binary question, human labels, validated evaluator.
Key terms in this lesson
Check your understanding
Pick an answer to see why it is right or wrong. These are graded in code, instantly.
Apply it
Write your answer in your own words. Eve grades it against a rubric and gives you feedback, the same way an LLM judge works in Lesson 5.
When does a failure mode get a code check instead of an LLM judge? Give one Sprout example of each kind.
Write a judge prompt for the failure mode Refund promised outside policy. Include a role, a precise binary definition, at least one PASS and one FAIL boundary example, what the judge is given, and a strict output format.
Homework 1 - Label traces and build a taxonomy
Label at least 60 support traces from your Sprout scenarios (or a comparable agent of your own) and turn them into a measurable taxonomy.
- Open-code every trace: read the whole thing and write the first failure in plain language.
- Group the notes into 5 to 8 binary failure modes. For each mode write a one-sentence definition, one real example from your traces, and a boundary ("counts / does not count").
- For each mode, say whether it will be measured by a code check or an LLM judge, and why.
- Report the prevalence of each mode with a bootstrap 95% confidence interval.
- For at least one judge-measured mode, write the judge prompt and report TPR and TNR on a dev split of your labels.
Paste your write-up below. Tables and short YAML or JSON snippets are welcome.
- A labeling sheet summary: number of traces, who labeled them, how long it took
- 5 to 8 binary failure modes, each with a definition, an example, and a boundary
- A code-check-or-judge decision for every mode, with a reason
- Prevalence with a bootstrap confidence interval for every mode
- One judge prompt with its TPR and TNR on a dev split
Paste your homework write-up below. Include every deliverable.
Instructor notes(for whoever is teaching this lesson)
Talking points
- Open with Pip's question, "how often?", and let the room feel how weak "we saw it a few times" sounds. The whole lesson is about producing a number you can defend.
- Binary beats scores. Ask two students to rate the same reply 1 to 5 and watch them disagree by one point. Then ask "did it promise a refund outside policy?" and watch them agree instantly.
- Code checks first, judges second. Every mode gets the question: can a few lines of code decide this from the trace? If yes, no judge. Judges cost money and need validation; code needs neither.
- TPR and TNR are two different mistakes. Say it plainly: TPR is "how much it catches", TNR is "how often it leaves good traces alone". Accuracy blends them and hides the one that matters.
- The test set is an exam. Once you have seen it and reacted, it is a practice exam.
- The corrected-prevalence formula looks scary and is two subtractions and a division. Do it on the board with the 500-trace numbers and then verify it backwards (34 true positives plus 46 false alarms equals 80 flags). The backward check is what makes it click.
Live demo idea
Bring 20 short traces on slides with Maya's labels hidden. Have the class label them for one mode. Reveal a judge's verdicts and build the confusion matrix live on a whiteboard, one tally at a time. Compute TPR and TNR from the tallies, then compute accuracy and show how it flatters a judge that mostly says pass. If there is time, bootstrap by hand: ten index cards, draw with replacement five times, and watch the prevalence swing from 0% to 50%.
Common misconceptions
- "High accuracy means a good judge." Only when failures are common. Show the always-pass judge.
- "The judge can grade itself." A judge is a guess until humans checked it on labeled traces.
- "A 1 to 5 score is more informative." It is less consistent and does not name a failure to fix.
- "Looking at test results for information is harmless." If it changed what you did next, it was tuning.
- "More traces fix everything." Sixty traces with seven failures still give a wide interval. Report the interval.
Timing
Sections 1 to 3: 15 minutes. Section 4 with the live confusion matrix: 20 minutes. Section 5: 5 minutes. Section 6 with the bootstrap and the correction: 15 minutes. Section 7 and the homework briefing: 10 minutes. Total 65 minutes.
If you only have 20 minutes
Do sections 1, 2, and 4 only: one binary evaluator per mode, code before judges, and TPR / TNR with the confusion matrix on the board. Assign sections 5 to 7 as reading and point students at Eve for the bootstrap.