Bonus: Evals Interview Prep

Why this matters

Dev is interviewing for an AI engineering role. The interviewer asks, "How would you evaluate a support agent?" Dev says, "We'd use an LLM judge." The interviewer waits. Dev adds, "and check the accuracy." Silence.

Six months of work on Sprout, and none of it made it into the answer: Maya's taxonomy, the judge validated on 60 labeled traces, the CI gate that caught the refund regression, the $3,000 bill that became $530. Interviews reward people who have run the loop and can say so with numbers. This lesson turns L1 to L9 into answers that show it.

1. The traps

Nine answers that sound reasonable and fail.

Trap 1: an LLM judge without labeled data

• What the candidate says: "We'd have a model grade the outputs."
• Why it fails: Nobody validated the judge, so its number is an opinion.
• The better answer: "A person labels 60 to 100 traces first. Then we write the judge on the training split, tune on dev, and report TPR and TNR on a frozen test split."

Trap 2: accuracy without TPR and TNR

• What the candidate says: "The judge is 95% accurate."
• Why it fails: If 5% of traces fail, a judge that never fires is 95% accurate and useless.
• The better answer: "TPR 0.8: it catches 8 of 10 real failures. TNR 0.9: it passes 9 of 10 real passes. Here is the corrected prevalence."

Trap 3: 1-to-5 rubrics instead of binary

• What the candidate says: "We'd score each response from 1 to 5 on helpfulness."
• Why it fails: Reviewers rarely agree on a 3 versus a 4, and an average of 3.7 says nothing about what to fix.
• The better answer: "One binary evaluator per failure mode, with a definition and boundary examples. Refund outside policy: yes or no."

Trap 4: never looking at the data

• What the candidate says: "We'd automate the evaluation pipeline end to end."
• Why it fails: You cannot automate detection of a failure you have not named.
• The better answer: "Read 20 traces first, note the first failure in each, and group the notes into modes until new traces add none."

Trap 5: tuning on the test set

• What the candidate says: "We iterated on the prompt until the test score hit 95%."
• Why it fails: A test set you tuned against measures your tuning, not the agent.
• The better answer: "Train, dev, test. All tuning on dev. The test slice is read once, at decision time."

Trap 6: confusing pass@k with pass^k

• What the candidate says: "It passes 99% of the time if we give it three tries."
• Why it fails: That is pass@3, "can it ever do it." A customer gets one try. Reliability is pass^k, all k runs passing.
• The better answer: "I report pass^k from reset-and-replay and gate CI on it."

Trap 7: answering security with "an injection filter"

• What the candidate says: "We'd add a prompt injection detector."
• Why it fails: Injection is text, and paraphrases evade filters. The tenth attacker gets a working refund tool.
• The better answer: "Design as if the model can be tricked: authorization in code, least privilege, T2 behind human approval, adversarial tests that fail closed. A filter is a speed bump, not the defense."

Trap 8: ignoring cost

• What the candidate says: "We'd use the most capable model for everything."
• Why it fails: An agent that costs more than the business it serves gets switched off.
• The better answer: "Profile tokens from traces, cache the stable prefix, calibrate a cascade on labeled data, and show the Pareto frontier before picking a model."

Trap 9: automating before a manual loop

• What the candidate says: "We'd run an optimizer over the prompt."
• Why it fails: An optimizer rewarded by a judge finds the cheapest way to please the judge, often a magic sentence.
• The better answer: "Three logged manual iterations first. Then a bounded loop on the dev slice, human review of every prompt diff, and a held-out test slice to catch reward hacking."

2. A framework for answering

Every evaluation question can be answered with the three verbs of this course.

The order matters. Analyze before Measure says you look before you count. Measure before Improve says you count before you change.

3. Five sample questions with strong answer outlines

Q1. "Our LLM judge says 96% of conversations are good. Do you trust it?"

1. Ask what it was validated against. No labeled data, no trust.
2. Explain how 96% can be wrong: a judge that knows no failure modes passes everything.
3. The fix: label 60 traces, name the modes, one judge per mode with TPR and TNR on a frozen test split. Corrected prevalence with an interval replaces the 96%.

Q2. "The agent passes our test suite 90% of the time. Is it ready to ship?"

1. Ask: 90% of what? Per run or per scenario, pass@k or pass^k?
2. Reset-and-replay each scenario several times. With p = 0.9 per run, pass^3 is 0.73.
3. Ask which modes are in the failing 10%: a T2 mode at 2% blocks the launch; wrong tone at 8% may ship behind a gate with monitoring.

Q3. "How would you protect the agent against prompt injection?"

1. State the premise: no reliable detector exists, so design for a tricked model.
2. Authorization in code: canCall(role, tool). The model may ask for issue_refund; the code decides; denials come back as tool results.
3. Least privilege, T2 behind human approval that times out to deny, input and output guards. Then red-team and turn each successful attack into a fail-closed adversarial test.

Q4. "Our agent bill is too high. What would you do?"

1. Profile from traces: tokens per span and per conversation, and the three suspects: history growth, retrieval depth, repeated schemas.
2. Free wins first: prompt caching on the stable prefix, retrieval depth tuned by recall@k, shorter schemas and outputs, each measured with the suite.
3. A cascade calibrated on labeled data, threshold chosen against the accuracy floor, then the upgrade drill on every model release.

Q5. "Tell me about a time an evaluation was wrong."

1. Pick a reward-hacking story: the optimizer that learned to write "I have verified this."
2. How you caught it: a second judge of a different kind (a code check) and spot-reading ten traces.
3. What changed: the judge definition, a held-out test slice, human review of prompt diffs. Then what you would do earlier next time.

4. Portfolio

The strongest interview answer is "here it is." Four artifacts, each one page, from the course project or your own.

Run this checklist before the interview:

• Every number traces to a file in the repo.
• Every judge in the table has a labeled set behind it.
• The test slice was read once, and the log shows when.
• The frontier chart uses the same suite, seeds, and workload for every point.
• You can explain each artifact in under two minutes without notes.

Summary

• The nine traps share one flaw: a number with nothing behind it. Labeled data, binary definitions, and a frozen test slice answer all of them.
• Answer as Analyze, Measure, Improve, in that order, with a number from a loop you ran attached to each verb.
• Know the arithmetic: TPR and TNR instead of accuracy, pass^k instead of pass@k, corrected prevalence with an interval.
• Security means authorization in code and fail-closed adversarial tests; cost means a profile, caching, a calibrated cascade, and a frontier.
• Bring a portfolio: a taxonomy, a judge validation table, a CI gate, and a frontier chart, each traceable to files.