Eval Cases for Coding Agents

Why this matters

Omar opens CLAUDE.md and adds one line: "Prefer concise diffs." Reasonable advice. Quill takes it seriously. Over the next month, every task Quill touches loses something: the comment explaining the gift card rounding rule, three docstrings in the order service, the header block on a migration. No production code breaks. make test is green on every PR. The diffs look tidy, which is what Omar asked for. Nobody notices until Priya asks why the refund limit has no explanation next to it anymore. Omar traces it back to his one-line edit. Nothing in the repository could have told him sooner, because nothing was measuring what Quill does across many tasks. This lesson builds that measurement: a suite of eval cases that runs every time the agent's configuration changes.

1. What an eval case is

An eval case is a real task paired with the outcome the team accepted. The task is the prompt, written in the words it arrived in. The outcome is a set of checks: tests pass, lint is clean, behavior is unchanged, policy is followed. Run the agent on the prompt, run the checks, and you get a pass or a fail.

Everything else in this lesson is about where cases come from, what checks look like, and how to keep the suite trustworthy.

Why "in the words it arrived in"? Because a polished prompt tests the polished prompt. The messy ticket Nadia wrote on a Friday is what Quill will see next time.

2. Start with 20 to 50 real tasks

You do not need a research project to build a starter suite. You need last month's merged pull requests and the tickets behind them. Each one is a task someone asked for and an outcome someone accepted, which is an eval case waiting to be written.

Twenty to fifty cases is enough to notice a regression and small enough to run nightly. Fewer than twenty and one flaky case swings the pass rate by five points. More than fifty and the team stops running it.

How to mine them:

1. List merged PRs from the last four to eight weeks.
2. For each, find the original request: the ticket, the chat message, the intent.md.
3. Copy the request verbatim as the prompt.
4. Write down what the reviewer checked before approving. Those are your checks.
5. Skip PRs where nobody can say what "accepted" meant. They are not cases yet.

3. Writing checks

A check is a command or rule that returns pass or fail with no human in the loop. There are four kinds, and a good case uses at least two.

The first two you already have. Behavior checks catch the quiet damage: the agent fixed the bug and also rewrote the date formatter. A snapshot is a saved copy of an output (a rendered page, an API response) that the check compares against. When the slow-orders-page snapshot showed 40 changed lines in a task that asked for zero visible changes, the case failed, and rightly so.

Policy checks are where coding-agent evals differ from ordinary tests. A policy is a rule the team has decided on, often owned by someone outside engineering. The check can be as small as a grep. When a grep cannot express the rule, a script can. When neither can, a model judge reads the diff and answers a yes or no question, and you trust its verdict less than a grep's.

4. Versioned in the repo

Eval cases live in the repository, under evals/, one JSON file per case. They are versioned files like everything else in the chain: they have history, they get reviewed, and an auditor can read them.

evals/
  README.md
  cr-101-gift-balance.json
  cr-102-refund-limit.json
  cr-103-newsletter-export.json
  typo-checkout.json
  gift-balance-isolation.json
  concise-diffs-keeps-docs.json
checks/
  consent-filter.sh
  owner-scope.sh
  docstring-count.sh

Every case carries an id that matches its file name, an owner who answers for it, and a source pointing at the PR or incident it came from. When a case fails, the owner decides whether the agent regressed or the case went stale.

5. Config is code

Quill is steered by three kinds of files. CLAUDE.md holds the instructions it reads at the start of every session. A skill is a reusable instruction pack under .claude/skills/ for one kind of job. A hook is a command in .claude/settings.json that runs before an action and can allow, ask, or block it. None of them is code in the usual sense. All of them change what code gets written.

Config is code means these files get the same treatment as source: reviewed, versioned, and regression tested. A regression is a behavior that used to be right and is now wrong. When you edit a function, the test suite tells you whether you broke something. When you edit CLAUDE.md, only the eval suite can.

6. Incidents become evals

The rule at Bramble is short: every production incident becomes a permanent eval, written by the team that owned the incident. Not by a central quality group, not later, not "if there is time." The team that felt the pain knows what the accepted outcome should have been.

An incident is anything that reached customers or auditors and should not have. The other-customer balance bug from L0 qualifies, and so does the consent-free export. The case captures three things: the prompt that led to the incident, as close to the original as you can find; a check that would have failed on the bad code; and a source line naming the incident so the case is never retired by accident.

7. Running the suite

A runner does the same four steps for every case: check out a clean copy of the repository, run the agent on the prompt, run each check, record the result. The pass rate is the share of cases where every check passed. The clean copy matters: each case starts from the same commit, so one case's edits cannot leak into the next.

#!/usr/bin/env bash
# run-evals.sh: run every case, print a table, exit non-zero under 90%
pass=0; total=0
for case in evals/*.json; do
  id=$(jq -r .id "$case")
  prompt=$(jq -r .prompt "$case")
  dir=$(mktemp -d); git worktree add -q "$dir" HEAD
  ( cd "$dir" && ./scripts/run-agent.sh "$prompt" > /dev/null 2>&1 )  # starts Quill, waits
  ok=1
  while read -r cmd; do
    ( cd "$dir" && bash -c "$cmd" > /dev/null 2>&1 ) || ok=0
  done < <(jq -r '.checks[].command' "$case")
  total=$((total + 1)); pass=$((pass + ok))
  printf "%-32s %s\n" "$id" "$([ "$ok" = 1 ] && echo PASS || echo FAIL)"
  git worktree remove -f "$dir"
done
echo "pass rate: $pass/$total"
[ $((pass * 100 / total)) -ge 90 ]

The last line is the verdict. Exit zero means the suite is at or above threshold; exit non-zero means the PR waits until someone reads the table. L6 wires this into .github/workflows/agent-evals.yml.

8. Keeping cases honest

A suite is only useful while the team believes it. Three things erode that belief.

Stale cases. The code moved and the case did not. The check greps for a file that was renamed, so the case fails forever, and people learn to ignore red. Rule: a case that fails three runs in a row gets a ticket to its owner, who fixes it or retires it.

Flaky checks. A check that passes and fails on the same code. Timeouts, network calls, tests that depend on the clock. Rule: a flaky check leaves the case until it is fixed; a case with no checks left is retired.

Mood cases. A case whose result depends on how the model phrased something rather than what it did. "The commit message is friendly" is a mood case. Rule: if you cannot name the observable behavior a check measures, delete the check.

Retiring is a normal, reviewed action, with one exception. An incident case is never retired because it passes. It is retired only if the rule behind it changes, and the policy owner signs off.

Summary

• An eval case is a real prompt, in the words it arrived in, paired with the checks the team already accepted.
• Start with 20 to 50 cases mined from recent PRs and tickets; include boring tasks.
• Checks come in four kinds: tests pass, lint clean, behavior unchanged, policy followed. Write them against behavior, not vocabulary.
• CLAUDE.md, skills, and hooks change what the agent writes, so they are code and get regression tested with the suite.
• Every incident becomes a permanent eval written by the team that had it, and stale, flaky, or mood cases are retired through review.