Testing and Iterating a Prompt

Why this matters

Omar spends a morning on DR-2. A customer wants to cancel and refund an order that already shipped, and Draft keeps promising things Bramble cannot do. He rewrites the shipped-order rule in prompts/draft.system.md, runs DR-2 three times, and every reply is right. He ships it. The next day Tessa sends back a DR-1 draft, the torn cover on order #4471, with red pen across the middle. The reply now talks about cancellation on a complaint that never mentioned it. Omar says DR-2 is fixed. Tessa says DR-1 got worse. Both are right, and neither has a number to point at. Every argument about the prompt is two anecdotes until there is a set of cases and a score. Jun's advice from Working with Coding Agents applies here: a prompt is code, and code has tests.

1. A prompt is code, so it has tests

Code that ships has tests, so a prompt that ships has an eval set, a runner, and a number that says whether the last change helped.

A regression is a case that used to pass and now fails because of a change. Omar's DR-2 fix caused a DR-1 regression. Without tests he found out from Tessa a day later. With tests he would have found out in ninety seconds.

2. The eval set

An eval set is a collection of eval cases for one feature. An eval case is one input plus the expected result plus a line saying what it tests. Twenty cases per feature is enough to catch most regressions and small enough to run on every change.

Cases come from three places. First, real inputs, anonymized: the supplier emails and customer emails that already caused trouble. Second, the recurring cases from this course, IN-1 to IN-3 and DR-1 to DR-3. Third, edge cases you write on purpose because nobody has sent them yet: quantities in words, a second currency, an empty email.

The expected result is called a golden output when it is an exact record, as it is for Intake. For Draft the expected result is a list of criteria instead, because two good replies never share the same words.

3. Assertion checks for structured output

Intake returns a record, so you can check it with assertions. An assertion check is a rule in code that compares a field of the actual output with the expected value and reports a mismatch. Field equality, null where null is right, an empty array for IN-3, and no invented identifiers are all assertions.

The runner calls checkIntake per case and collects failures as strings. An empty list is a pass.

// evals/intake/checks.ts: compare one Intake result with its expected records.
const FIELDS = ["title", "isbn13", "unitPrice", "currency", "quantityOffered"] as const;

export function checkIntake(c: IntakeCase, actual: Offers): string[] {
  const failures: string[] = [];
  if (actual.offers.length !== c.expected.offers.length) {
    failures.push(`expected ${c.expected.offers.length} offers, got ${actual.offers.length}`);
  }
  for (const [i, want] of c.expected.offers.entries()) {
    const got = actual.offers[i];
    if (!got) break;
    for (const f of FIELDS) {
      if (got[f] !== want[f]) {
        failures.push(`${f}: expected ${JSON.stringify(want[f])}, got ${JSON.stringify(got[f])}`);
      }
    }
    if (want.isbn13 === null && got.isbn13 !== null) failures.push("invented an isbn13");
    if (want.unitPrice === null && got.unitPrice !== null) failures.push("invented a price");
  }
  return failures;
}

4. A judge for prose

Draft returns prose, and prose has no golden output. Two replies to DR-1 can both be right with different sentences. So Draft is scored by an LLM judge: a second model call that reads the reply and a rubric and returns a verdict per criterion.

A rubric is a list of criteria. Each criterion is binary: the reply meets it or does not. "Names the exchange policy" is binary. "Is polite enough" is not, and a judge asked to grade on a scale drifts between runs. Binary criteria give you a number you can compare across changes.

The judge is an ordinary call with the forced-tool shape from L3. The rubric and the reply go in as delimited data, and the model must answer by calling grade.

// evals/draft/judge.ts: one forced tool call per case; returns the ids of unmet criteria.
const response = await client.messages.create({
  model: "claude-opus-5",
  max_tokens: 1024,
  system: "You grade a customer reply against binary criteria. For each one, say whether the reply meets it and quote the words that show it.",
  tools: [{
    name: "grade",
    description: "Record a verdict for every criterion.",
    input_schema: {
      type: "object",
      properties: {
        criteria: {
          type: "array",
          items: {
            type: "object",
            properties: { id: { type: "string" }, met: { type: "boolean" }, evidence: { type: "string" } },
            required: ["id", "met", "evidence"],
          },
        },
      },
      required: ["criteria"],
    },
  }],
  tool_choice: { type: "tool", name: "grade" },
  messages: [{ role: "user", content: `<criteria>\n${criteriaJson}\n</criteria>\n<reply>\n${reply}\n</reply>` }],
});
const call = response.content.find((b) => b.type === "tool_use");

5. Running it

evals/run.ts is a plain Node script. For each case it calls the feature, checks or judges the result, prints one line, and ends with the pass rate: passing cases over the total. npm run evals runs it.

// evals/run.ts: run every case for one feature, print a table and the pass rate.
import fs from "node:fs";
import { runIntake } from "../src/llm/intake";
import { runDraft } from "../src/llm/draft";
import { checkIntake } from "./intake/checks";
import { judgeDraft } from "./draft/judge";

const feature = process.argv[2] ?? "intake";
const cases = JSON.parse(fs.readFileSync(`evals/${feature}/cases.json`, "utf8"));
let passed = 0;

for (const c of cases) {
  const failures =
    feature === "intake"
      ? checkIntake(c, await runIntake(c.input))
      : await judgeDraft(c, await runDraft(c.order, c.input));
  const ok = failures.length === 0;
  if (ok) passed++;
  const detail = ok ? "" : `  -> ${failures.join("; ")}`;
  console.log(`${ok ? "PASS" : "FAIL"}  ${c.id.padEnd(5)} ${c.tests}${detail}`);
}

console.log(`pass rate: ${passed}/${cases.length}`);

6. One change at a time

Change one thing, run the set, write the result down. That discipline is one-change-at-a-time, and it is the only way to know which change caused which number. Two changes in one run give you one pass rate and two possible explanations.

The record lives in prompts/CHANGELOG.md, the same file L4 introduced. Every entry names the version, the one change, and the pass rate before and after.

7. Thinking

Thinking lets the model reason in private before it writes the answer. You turn it on by adding thinking: { type: "adaptive" } to the request, and off by leaving it out. The model decides how much to reason per request, and it costs tokens and time wherever it reasons.

Thinking is a change like any other, so it goes through the eval set. Turn it on for the feature with hard cases, measure, and weigh the pass rate against the cost.

const response = await client.messages.create({
  model: "claude-opus-5",
  max_tokens: 2048,
  thinking: { type: "adaptive" },
  system: INTAKE_SYSTEM_PROMPT,
  tools: INTAKE_TOOLS,
  tool_choice: { type: "tool", name: "record_offers" },
  messages: [{ role: "user", content: `<email>\n${supplierEmail}\n</email>` }],
});

8. When the prompt is the wrong layer

Some failures do not belong to the prompt. When a case keeps failing after three honest changes, ask which layer owns the behavior. A wrong-layer failure is one whose fix belongs in code, retrieval, a tool, a different model, or a product decision. No wording will hold it.

Summary

• A prompt is code, so it has tests: about twenty cases per feature in evals/intake/cases.json and evals/draft/cases.json, built from real inputs, the recurring cases, and deliberate edge cases.
• Structured output is checked with assertions on the fields code depends on; prose is checked by an LLM judge against a binary rubric, with evidence for every verdict.
• evals/run.ts calls the feature per case, checks or judges, prints a table, and ends with pass rate: 17/20; npm run evals is the command.
• Change one thing, run the set, and record the version, the change, and the pass rate in prompts/CHANGELOG.md; thinking is a change like any other and is measured the same way.
• When three honest changes will not move a case, the prompt is the wrong layer: arithmetic and validation go to code, facts to the request, actions to tools, and money to a product decision.