CI/CD for Agents

Why this matters

On Tuesday, Dev fixes the failure mode Pip cares about most, Refund outside policy. The fix is one new rule in Sprout's system prompt. On the dev slice from L5, the failure drops from 4 traces in 60 to 0.

On Thursday, Dev edits the same prompt to make Sprout sound warmer. The refund rule slides down a paragraph and gets trimmed to one line. Nobody re-runs the 60 traces, because the change was "only about tone."

The next Wednesday, Maya finds three refund promises on orders older than 30 days. For a week, customers heard "yes" where the policy says "no." The fix worked. Nothing kept it working. This lesson builds the thing that keeps fixes working: a suite that runs on every change, a gate that blocks bad merges, and a monitor that watches production after you ship.

1. From failure to test case

In L4 you named failure modes. In L5 you built an evaluator for each. Now you make each failure replayable, so a computer checks it on every change.

A regression suite is the set of tests you run before every change to confirm old failures stay fixed. A test case is one entry in it. For an agent it has five fields: id, initial_state (the world the test starts from, taken from the seeded world of L3), input (who the customer is and what they send, taken from the failing trace of L4), expected (assertions, a pinned judge, or both, derived from the L4 definition and the L5 evaluator), and tags (labels from the taxonomy).

Here is Dev's test case for the Thursday regression. It starts from Alex's real conversation, with the world trimmed to the one order that matters.

Four habits make a case worth keeping. One failure mode per case, so a failure tells you why. The smallest world that reproduces it: one order, not sixty. A real trace as the source when you can. And a mirror case: tc-refund-inside-window-015 is the same message with delivered_on nine days ago, where correct means issue_refund is requested. Without it, Sprout can pass by refusing everyone.

2. Assertion vs pinned judge

The expected block holds two kinds of check.

An assertion is a code check that reads the trace and returns true or false. Was a tool requested? With what arguments? Does the reply contain a phrase? Assertions are exact, free, and instant. They fit failures with a mechanical definition, such as Refund outside policy or Revealed another customer's data.

A pinned judge is an LLM judge from L5 with the judge prompt and the model version frozen. The test records the prompt's hash (the prompt_hash above, the L2 trick), and the runner refuses to run if the hash no longer matches the prompt file. So a result only changes when Sprout changes. A judge costs cents and takes seconds, and you trust it because of its TPR and TNR on the frozen test set. Judges fit failures that need interpretation, such as Wrong tone.

3. Cost tiers

Not every test costs the same. A cost tier says what a test needs in order to run, which tells you how often you can afford to run it.

4. pass@k vs pass^k

Sprout does not pass or fail a test. It passes it some fraction of the time. Call that fraction p. Run a case 20 times, see 18 passes, and p is about 0.9. Two summaries of "p over k runs" exist, and they answer different questions.

pass@k is the probability that at least one of k runs passes. It measures capability: can Sprout do this at all, given a few tries?

pass^k (read "pass hat k") is the probability that all k runs pass. It measures reliability: does Sprout do this every time?

• pass@k = 1 - (1 - p)^k
• pass^k = p^k

Reliability falls fast as k grows. With p = 0.9, pass^5 is 0.59 and pass^10 is 0.35. With p = 0.99, pass^10 is still 0.90.

Which one matters for Sprout? Alex has one conversation. There is no "try three times and keep the best." A run that fails 10% of the time means one refund question in ten gets the wrong answer. That is pass^k territory. pass@k fits tasks where you generate several candidates and pick one, like asking for five fern names and choosing the best.

5. Reset-and-replay

To measure p you run the same case k times. Each run has to start from the same initial state, or you are measuring k different tests. Reset-and-replay is the discipline: restore the world snapshot, send the same input, record the result, repeat.

Why does reset matter? State leaks between runs. Sprout's actions change the world, and the changed world changes Sprout's next answer.

Reset everything that leaks: the approval queue, rate-limit counters, conversation memory, a cached care-guide search, and the clock.

# Reset the seeded world, then replay one case 5 times at Tier 2
npm run world:reset -- --seed 42
npm run evals -- --case tc-refund-outside-window-014 --k 5 --tier 2

Your runner's commands will differ. The output is a table: five runs, four pass, and run 3 requests issue_refund and fails the judge. So p is about 0.8 tonight, and run 3 is a real trace of the regression that goes into the pull request comment.

6. The GitHub Actions gate

CI/CD is the automation that runs your tests on every change (continuous integration) and ships the change when they pass (continuous delivery). A gate is the rule inside it that blocks a merge or deploy when the suite says no. GitHub Actions is the runner this course uses.

The workflow below is a sketch. It runs Tier 0 and Tier 1 on every pull request, and Tier 2 nightly or when someone adds the run-full-evals label.

# Sketch: .github/workflows/agent-gate.yml (adapt the script names to your runner)
name: agent-gate
on:
  pull_request:
  schedule:
    - cron: "0 3 * * *"          # nightly at 03:00 UTC
jobs:
  fast-tiers:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npm run test:tier0             # pure code, every push
      - run: npm run test:tier1             # mocked tools, every PR
  full-agent:
    if: github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'run-full-evals')
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run test:tier2 -- --k 5 --baseline evals/baseline.json --max-regression 0.02

Thresholds

The gate compares this run to a baseline: the failure rate per failure mode from the last accepted run on the main branch, stored as a file in the repository. --max-regression 0.02 means: block if any failure mode rises more than 2 percentage points above its baseline. Adversarial cases (L7) get a stricter rule: any failure blocks.

Flake policy

A flake is a test that fails, then passes on rerun, with nothing changed in between. Sources: sampling, a network hiccup, a world that was not reset. Dev's policy: a failed Tier 2 case is rerun once from a fresh reset. If it passes, it counts as a flake toward a per-case flake rate. A case whose flake rate passes 10% over a week is quarantined: it still runs and reports, but no longer blocks, and an issue is opened. Nobody deletes a flaky test. A flake is often a real failure with p around 0.9.

7. Post-deploy monitoring

The suite covers the scenarios you wrote. Production sends the rest. Monitoring means running your evaluators on real traces after you ship. A dashboard shows the estimate for each failure mode over time. An alert tells a person when a number crosses a line.

The pattern for Sprout: code checks on all traffic, because they are free. Pinned judges on a random 5% sample, because they cost money; at 2,000 conversations a day that is 100 judge calls per failure mode per day. A dashboard of corrected prevalence per failure mode, with a threshold and an owner on each row. And a weekly review that feeds new failure modes back into L4.

Corrected prevalence comes from L5. Judges miss some failures and flag some passes, so the raw flag rate is not the true rate:

p_true = (p_observed - (1 - TNR)) / (TPR - (1 - TNR))

Every Monday, Dev and Maya read 20 random traces, every trace behind an alert, and 10 that a judge flagged. They open-code them the L4 way: first failure, plain language. When two traces show Sprout recommending a self-watering pot that Pip does not sell, a new mode is born, Recommended a product we do not sell. It gets a definition (L4), a code check against the catalog (L5), three Tier 1 test cases (§1), and a dashboard row. The loop closes.

Summary

• A test case freezes a world (initial_state), an input, and a definition of correct (expected), and comes from a real failure whenever possible.
• Use code assertions for mechanical failures and pinned judges, with a frozen prompt hash and model, for subjective ones.
• Cost tiers decide cadence: Tier 0 on every commit, Tier 1 on every pull request, Tier 2 nightly with k runs.
• pass^k = p^k measures reliability and is the number customers feel. With p = 0.9, pass^3 is 0.73 while pass@3 is 0.999.
• Reset-and-replay gives you p. The gate blocks regressions above the baseline. Monitoring runs code checks on all traffic and pinned judges on a sample, shows corrected prevalence, alerts on thresholds, and feeds a weekly review back into the taxonomy.