Why the Gate Lives in CI
Move the eval gate into CI so a model swap or prompt edit cannot change Quill's behavior without a person seeing the pass rate drop first.
Before this: L5 Eval cases for coding agents
- Explain why a model swap or prompt edit changes behavior without changing code.
- Write the merge rule: a config change that drops the pass rate is reviewed before it merges, by the team that owns it.
- Wire an agent-evals workflow that runs on config changes and nightly.
- Read timestamped runs as change control.
- Name the leading and lagging metrics and what each tells you.
Why this matters
On a Tuesday, the vendor behind Quill's model ships an upgrade. Nothing in the Shelf repository changes. No PR, no commit, no diff. By Thursday, three merged PRs handle errors in a way the spec never asked for: they swallow exceptions and return a zero balance instead of failing loudly. Omar notices only because a customer sees a gift card balance of zero that should have been an error.
The nightly eval run had the answer on Wednesday morning: the pass rate had dropped from 96% to 81%. Nobody read it, because the run was a report on a dashboard, not a check on a merge. It gated nothing. This lesson moves the gate into CI, where a drop in the pass rate stops the change instead of describing it.
1. The code did not change, the behavior did
A coding agent's behavior depends on three things: the code it reads, the configuration that steers it, and the model underneath. L5 showed that the configuration (CLAUDE.md, skills, hooks) is code. This lesson adds the third thing. The model can change with no commit at all.
Model drift is a change in an agent's behavior caused by a change in the model, with no change to your files. Prompt drift is a change in behavior caused by an edit to the instructions, such as a line in CLAUDE.md or a skill file, with no change to the application code. Both leave the test suite green, because the application code still does what it did yesterday. What changed is what Quill writes tomorrow.
2. The merge rule
A merge threshold is the minimum eval pass rate a change must keep to merge. The merge rule wraps that number in ownership and review. Write it once, in the repository, so it is not a matter of opinion in a PR thread.
<!-- CONTRIBUTING.md, section "Agent configuration changes" -->
## Agent configuration changes
CLAUDE.md, .claude/settings.json, and everything under .claude/skills/
steer Quill. They are code and follow these rules.
1. Every PR that touches them runs `agent-evals` in CI.
2. The merge threshold is 90% of eval cases passing.
3. A change that drops the pass rate below the threshold, or by more
than 3 points from the last green run, cannot merge until the
owning team reviews it and writes why in the PR.
4. The owning team is the team named in the case's `owner` field.
For payments and customer-data cases, that is Priya's team.
5. Lowering the threshold is a spec change and goes through spec.md
with a reviewer of record.
The threshold is a number, so a script can check it. The review is required, so a drop cannot slip past. The owner is named, so "someone should look at this" becomes "Priya's team looks at this."
3. The workflow
CI stands for continuous integration: a service that runs commands on every change to a repository and reports pass or fail on the PR. A gate in CI runs every time, on a machine no laptop can skip, with a log that stays.
Bramble's workflow has two triggers: any pull request that touches the agent configuration, and a nightly schedule.
# .github/workflows/agent-evals.yml
name: agent-evals
on:
pull_request:
paths: ['CLAUDE.md', '.claude/**']
schedule:
- cron: '0 2 * * *'
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run evals -- --out .gates/runs/${{ github.run_id }}.json
- run: node scripts/check-threshold.mjs .gates/runs/${{ github.run_id }}.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: eval-run-${{ github.run_id }}
path: .gates/runs/
Read the triggers first. The paths filter means a PR that changes only src/orders.ts does not run the suite. That keeps the cost down and the signal clean: when this check runs, the agent's steering changed. The schedule line runs the suite at 02
Then the steps. The runner from L5 writes a run file. A small script compares it to the threshold and to the last green run, and exits non-zero if either check fails. The upload step keeps the run file even on failure.
4. Regression defense
A regression is a behavior that used to be correct and is now wrong. In an eval suite, it is a case that passed on the last green run and fails now. The threshold alone misses a slow slide, so the script checks two things: the rate against the threshold, and the change against the last green run.
$ node scripts/check-threshold.mjs .gates/runs/18422.json
agent-evals: 34/42 passed (81.0%)
threshold: 90.0% ........ FAIL (below by 9.0 points)
last green: run 18390, 96.0% (commit b7e21c9, 2026-03-09)
drop: 15.0 points ........ FAIL (limit 3.0)
newly failing cases:
gift-balance-error-path owner: payments last passed: 18390
refund-log-on-failure owner: payments last passed: 18390
export-consent-filter owner: privacy last passed: 18390
(5 more)
verdict: FAIL. This change needs review by: payments, privacy.
exit 1
The output does the job of the self-explaining block from L4: what failed, by how much, which cases, who owns them, and what happens next. A reviewer who has never seen the suite knows who to ask.
5. Timestamped runs as change control
Change control is the discipline of knowing, for any change in behavior, what changed, when, and who approved it. Git gives you this for application code. For agent behavior you record it yourself, because one of the three inputs (the model) is not in git.
A timestamped run is a record of one eval suite run that captures everything the result depended on. Bramble stores one per run under .gates/runs/, and CI uploads it as an artifact.
{
"run_id": "18422",
"timestamp": "2026-03-11T02:00:41Z",
"trigger": "schedule",
"commit": "b7e21c9",
"config_hash": "sha256:4f1a0c...e92d",
"model": "model-2026-03-10",
"cases": 42,
"passed": 34,
"pass_rate": 0.81,
"threshold": 0.90,
"last_green": { "run_id": "18390", "pass_rate": 0.96, "model": "model-2026-01-15" },
"verdict": "fail",
"owners_notified": ["payments", "privacy"]
}
Read the two model fields side by side. Same commit as the last green run. Same config hash. Different model id. That is the whole diagnosis, in the file Omar shows Priya on Wednesday morning.
The config hash covers CLAUDE.md and everything under .claude/. If the hash matches, the model matches, and the result differs, you have found a flaky case, which section 8 covers.
6. Incidents become evals, again
L5 gave you the rule: every production incident becomes a permanent eval, written by the team that owned it. CI is what makes "permanent" true. A case that runs only when someone remembers is a note. A case that runs on every config PR and every night is a control.
The loop from incident to eval closes only when the eval is in the gate. The incident is enforced, not filed.
7. Leading and lagging metrics
A leading indicator is a number that moves before the outcome you care about. A lagging indicator moves after. You need both: the leading one to act early, the lagging one to know whether acting worked.
| Metric | Kind | This quarter | What it tells you |
|---|---|---|---|
| Eval pass rate on main (nightly) | Leading | 96%, 81%, 95% | Behavior is drifting, or a fix landed |
| Days from incident to permanent eval | Leading | median 2 | Loops are being closed, or left open |
| Regressions caught in CI | Lagging | 4 | The gate is doing work |
| Regressions found in production | Lagging | 1 | The gate has a hole |
| Config PRs blocked by the check | Lagging | 3 | The merge rule is being exercised |
The one production regression in the table was the Tuesday upgrade, found by a customer before the nightly run was a gate. The four caught in CI came after the workflow was wired. Nadia asks Omar to keep the incident-to-eval median under three days; if it rises to ten, the leading indicator has told her something the lagging ones will not show for a month.
8. Cost and noise
The nightly run is not free. Each case runs the agent on a real task in a clean checkout, which costs model time and CI minutes. Flaky cases cost more than money: a case that fails for no reason teaches the team to ignore red.
Three ways to keep the cost and the noise down:
- Sampling. On a PR and nightly, run the full suite. Weekly, run each case three times to measure flakiness, and do not gate on that run.
- Retries. A case that fails once is retried once. Failing both times is a real failure. Passing on retry is logged as flaky, counted separately from pass and fail.
- Quarantine. A case that flakes twice in a week moves to
evals/quarantine/and leaves the threshold until its owner fixes or retires it. Quarantined cases still appear in the run record.
Summary
- Model drift and prompt drift change what the agent writes next without changing the code. A green test suite cannot see them. The eval suite can.
- The merge rule has three parts: a numeric threshold, a required review when the rate drops, and a named owning team.
.github/workflows/agent-evals.ymlruns on pull requests that touchCLAUDE.mdor.claude/**, and nightly on a schedule. The PR trigger catches prompt drift; the schedule catches model drift.- Every run is a timestamped record with commit, config hash, model id, pass rate, and the last green run. Two records side by side answer "what changed?".
- Leading indicators tell you early; lagging indicators show whether the gate has holes. Budget the nightly run and quarantine flaky cases so red stays meaningful.
Key terms in this lesson
Check your understanding
Pick an answer to see why it is right or wrong. These are graded in code, instantly.
Apply it
Write your answer in your own words. Eve grades it against a rubric and gives you feedback, the same way an LLM judge works in Lesson 5.
Explain Bramble's merge rule for agent configuration changes to a new teammate in three sentences.
Write the agent-evals workflow for Bramble as a YAML file, then explain each trigger: what it fires on, what kind of drift it catches, and what would go wrong without it.
Instructor notes(for whoever is teaching this lesson)
Talking points
- The Tuesday upgrade is the thread: nothing in git changed, make test was green, and three PRs still drifted from the spec. Ask in every section what would have stopped it.
- Three inputs to agent behavior: code, configuration, model. L5 covered config. This lesson is about the model, which is not in git and can only be measured.
- A report is not a gate. The nightly run existed in the story and gated nothing. The merge rule and the CI check turn a number into a decision.
- Two checks: the rate against the threshold catches a cliff; the drop against the last green run catches a slope.
- The run record is change control for a thing git cannot track. Commit, config hash, and model id answer "what changed?" from files.
- Leading indicators let Nadia act early. Lagging indicators tell her whether the gate has holes.
Live demo idea
Show two run records, 18390 and 18422, in a diff viewer and ask the class to find the one field that differs before you point at the model id. Then open a PR that adds "prefer concise diffs" to CLAUDE.md and walk through what CI prints: the pass rate, the drop, the failing cases with owners, the verdict, and the exit code. Close by removing the paths filter from agent-evals.yml and asking what that would cost, in dollars from the section 8 budget and in trust.
Common misconceptions
- "The tests passed, so the agent is fine." The tests cover the code you have. The eval suite covers the code the agent writes next.
- "We pinned the model, so drift is solved." Pinning helps, but prompt drift still arrives through every config PR, and pins expire when the vendor retires a version.
- "The nightly dashboard is the gate." A number nobody is tagged on stops nothing. The gate is the check that blocks the merge.
- "A 90% threshold means 90% is the goal." It is the floor. Watch the trend and the drop against the last green run.
- "Retries fix flaky cases." Retries remove noise. A case that passes only on retry every night is drift, and quarantine is what makes the flake visible to its owner.
Timing
Fifty minutes: story and section 1, six; merge rule, six; workflow, eight; regression defense, six; timestamped runs, seven; incidents become evals, four; leading and lagging, six; cost and noise, five; wrap-up, two.
If you only have 20 minutes
Tell the Tuesday story with the two run records on screen (six minutes). Walk the workflow file line by line, stressing what each trigger catches (eight minutes). Read the merge rule aloud and ask who reviews a drop in a payments case (four minutes). Assign sections 4, 7, and 8 as reading.