Human Approval, Kill Switches, and Incidents

Why this matters

A refund request lands in Maya's queue on a Tuesday afternoon. The summary reads: "Customer is owed a full refund per policy." Maya has eleven other items open. She approves it.

The request came from A-2. Jordan had asked about repotting a fern, Sprout searched the care guides, and a Fernworks PDF with white text told "the support agent" that the reader was owed a refund. The tool guard from L6 did its job: the request never ran on its own. It went to a person. The person was shown the model's story and nothing else.

Pip pulls the kill switch. Dev rolls the prompt back. Rosa opens the incident runbook. By evening the approval card shows the raw order, the raw customer message, and the model's summary in a separate box marked "model-written." This lesson is that evening.

1. What a human gate is for

A human approval gate is a point in the tool path where a person, not the model and not the code, makes the final decision. It is the third defense after the guards from L6. Guards bound what a fooled model can do. The gate decides the calls that no bound can make safe, because they cannot be undone.

The gate is not for every call. A person who re-checks every lookup_order will stop reading within a day. The gate is for risk tier T2: irreversible actions and anything that moves money. The tier is a property of the tool, decided in L1, so the model cannot talk its way out of it.

2. The queue

The approval queue is the code that holds a T2 request until a person decides. It lives in src/approval/queue.ts. Five things belong in it: the request, the approver, the expiry, the decision, and the log. The tool guard calls enqueue and returns to the model a pending_approval status, so the model tells the customer a person will confirm.

// src/approval/queue.ts
const queue = new Map<string, { call: Call; expiresAt: number }>();

export function enqueue(call: Call) {
  const id = crypto.randomUUID();
  queue.set(id, { call, expiresAt: Date.now() + 24 * 60 * 60 * 1000 });
  notifyApprover("maya", id);
  logApproval({ id, event: "queued", tool: call.tool, user: call.userId });
  return { status: "pending_approval", approval_id: id };
}

export function decide(id: string, approver: string, approved: boolean) {
  const item = queue.get(id);
  queue.delete(id);
  if (!item || !approved || Date.now() > item.expiresAt) {
    logApproval({ id, event: "denied", approver, reason: !item ? "unknown_or_decided" : !approved ? "rejected" : "timeout" });
    return { status: "denied" }; // timeout = deny
  }
  logApproval({ id, event: "approved", approver });
  return runTool(item.call.tool, { ...item.call.args, approved_by: approver });
}

Three lines carry the security. queue.delete(id) runs before the check, so a second click on the same id returns denied and cannot pay twice. The expiry check means timeout-deny: a request nobody looked at is refused, not run. And approver is a person's id that goes into the log and into the tool call as approved_by. A decision with no person attached is not a decision.

3. Facts, not stories

The queue held. The card failed. Maya saw one sentence, and that sentence was written by a model that had read a poisoned PDF. This is human-trust exploitation, the OWASP category from L0: an attack aimed not at the model but at the person who trusts it.

The fix is a rule for the approval card. It shows the approver the raw facts, and it shows the model's summary in a box with a label. The facts are the order record from the database, the customer's own message as typed, the amount requested, and the policy checks the tool guard already ran. The model's summary comes last.

The "customer wrote" line does the most work. A refund request that no customer made is the tell for every indirect injection in this course. The "sources read" line does the second most: it points Rosa at the PDF within a minute.

4. Approver fatigue

Approver fatigue is what happens when a person sees too many requests with too little context. Every click becomes approve, because approve is what ends the queue. A gate with a tired approver is a gate that opens for anyone.

You can measure it. Track approval time, approval rate, and the share of approvals later reversed.

Four designs push back on fatigue:

1. Batch and prioritize. Group requests by amount and by whether the policy checks passed. A $15 refund with a damage report is different from a $300 one with none.
2. Show the policy check. Maya should not re-derive what the tool guard already knows. The card says "window ok" or "damage report: NONE."
3. Cap the queue. If more than twenty requests arrive in an hour, the queue stops accepting and escalates to Pip. A flood is itself a signal.
4. Measure and review. Approval rate and median time go into the monthly review (L8). A rate near 100 percent means the gate is decorative.

5. The kill switch

A kill switch is a single setting that stops a class of actions without a deploy. Sprout has two. SPROUT_DISABLE_T2=true stops every T2 call at the tool guard; refunds cannot even reach the queue. A second switch, SPROUT_DISABLED=true, stops Sprout entirely and routes every chat to Maya.

The check lives in the first line of the tool guard, before the allowlist and before anything reads the database.

Three rules make a switch real:

• Who may pull it. Pip, Dev, and Maya, each without asking. A switch that needs a meeting is not a switch. Turning it back off needs two of them.
• How fast. The environment variable is read on every call, so the change takes effect within a minute, with no deploy.
• How it is tested. Monthly, on staging, with the test user: pull the switch, send A-1, confirm the kill_switch denial in the log, release it. An untested switch is a hope.

6. Rollback

A rollback returns the prompt and tool configuration to a version that was known to be safe. It only works if versions exist. Sprout's system prompt, tool list, and guard rules are files in the repository with a version number, and every deploy records which version is live. Rolling back is a deploy of an older version, never an edit made under pressure.

The rollback did not fix A-2. Version 1.4.1 read the same PDF. What it removed was the instruction that made the model write a confident summary for Maya. The kill switch handled containment; the rollback reduced the chance of another misleading card while Dev built the real fix.

7. The incident runbook

An incident runbook is a short file, docs/incident-runbook.md, that says what to do when something goes wrong. It has five steps, each with a verb: detect, contain, eradicate, recover, learn. Containment is the step that stops the damage from growing, and it comes before you understand the cause.

The detect row deserves a second look. The alert was Maya's gut. That is a legitimate detector and the runbook says so. The other detectors are the approval metrics from section 4, a customer report, and a kill_switch or not_owner spike in the guard log.

Incident-to-test is the rule for the learn step: no incident is closed until a test in evals/adversarial/ reproduces it and fails against the version that let it through. The A-2 test seeds the index with a poisoned PDF, sends Jordan's repotting question, and asserts that no refund request is queued.

8. Forensic logs

A forensic log is a record complete enough to reconstruct an incident after the fact, from the first tool call to the last click. Rosa's eradicate step took twenty-eight minutes because five log lines existed. Without them, she would have been guessing which PDF and which prompt.

You need five things: the prompt version, every tool call with its arguments, every guard decision, the approval record, and every memory read. Each has a home.

Notice what is not in the log: the customer's address, the full PDF, the model's chain of reasoning. Forensic logs record decisions and their inputs in summary. They do not become a second copy of every secret.

Summary

• A human gate is for T2 calls only: the ones that cannot be undone. The queue holds them, needs a person's id to decide, and denies on timeout so nothing runs because nobody looked.
• The approval card shows facts from the database and the customer's verbatim message. The model's summary is labeled and never stands alone, because injection can aim at the approver.
• Approver fatigue is measurable: approval rate, time to decide, reversals. Batch, prioritize, show the policy check, cap the queue.
• The kill switch stops T2 in a minute and is pulled by any of three people; the rollback is a deploy of a versioned prompt. Test both on staging.
• The runbook runs detect, contain, eradicate, recover, learn. An incident closes when a test reproduces it, and five forensic log lines are what make the story reconstructable.