Improving Agents: Cost

Why this matters

Sprout works. The optimized config from L8 scores 92% on the suite, the CI gate is green, and Maya says the refund escalations finally make sense. Then the invoice arrives: $3,000 for the month. Pip's Plant Shop made $8,000 in the same month. Pip does not say no. Pip asks a better question: "Can it be cheaper without getting worse?"

Dev opens the billing page and sees one number, with no breakdown by conversation, turn, or tool. The traces from L2 have that breakdown, down to the token. This lesson turns them into a cost profile, a list of fixes with measured savings, and a rule for "without getting worse": the suite decides, not a feeling.

1. Where the money goes

Every model call is billed the same way:

cost = input tokens × input price + output tokens × output price

Input tokens are everything you send: system prompt, tool schemas, the conversation so far, tool results. Output tokens are what the model writes back, including tool calls. Input is usually far larger, because you resend the whole conversation on every call (L0).

Where do 53,500 input tokens come from in six calls? Three usual suspects:

1. History growth. Every call resends the system prompt, the tool schemas, and every earlier message.
2. Retrieval depth. search_care_guide returns ten article chunks, and they are resent on every later call.
3. Repeated tool schemas. Six tool definitions with long descriptions ride along on every call, used or not.

2. Profiling

A cost profile says which tokens came from where. You build it from traces (L2), because every model-call span records its token usage.

Here is Alex's conversation about order #1042, call by call. Sprout's system prompt is 3,500 tokens and the six tool schemas are 2,500, so 6,000 tokens of fixed prefix ride on every call.

3. Fixes

Work down the profile from the largest bucket, measuring each fix twice: cost per 1,000 conversations and suite accuracy. A fix that drops accuracy below the spec's floor is not a fix. Together, these are token reduction.

Prompt caching

Prompt caching lets the provider store the beginning of your request and charge much less when the next call starts with the same bytes. It is a prefix match, so order matters: tool schemas first, then the system prompt, then the messages, with anything that changes (dates, customer names, the new message) after the cache breakpoint. A cache write, the first time the prefix is stored, costs 1.25× the normal input price. A cache read costs 0.1×. Caches expire after a few minutes without use.

In the SDK, you mark the breakpoint with cache_control: { type: "ephemeral" } on the last stable system block, and you send tools in the same order on every call.

History trimming

History growth has two cures: a window (keep only the last N turns) or a summary (replace old turns with a short recap). Both can lose facts. Dev's four-turn window saved $3 per 1,000 conversations, because Sprout's conversations are short, and produced a new failure: Sprout forgot Alex's order number and asked for it again. Reverted, logged.

Retrieval depth

Dev checked recall@k on the retrieval labeled set from L5: recall@10 was 0.96 and recall@3 was 0.94. Three chunks are 1,260 tokens instead of 4,200, resent three times: 8,820 tokens saved per conversation, $44 per 1,000. Suite accuracy moved from 92.0% to 91.5%, inside the bootstrap interval. Kept.

Tool-schema dedupe

Two rules: do not send tools the current role cannot call (a customer never needs the analyst tools from L3), and keep each description to what the model needs in order to choose the tool.

Shorter outputs

Output tokens cost five times what input tokens cost. Sprout's final care reply to Alex was 620 tokens of repotting instructions. A prompt rule ("answer in under 150 words unless the customer asks for detail") and a max_tokens backstop brought output from 1,300 to 930 tokens: about $9 per 1,000. Accuracy held.

4. Routing and cascades

Model routing sends each request to a model chosen by a rule. A cascade is a routing pattern where a cheap model tries first and a bigger model takes over when the cheap one is unsure or the task is hard.

The cheap model costs about a fifth as much, but on the dev slice it scores 87%, below the 90% floor in Sprout's spec. The question is which conversations it handles well. Only labeled data can answer that.

Calibration means checking a confidence score against reality. Sprout's cheap configuration returns a confidence number between 0 and 1 with each reply. On its own it is a guess the model made about itself. Calibration asks how often, at each confidence level, the cheap model was actually right.

Now turn thresholds into a decision. An escalated conversation costs the cheap attempt plus the big model's run.

Dev picks 0.8: the lowest threshold that still clears the floor, at $53 per 1,000 conversations. Then, once, the test slice: 90.8%. The decision holds.

Rules can be code too: messages with "refund," "lawyer," or "chargeback" skip straight to the big model, because those filled the cheap model's worst bucket.

5. The weights track

Everything so far changed prompts, tools, harness code, and routing. The weights track changes the model itself. Three doors:

• Distillation: train a small model to imitate a big model's outputs on your traffic, so it can take over more buckets.
• Supervised fine-tuning (SFT): train on labeled examples of correct behavior, so the model learns your conventions directly.
• Reinforcement learning (RL): train against a reward signal, such as your suite, so the model is pushed toward what your evaluators pass.

All three are expensive to start and to keep. You now own a model: hosting or a tuning bill, versioning, a full suite run on every retrain, and, for RL, a fresh chance at reward hacking (L8). Open a door only when the cheaper rungs are exhausted.

6. The upgrade drill

Models ship every few months, and each one moves every point on your frontier. The upgrade drill finds out how without guessing. Because every configuration is a file (L8), it is mechanical:

1. A new model ships.
2. Add one frontier config file per candidate: the new model in each role it could play.
3. Run the full suite on every committed config, same seeds, same workload.
4. Redraw the frontier.
5. Decide per config: deploy it, keep it on file for the next drill, or retire it because something now dominates it.

for cfg in configs/*.json; do
  npm run suite -- --config "$cfg" --seed 42 \
    --out "results/2026-11-20/$(basename "$cfg" .json).json"
done
npm run frontier -- results/2026-11-20   # prints the table, marks dominated configs

7. Homework 2

Homework 2 closes the Improve verb with two parts, graded against the rubric below. Part A: route a prompt, tool, or harness change to the top failure mode in your own taxonomy, run the manual fix loop on your dev slice, and freeze a winner on your test slice, read once. Part B: profile cost from your traces, measure one caching change, calibrate a cascade on labeled data, and run the upgrade drill across at least two committed frontier configs.

Summary

• Cost is tokens in times price plus tokens out times price. Input dominates because every call resends the whole conversation. Profile it from traces.
• The usual suspects are history growth, retrieval depth, and repeated tool schemas. Cache the stable prefix first, then trim retrieval, schemas, and outputs, measuring each change with the suite.
• A cascade sends easy conversations to a cheap model and escalates the rest. Calibrate the threshold on labeled data and pick the lowest one that clears the accuracy floor.
• The weights track (distillation, SFT, RL) is for flat prompt search, true capability limits, and savings that pay for owning a model.
• Keep every configuration as a file and run the upgrade drill when a model ships: full suite, redraw the frontier, then deploy, keep, or retire.