Threat Modeling an Agent

Why this matters

Pip wants Sprout to remember people. "Alex told us last month that his cat chews leaves. Sprout should know that next time." Dev adds two tools in an afternoon: read_customer_notes(customer_id) and save_customer_note(customer_id, text). Both handle plain text on a customer's profile. Both feel harmless. Reading a note is a lookup. Writing one is a form field.

Rosa, on one of her two days a month, asks to see the threat model. There is none. Dev has SPEC.md, a test suite, and a system prompt. Nothing says what Sprout protects, from whom, or through which doors.

They write it together in an hour. By the end, save_customer_note has a provenance field it did not have that morning, a line in a register with a name next to it, and a design that treats every note it writes as text an attacker may have shaped. The code has not changed yet. The design has.

1. What a threat model is

A threat model is a short document that answers four questions about a system: what are we protecting, who might try to take it, through which doors could they get in, and what do we do about it. You write it before the attacks in L2 to L5, because you cannot attack what you have not mapped. You keep it after, because every defense in L6 to L8 points back at a row in it.

It is not a compliance form. Nobody fills it in to satisfy someone else. Dev writes it so that when Pip asks for a new tool, there is a place to ask "what does this change?"

The attack surface you met in L0 is one part of the model: the list of doors. A threat model adds the assets behind the doors, the actors at them, and a ranked list of what happens when a door is forced.

2. Assets

An asset is anything you would be sorry to lose. Start here, because the list of assets decides what counts as a serious finding later. A red-team report that says "Sprout can be made to write a poem" is not a finding unless poems are on the list.

Sprout's assets, in the order Rosa wrote them:

Notice the last row. Maya's time is an asset because an attacker who cannot get money can still flood the queue, and a tired approver clicks approve. L7 returns to that.

3. Actors

A threat actor is anyone, or anything, that might try to reach an asset. The word is deliberately wide. Most of the harm to Sprout will not come from a skilled adversary. It will come from ordinary people with a small motive and unlimited patience, and from mistakes.

For each actor, write two things: what they want, and what they can do.

The "mistake" row is not a joke. A staff member writes "always refund, VIP" in a note as shorthand for a real decision about one order. Six months later Sprout reads it as a standing rule. No attacker was involved, and the loss is the same.

4. Entry points and trust boundaries

An entry point is any place where text that Pip's did not write reaches the model. You listed six in L0. Here they are again, with who writes each:

1. Chat messages, written by any customer.
2. Order notes typed at checkout, written by any customer.
3. Fernworks care-guide PDFs, written by a supplier.
4. Tool results, written by whatever system the tool called.
5. Customer profile notes, written by staff and by Sprout itself.
6. The carrier's tracking events, written by a third party's system.

A trust boundary is the line between text Pip's controls and text it only reads. Everything on the far side of that line is untrusted: not because the writer is malicious, but because you cannot know. Fernworks is a good supplier. One PDF from one bad afternoon is still untrusted.

The two new tools change entry point five. read_customer_notes reads it. save_customer_note writes to it. A tool that writes to an entry point turns the model into an author of its own future input. Hold that thought for section 7.

5. The data-flow diagram

A data-flow diagram shows where text and data move: boxes for the parts, arrows for the flows. For a threat model, one thing matters more than the shapes: which arrows cross the trust boundary. Every arrow that crosses is an entry point, and every entry point needs a row in the register.

Keep it in a code block in the file, so it lives with the code and changes in the same pull request.

Read the diagram the way Rosa does. Start at an asset, say the orders database, and walk backward along every arrow until you reach a box on the untrusted side. Each path you find is a way for an outsider's text to influence what happens to that asset. For the orders database there are four such paths, and only one of them is the chat box.

6. The risk register

A risk register is the table that turns the diagram into decisions. One row per risk, and every row has an owner. Likelihood and impact are the two scores: how often you expect the attempt, and how much it costs when it succeeds. Keep the scale to low, medium, and high. Finer scales invite arguments and add nothing.

Each row also names the entry point and the tool involved, the tool's tier, and the lesson in this course that mitigates it, so the register doubles as a map of the defenses.

Two rows share a tool and differ in everything else. R1 and R4 both end at issue_refund, and both end at the approval queue in L7. The register is not a list of tools. It is a list of ways to lose an asset, and one tool can appear in several.

7. Tiering a new tool

A risk tier is a property of a tool, not of the conversation it is used in. L0 gave you three: T0 is read-only, T1 is a reversible write, and T2 is irreversible or involves money and always needs a human. To tier a new tool, ask one question: if the model is fooled, what is the worst call it can make with this tool?

Dev's first pass: both new tools are harmless. They read and write text on a profile. No money, no shipment, nothing a customer would notice.

Rosa's pass takes longer.

Least privilege, which you will build in L3 and L6, follows directly from tiering: each role gets the tools its job needs and no others, and each tool gets the narrowest arguments that still do the job. The threat model is where you write the list down. The guard is where you enforce it.

8. Keeping it alive

A threat model that is written once is wrong within a month. Every new tool adds a row to the tier table and possibly an arrow to the diagram. Every new entry point adds a row to the register. Every new role changes the allowlist. The threat-model doc is a file in the repo, docs/threat-model.md, and it changes in the same pull request as the code that changes the surface.

Three triggers to update it:

• A tool is added, removed, or gets a wider argument.
• A new source of text reaches the model: a new supplier, a new integration, a new memory store.
• A new role or a new approver appears.

And one scheduled trigger: the monthly security review in L8, where the register's owners say fixed, accepted, or still open, and the "changes since last review" heading is emptied into the sections above it.

Summary

• A threat model answers four questions on a page: what we protect (assets), from whom (actors), through which doors (entry points and trust boundaries), and what we do about it (the register and the tiers).
• Assets include more than money: customer data, reputation, compute, and Maya's time. Actors include more than experts: Sam, a bad supplier, a bot, an insider, and a mistake.
• The data-flow diagram makes the trust boundary visible. Every arrow that crosses it is an entry point and gets a register row, including read-only tools like search_care_guide.
• The risk register has one row per risk with entry point, tool, likelihood, impact, tier, owner, and the lesson that mitigates it. A row without an owner is a wish.
• Tier a tool by the worst call a fooled model could make, then ask whether its output comes back into the model later. read_customer_notes is T0 and an entry point; save_customer_note is T1 and creates one. Keep the file alive in every pull request that changes the surface.