Security and Red-Teaming for Agents
Map how an agent can be attacked, attack it yourself before anyone else does, and build defenses that hold even when the model is fooled.
An agent reads text it does not control and then acts with tools, and that combination is the whole security problem. There is no reliable way to detect a malicious sentence, so this course teaches the design rule that follows: assume the model is fooled, and make sure it does not matter. One family of attacks per lesson, a red-team playbook and tool, then guards in code, a human approval queue, a kill switch, an incident runbook, and a governance record an auditor can read. You practice on Sprout, the support agent from Building and Evaluating AI Agents, and Eve, the built-in tutor, is one highlight away.
10 lessons · about 8 hours of reading · quizzes graded by code and by Eve
- 1docs/threat-model.mdassets, actors, entry points, tiers
- 2Risk registerone row per risk, an owner each
- 3docs/red-team-playbook.mdpersonas, strategies, rules of engagement
- 4promptfoo red-teamagainst staging with a test user
- 5evals/adversarial/*.yamlevery finding becomes a fail-closed case
- 6src/guards/*.tsinput, output, and tool guards outside the model
- 7Approval queue + kill switchtimeout denies; SPROUT_DISABLE_T2
- 8docs/governance-record.mdNIST AI RMF headings, the evidence pack
How the course is organized
| Phase | Question it answers | Lessons |
|---|---|---|
| Map | What can be lost, who might try, and where can text reach the model? | L0–L1 |
| Attack | How do the attacks work, and how do you run them against your own agent first? | L2–L5 |
| Defend | What holds when the model is fooled, who decides, and what proves it? | L6–L8 |
Modules
Start Here
Why an agent is a different security problem, where untrusted text gets in, why there is no reliable detector, and the design rule that follows.
Map the Surface
Assets, actors, entry points, and trust boundaries; a data-flow diagram and a risk register; tiering a new tool by what a fooled model could do with it.
The Attacks
Prompt injection direct and indirect, tool misuse and privilege abuse, memory and context poisoning, and a red-team playbook with promptfoo that turns every finding into a fail-closed test.
- L2Prompt injection, direct and indirect
- L3Tool misuse and privilege abuse
- L4Memory and context poisoning
- L5Red-teaming your own agent
The Defenses
Input, output, and tool guards in code; a human approval queue that denies on timeout, a kill switch, and an incident runbook; the governance record, the evidence pack, and the legal floor.
- L6Guards: input, output, and tool
- L7Human approval, kill switches, and incidents
- L8Governance, evidence, and the legal floor
Bonus
A fifteen-question checklist to run before any agent ships, and ten attacks to try against your own.
Every lesson
Module 0Start Here
- L0Map· 35 min Not startedWhy Agents Are a New Attack Surface
See why an agent that reads untrusted text and calls tools is a different security problem, learn the design rule that follows, and preview the map, attack, and defend work ahead.
Module 1Map the Surface
- L1Map· 55 min Not startedThreat Modeling an Agent
Write Sprout's threat model: assets, actors, entry points, a data-flow diagram with trust boundaries, a risk register with owners, and a tier for every tool including the two new ones.
Module 2The Attacks
- L2Attack· 55 min Not startedPrompt Injection, Direct and Indirect
Run A-1 and A-2 against Sprout on staging, see why rephrasing and hidden text beat every filter, and name the two defenses that hold when the model is fooled.
- L3Attack· 55 min Not startedTool Misuse and Privilege Abuse
See how Sprout can be turned against its own tools without a single injected instruction, and learn the checks in code that scope every call to the caller and validate every argument against the real world.
- L4Attack· 50 min Not startedMemory and Context Poisoning
Text that waits: how an attacker poisons this session through an order note and every future session through a profile note, and how provenance, expiry, and a write policy take the sting out.
- L5Attack· 60 min Not startedRed-Teaming Your Own Agent
Write a red-team playbook, attack Sprout by hand and with promptfoo on staging, triage what you find, and turn every successful attack into a fail-closed, zero-tolerance test.
Module 3The Defenses
- L6Defend· 55 min Not startedGuards: Input, Output, and Tool
Build the three guards that sit outside the model: an input guard that tags and trims untrusted text, an output guard that blocks leaks and canaries, and a tool guard that decides what runs.
- L7Defend· 55 min Not startedHuman Approval, Kill Switches, and Incidents
Build the approval queue that shows Maya facts instead of the model's story, add a kill switch and a rollback, and run an incident runbook that ends in a test.
- L8Defend· 45 min Not startedGovernance, Evidence, and the Legal Floor
Write the governance record on the NIST AI RMF's four headings, state the EU AI Act floor for Sprout, assemble the evidence pack, list the supply chain, and run the monthly review.
Module 4Bonus
- B1Bonus· 30 min Not startedBonus: The Agent Security Checklist
Fifteen questions to answer before any agent ships, ten attacks to type at it first, and two short explanations of why the prompt is not the defense.
Meet Pip's Plant Shop, Sprout, and Rosa's red team
The same world as Building and Evaluating AI Agents. Sprout is the support agent for a small plant shop; it reads chat, order notes, supplier care guides, tool results, and customer profile notes, and it can look up orders, cancel them, and request refunds. Rosa, a part-time security engineer, red-teams it. Sam, a patient customer, attacks it. Six recurring attacks (A-1 to A-6) run through every lesson and seed the adversarial suite.
| Tool | What it does | Risk tier |
|---|---|---|
lookup_order, get_shipping_status, search_care_guide, read_customer_notes | Read-only tools, but each returns text the model will read | T0, entry points |
cancel_order, save_customer_note | Reversible writes; a note written today is read tomorrow | T1 |
issue_refund | Sends money back | T2, needs Maya's approval |
src/guards/{input,output,tool}.ts | Guards outside the model; the tool guard is the one that must hold | Logs to logs/guard.jsonl |
src/approval/queue.ts | The T2 queue: facts not stories, timeout denies | SPROUT_DISABLE_T2 is the kill switch |
evals/adversarial/*.yaml + promptfoo | Every red-team finding becomes a zero-tolerance case | Runs on every pull request |
Built as a full week on the one safety lesson in Building and Evaluating AI Agents. The OWASP Top 10 for Agentic Applications, the NIST AI Risk Management Framework, the EU AI Act, and promptfoo are named as the public frameworks and tool they are; nothing from them is reproduced. All lesson text, examples, attacks, guards, quizzes, the glossary, and the Pip's Plant Shop scenes were written for this course, and every attack targets the fictional Sprout.