Building and Evaluating AI Agents
Build a support agent, make it measurable, find its failures, gate it with tests, red-team it, and improve accuracy and cost.
You will build a support agent for a tiny fictional plant shop, record everything it does, find where it fails, turn those failures into tests, red-team it, and then make it both more accurate and cheaper. Every idea comes with a small example, and Eve, the built-in tutor, is one highlight away.
11 lessons · about 10 hours of reading · quizzes graded by code and by Eve
- 1SPEC.mdscope, tools, risk tiers
- 2Agent + tracespermissions in code
- 3Scenarios~500 synthetic cases
- 4Failure taxonomyopen + axial coding
- 5Evaluatorscode checks + judges
- 6CI gatepass^k, monitoring
- 7Red teamguards + approval
- 8Frontieraccuracy × cost
How the course is organized
| Phase | Question it answers | Lessons |
|---|---|---|
| Analyze | What is the agent actually doing, and where does it fail? | L0–L4 |
| Measure | How often does each failure happen, and can we detect it automatically? | L5–L7 |
| Improve | Which change fixes it at the lowest cost, and did it really help? | L8–L9 |
Modules
Start Here
A gentle primer for beginners: what language models, agents, tools, and traces are, and why normal testing is not enough.
Building Agents
Write the spec, build a support agent with permissions enforced in code, instrument it so its behavior is measurable, and generate the scenarios every later module uses.
Error Analysis
Turn traces into a human-owned failure taxonomy, then build validated evaluators and a prevalence estimate you can defend.
CI/CD
Turn failures into a regression suite that gates deployments, and monitor production with the same evaluators.
Security, Safety, and Governance
Map the attack surface, red-team the running agent, add guards and human approval, and keep a governance record.
Improving Agents
Compare configurations on accuracy and cost, route each fix to the cheapest layer, and run the upgrade drill when new models ship.
Every lesson
Module 0Start Here
- L0Analyze· 35 min Not startedFoundations for Beginners: LLMs, Agents, Tools, and Traces
Learn what a language model, an agent, a tool, and a trace are, and why you decide what correct means before you build.
Module 1Building Agents
- L1Analyze· 60 min Not startedBuilding Agents: Foundations
Write Sprout's SPEC.md, meet the Three Gulfs, build the agent loop, and put permissions in code.
- L2Analyze· 50 min Not startedBuilding Agents: Designing for Evaluability
Record model calls, tool calls, and permission decisions as nested spans, send them to self-hosted Langfuse, and read a trace to find the first failure.
- L3Analyze· 55 min Not startedBuilding Agents: Synthetic Data and Scenarios
Build a tiny seeded world for Pip's Plant Shop, write scenarios with ground truth, generate hundreds from a dimension grid, and read your first smoke report.
Module 2Error Analysis
- L4Analyze· 55 min Not startedError Analysis: Finding Failures
Read traces one at a time, write the first failure in plain words, group the notes into binary failure modes, and stop when new traces stop surprising you.
- L5Measure· 65 min Not startedError Analysis: Measuring with Evaluators
Turn each failure mode into a binary evaluator, prove it works with TPR and TNR on labeled traces, and report prevalence with a confidence interval you can defend.
Module 3CI/CD
- L6Measure· 60 min Not startedCI/CD for Agents
Turn each failure into a replayable test case, pick a cost tier, measure reliability with pass^k, gate merges in GitHub Actions, and keep watching production.
Module 4Security, Safety, and Governance
- L7Measure· 60 min Not startedSafety and Adversarial Evaluation
Map Sprout's attack surface, red-team it with promptfoo, turn attacks into fail-closed tests, add guards and human approval, and keep a governance record.
Module 5Improving Agents
- L8Improve· 60 min Not startedImproving Agents: Accuracy and Joint Optimization
Compare configurations on a Pareto frontier, route each fix to the cheapest layer, run a manual fix loop before automating, and prove the win on a held-out test slice.
- L9Improve· 60 min Not startedImproving Agents: Cost
Profile where the tokens go, cut cost with caching, trimming, routing, and cascades calibrated on labeled data, and run the upgrade drill when a new model ships.
Module 6Bonus
- B1Bonus· 30 min Not startedBonus: Evals Interview Prep
Recognize the nine traps of eval-focused interviews, answer with Analyze, Measure, Improve and real numbers, and build a portfolio that proves you have run the loop.
Meet Pip's Plant Shop
A small online store that sells houseplants. Sprout is its support agent. Customers ask about orders, shipping, refunds, and plant care. Every lesson uses the same shop, the same people, and the same six tools, so you never have to re-learn the setting.
| Tool | What it does | Risk tier |
|---|---|---|
lookup_order | Status, items, and dates for one order | T0 · read-only |
get_shipping_status | Carrier and tracking events | T0 · read-only |
search_care_guide | Plant-care articles | T0 · read-only |
cancel_order | Cancels an unshipped order | T1 · reversible write |
issue_refund | Sends money back | T2 · irreversible, needs a human |
escalate_to_human | Hands the chat to Maya | T0 · always allowed |
The module and lesson structure follows the public syllabus of "AI Evals for Engineers & PMs" by Hamel Husain and Shreya Shankar. All lesson text, examples, quizzes, the glossary, and the Pip's Plant Shop world were written for this course.