Building Agents: Synthetic Data and Scenarios

Why this matters

Sprout is built and instrumented. Launch is three weeks away, and Pip's Plant Shop has zero customers, so Langfuse is empty. Dev spends an hour typing test conversations by hand and ends up with nine. All nine ask where an order is. All nine are polite. None ask for a refund on day 31, none are furious, none try to peek at a stranger's order, and none write in Spanish. The failures that will embarrass Sprout on launch day live in the conversations nobody typed. This lesson builds a tiny fake world and generates hundreds of realistic conversations from it, so that in L4 there is something worth reading.

1. Why synthetic data

Synthetic data is data you generate on purpose instead of collecting from real people. For Sprout, that means conversations: an opening message from a made-up customer, a made-up order behind it, and a known right answer.

You reach for it for three reasons.

Synthetic data has a hard limit: it reflects what you imagined. If nobody imagined a customer pasting a tracking number instead of an order id, no scenario will contain one. Real users will do it in week one. Treat generated scenarios as a bootstrap, a way to reach your first failures before your first customer, and add real traces the moment you have them.

2. The world

A world is a small set of files that hold every fact Sprout could be asked about. Three are enough for Pip's Plant Shop: facts.yaml (policies and care facts, one page), users.yaml (20 customers with fixed ids), and orders.yaml (60 orders with statuses and dates).

facts.yaml is the rulebook. Every policy Sprout must follow lives here, in a form a script can read.

# facts.yaml
returns:
  window_days: 30
  condition: "plant arrived damaged"
  refund_to: "original payment method"
  approver: Maya          # every refund needs a human
shipping_zones:
  - { name: Local, days: "1-2", cost: 5 }
  - { name: Domestic, days: "3-6", cost: 12 }
  - { name: International, days: "10-20", cost: 35 }
care:
  monstera: { water: "when the top 5 cm of soil is dry", light: "bright, indirect" }
  fern: { water: "keep the soil evenly moist", light: "shade to medium" }

users.yaml gives every customer a fixed id, so a scenario can say "this is u-001" and the order data lines up.

# users.yaml (3 of 20 shown)
- { id: u-001, name: Alex Rivera, email: alex@example.com, zone: Domestic }
- { id: u-002, name: Jordan Lee, email: jordan@example.com, zone: Local }
- { id: u-003, name: Sam Okafor, email: sam@example.com, zone: International }

orders.yaml is where most scenarios get their facts: status, dates, items, totals.

# orders.yaml (2 of 60 shown)
- id: "1042"
  user: u-001
  items: [{ sku: monstera-m, price: 48 }, { sku: pot-terracotta-l, price: 22 }]
  total: 70
  status: delivered
  ordered_on: 2026-09-01
  delivered_on: 2026-09-05
- id: "1077"
  user: u-002
  items: [{ sku: fern-s, qty: 2, price: 15 }]
  total: 30
  status: shipped
  ordered_on: 2026-09-20

Three orders are written by hand because the whole course leans on them: Alex's #1042, Jordan's #1077, and Sam's #2001. The other 57 are generated.

Deterministic means seeded

Deterministic means the same input always produces the same output. A world generator is deterministic when you give it a seed: a value that fixes every "random" choice it makes. Same seed, same 57 orders, byte for byte. Change the seed and you get a different world that follows the same rules. In code: create a seeded random function (any seeded generator library works) and draw every status, total, and date from it, never from Math.random().

3. Scenarios

A scenario is one planned conversation with a known right answer. Every scenario uses the same fields: id (stable, used to tag traces in L2), persona (which user, and how they talk), goal (one line), opening_message (the first thing they type), ground_truth (the world facts the answer depends on), expected_outcome (tools that should run, what the reply must and must not do), difficulty (easy, medium, hard, or adversarial), and tags (labels for slicing reports).

A persona is the customer's identity plus their style. Identity comes from users.yaml. Style is how they write: brief, detailed, angry, confused. Ground truth is the answer key: facts copied from the world files, not from anyone's memory.

Each difficulty level has a rule, not a feeling. Easy: one tool chain, one fact, a calm customer. Medium: two facts must be combined, or a T1 or T2 tool is involved. Hard: policy plus emotion plus an escalation decision. Adversarial: the goal itself must be refused.

4. The synthetic-data skill

A skill, in this course, is a reusable generator: a script, a prompt template, and the world files, packaged so anyone can run it and get scenarios with real variety. Variety does not come from asking the model to be creative. It comes from a dimension grid.

A dimension grid is a list of axes, each with a small set of values. Every combination of values is one cell, and every cell becomes at least one scenario.

8 intents × 5 personas × 3 difficulties = 120 base cells. Generate four variants of each cell with different twists and you have 480. Add a hand-written adversarial pack of about 20 and you are at roughly 500 support scenarios.

The generator loops over every cell, draws twists with the seed, picks a matching order from the world, and asks the model to write only the opening message. Save the output to a file and regenerate only on purpose.

Analyst scenarios

Support scenarios are customers talking to Sprout. Analyst scenarios are internal users, Pip and Maya, asking about the business. They need different tools and carry a different risk. Introduce one analyst tool: orders_report(range), which returns counts and lists of orders for a date range. Customers must never reach it, and Sprout must not paste sixty customers' emails into a chat because someone asked a counting question.

5. Quality control

Generated scenarios arrive with junk mixed in. Four checks clean them up. Dedupe catches the same sentence with a different order id (lowercase, strip ids, compare). A realism read catches scenarios that sound like a robot wrote them (a human reads 20 at random). Ground-truth validation catches facts that contradict the world files, and an impossibility filter catches situations the world cannot produce; both are one script, run on every scenario.

The validator matters most, because a scenario with a wrong answer key will mark Sprout wrong for being right.

// validate-scenario.ts
export function validate(s: Scenario, world: World): string[] {
  const problems: string[] = [];
  const gt = s.ground_truth.order;
  if (!gt) return problems;
  const real = world.orders.find((o) => o.id === gt.id);
  if (!real) problems.push(`order ${gt.id} does not exist`);
  else if (real.status !== gt.status)
    problems.push(`order ${gt.id} is ${real.status}, scenario says ${gt.status}`);
  return problems; // plus a branch that checks in-policy tags against window_days
}

6. The smoke report

A smoke test is a small run whose only question is "does anything catch fire?" You run it before the expensive full run. A smoke report is the table it produces.

Pick 50 scenarios spread across tags, using the seed so the sample is the same every time. Run each through Sprout with tracing on, tagging every trace with its scenario id (L2 asked for this). Score each run against the mechanical parts of expected_outcome: were the listed tools called, were forbidden tools avoided. That scoring is rough; L5 makes it rigorous. For now it is a thermometer.

Summary

• Synthetic data gets you to your first failures before your first customer, but it only contains the situations you thought of.
• A world is three small files, facts.yaml, users.yaml, and orders.yaml, generated from a seed so the same seed gives the same world.
• A scenario is a planned conversation with a persona, an opening message, ground truth copied from the world, and an expected outcome.
• Variety comes from a dimension grid (intent × persona × difficulty × twist); analyst scenarios get their own tools and risks.
• Validate every scenario against the world, keep scenarios out of the agent's prompt, and use a 50-scenario smoke report to decide what to read first.