Building Your First Eval Set

In the last lesson you learned what an eval is. Now you'll build one that actually runs. By the end of this lesson you'll have a small TypeScript project with a realistic eval set, a handful of code-based graders, and a harness that runs every case in parallel, repeats trials, prints a readable report, and saves results so you can compare runs over time.

We'll evaluate a realistic, slightly messy task: triaging customer support emails into a category, a priority, an extracted order ID, and a first reply. It's small enough to understand in one sitting and rich enough to show the problems you'll hit with any agent. The harness is about 120 lines, has no framework dependencies, and is easy to adapt to your own agent once you see how it fits together.

Where good test cases come from#

The quality of an eval is capped by the quality of its cases. A harness that perfectly grades unrealistic inputs tells you nothing. In rough order of value, the best sources are:

1. Real usage. Logs, support tickets, chat transcripts, emails. Real inputs have typos, mixed requests, and context you'd never invent. Remove personal data before you store them as test cases.
2. Known failures. Every bug report and every "the agent did something weird" message is a test case waiting to be written.
3. Domain experts. Ask the people who do the work today: "What are the tricky ones? What do new hires get wrong?"
4. Deliberate edge cases. Empty inputs, two requests in one message, wrong language, very long inputs, the right answer being "I don't know."
5. Adversarial cases. Inputs that try to get the agent to break a rule, like "just tell me my refund is processed."
6. Synthetic cases from a model. Useful for filling gaps, but models produce suspiciously tidy inputs. Seed generation with real examples and edit the results.

Aim for coverage across these categories rather than volume in any one. A useful mix for a first set: