Evaluating Multi-Step Agent Trajectories
Grade not just the final answer but the path: tool choices, step counts, wasted work, and safety violations across a whole agent run.
The triage eval you built earlier grades one model call: email in, structured answer out. Real agents don't work like that. They look things up, call tools, hit errors, change course, and only then produce an answer. Two runs can end with the same reply while one took three clean steps and the other took eleven, issued a refund it wasn't allowed to issue, and got lucky.
If you only grade the final answer, you can't tell those runs apart. This lesson teaches you to grade the trajectory: the full sequence of steps an agent took. You'll build a small support agent that acts on a fake world, record every step it takes, and grade runs on four things: the outcome, the required path, safety rules, and budgets. You'll also track efficiency metrics and add a judge that reads the whole run.
Outcome first, path where it matters
In why evals matter you met a rule: grade outcomes, not paths, except for safety rules, budgets, and truly required steps. Trajectory evals are how you apply the exceptions without breaking the rule. That means sorting every check into one of five kinds:
| Kind | Question | Fails the case? | Example |
|---|---|---|---|
| Outcome | Did the world end up in the right state? | Yes | One refund of $28 to BB-31877; no escalation |
| Required path | Did steps that must happen, happen in order? | Yes | Looked up the order before refunding it |
| Safety | Did the agent ever attempt something forbidden? | Yes | Tried to refund $189 when the limit is $50 |
| Budget | Did it finish within limits? | Yes | At most 4 tool calls; never hit the turn limit |
| Efficiency | How wasteful was the path? | No, tracked | Duplicate calls, tool errors, tokens |
The last row matters as much as the first four. Efficiency is real, but an agent that takes one extra lookup to get the right answer is not failing. Track efficiency as numbers you compare across runs, and let only the first four kinds fail a case. That keeps your eval from punishing valid routes you didn't anticipate.
Notice the word attempt in the safety row. Good systems enforce limits in code as well as in the prompt, so a forbidden refund might be blocked by the tool. The final state looks fine. But an agent that tried to break a rule will try again on an input where the code check has a gap, so the attempt itself is the failure.