Monitoring and Improving in Production
Dashboards, alerts, feedback loops, and turning production failures into new evals.
Shipping is the moment your agent starts learning things you couldn't teach it in testing. Real users phrase requests in ways your eval set never imagined, hit tools with inputs you didn't expect, and find the edge cases that matter. Some of those runs will fail quietly: no exception, no error page, just a confident wrong answer or a task left half done.
Monitoring is how you see that happening, and a feedback loop is how you turn it into a better agent. This lesson shows you what to measure at each layer, which dashboards and alerts are worth having, how to collect useful feedback from users, and how to turn production failures into new eval cases so the same mistake never ships twice.
Four layers of measurement
Traditional services mostly need the first layer below. Agents need all four, because an agent can be perfectly healthy as software while failing at its job.
| Layer | Question it answers | Example metrics |
|---|---|---|
| System health | Is the service up and moving? | Error rate, queue depth, time waiting in queue, run duration (p50/p95), worker restarts, model API errors and rate limits |
| Cost | What does this cost, and is it drifting? | Tokens and cost per run, cost per user per day, cache hit rate, runs hitting budget limits |
| Agent behavior | Is the agent working the way we designed? | Steps per run, tool calls per run, tool error rate, stop reason mix, runs hitting the step limit, escalation rate, refusals |
| Outcome quality | Is it actually doing the job well? | Task success rate, thumbs up/down, user edits to agent output, retries and regenerations, approval and rejection rates |
The top two layers tell you when something is broken. The bottom two tell you when something is wrong, which for agents is the more common and more expensive problem.