Online Evals and A/B Tests in Production
Measure agents with real traffic: online scoring, user feedback signals, shadow runs, and A/B tests that tell you which version actually wins.
Your new prompt passes the eval suite with a better score than the old one. CI is green. You ship it, and a week later the share of drafts that support staff send without editing has quietly dropped. How? Your eval set is 50 cases you chose. Production is thousands of requests you didn't choose, from people who judge the agent by whether it saved them time, not by whether it passed your graders.
Offline evals, the ones you run on a fixed test set before shipping, tell you whether a change is safe to try. Online evals measure what happens with real traffic. This lesson covers the three online techniques that matter most: scoring a sample of live runs with your existing graders, shadow runs that test a candidate version on real inputs without users seeing it, and A/B tests that let real outcomes pick the winner.
What offline evals can't tell you
Offline evals are a gate. They answer "did we break anything we know about?" They can't answer four questions that only production can:
- Does it work on the real distribution? Your test set over-represents the cases you thought of. Production has its own mix, and it shifts over time as users and products change.
- What happens without an answer key? Live runs have no expected values. You need graders that judge an output on its own.
- Do users actually benefit? Passing every grader doesn't mean the reply saved anyone time. Only user behavior tells you that.
- Which of two good versions is better? When both versions pass your suite, small differences in your eval are mostly noise. Real traffic gives you enough volume to tell.
So the workflow is: offline evals and CI decide whether a version may reach users at all, then online evals decide whether it's actually better. When the two disagree, that's a gift: it shows you exactly where your eval set doesn't look like reality.