AI agent glossary
Online evals
Measuring an agent on real production traffic, by scoring live runs, watching user behavior, running shadow tests, and A/B testing versions.
Offline evals run on a fixed test set before you ship and tell you whether a change is safe to try. Online evals measure what actually happens with real users, whose requests you did not choose.
The main techniques are scoring a sample of live runs with your existing graders, tracking behavior signals such as whether people accept, edit, or abandon the agent's output, shadow runs that send real inputs to a candidate version without showing users its answers, and A/B tests that split traffic between versions and compare outcomes.
Online results close the loop: failures found in production become new offline eval cases, so the same mistake does not ship twice.
A new prompt passes offline evals, so the team routes 10% of traffic to it. After a week, support staff send its drafts unedited less often than the old version's, so they roll it back.
Learn more in the course
Related terms
Go from definitions to doing
Learn to use and build AI agents in 130 practical lessons.