AI agent glossary
LLM-as-judge
Using a language model to grade the outputs of another model or agent against a rubric, so quality can be measured at scale.
Many agent outputs, like summaries, support replies, or research reports, cannot be checked with an exact match. Having people grade every one is slow and expensive. LLM-as-judge uses a model to do the grading, guided by a clear rubric.
Judges work best with specific, narrow questions. 'Does the reply mention the refund deadline? Yes or no' is far more reliable than 'rate this reply from 1 to 10'. Asking the judge to explain its reasoning before giving a verdict also helps.
A judge is itself a model that can be wrong or biased, for example toward longer answers. Check it against human grades on a sample before trusting it, and recheck when you change the judge's prompt or model.
For each research report, a judge model answers three yes or no questions: Is every claim cited? Do the citations support the claims? Does the report answer the original question?
Learn more in the course
Related terms
Go from definitions to doing
Learn to use and build AI agents in 65 practical lessons.