LLM-as-Judge: Grading at Scale
Use models to grade open-ended outputs with rubrics — and calibrate the judge so you can trust its scores.
Code graders are fast, free, and perfectly consistent, but they can only check what you can define precisely. Was the reply actually helpful? Did the summary leave out the one point that mattered? Did the agent promise a refund in words your regex never anticipated? For questions like these, you either read every output yourself, which doesn't scale past a few dozen, or you ask a model to grade.
That second option, LLM-as-judge, is how most teams grade open-ended agent output at scale. It's also easy to do badly: an uncalibrated judge produces confident scores that don't match what a careful human would say, and you end up optimizing for the judge's quirks. In this lesson you'll write rubrics that judges apply consistently, build a judge that plugs into your eval harness, and calibrate it against your own labels so you know how far to trust it.
When you need a judge (and when you don't)
Reach for a model judge when the quality you care about is real but hard to pin down in code:
- Faithfulness: does the answer stick to the provided documents or tool results, without adding facts?
- Completeness: did it address every part of a multi-part request?
- Policy compliance in any wording: no promises, no medical advice, no blame.
- Tone and clarity for a particular audience.
- Comparisons: is version B of the prompt producing better answers than version A?
Don't use a judge for things code can check: exact labels, extracted values, JSON validity, word counts, whether tests pass. A judge for those is slower, costs money, and is less reliable than a one-line comparison. In a typical eval, most graders are code and a few targeted judges cover the rest.