Human Review and Annotation Workflows
Build the human side of evaluation: labeling guidelines, review queues, inter-rater agreement, and turning reviews into better evals.
Every automated grader you've built so far rests on human judgment. The expected categories in cases.jsonl came from someone deciding what was right. The judge was calibrated against labels you wrote by hand. The thresholds in CI encode what a person decided was good enough. When that human judgment is sloppy, rushed, or inconsistent, everything built on top of it inherits the problem, and it does so with the confidence of a number.
Most teams treat human review as an afterthought: someone skims a few outputs when they have a spare hour. This lesson makes it a deliberate system. You'll write labeling guidelines that two people can apply the same way, build a review queue that puts the right outputs in front of reviewers, measure whether your reviewers actually agree, and turn every review into a better eval.
Where humans fit in an eval system
Humans are expensive graders, so use them where nothing else works. In a mature eval system, people do four jobs:
| Job | What reviewers do | What it feeds |
|---|---|---|
| Define truth | Write expected answers for test cases and settle policy questions | cases.jsonl |
| Calibrate judges | Label outputs blind so you can measure the LLM judge | labels.jsonl and calibrate.ts |
| Audit | Review a sample of outputs the automated graders passed | New cases, judge fixes |
| Adjudicate | Resolve disagreements between reviewers, or between a reviewer and a grader | Guidelines, rubrics |
Notice what's missing: grading every output. If humans are reviewing everything, either your volume is tiny (fine, for now) or your automated graders aren't good enough yet and the review time would be better spent fixing them.
Who reviews matters as much as how. The people who built the agent know what it's supposed to do but are biased toward seeing it succeed. Domain experts, like the support leads who answer these emails today, know what a good reply looks like but may not know the policy edge cases you've decided. The best setups use both: builders write the guidelines with experts, and experts do most of the labeling.