Golden dataset

A golden dataset is the answer key for your evals. Each case contains an input your system will see in practice and what a good output looks like: an exact answer, the facts it must include, the tool it should call, or a reference response for comparison.

The best cases come from real usage: actual user questions, past failures, and tricky edge cases. Every case should be checked by someone who knows the domain, because a wrong reference answer quietly teaches you the wrong lesson.

Keep the dataset versioned and growing. Add a case whenever you find a new failure in production. A few dozen well-chosen cases are enough to start, and they are far more useful than a large set nobody has reviewed.