Regression Testing and Evals in CI
Run your eval suite on every prompt or model change, set pass thresholds, and catch regressions before users do.
You now have two eval suites: the triage eval with code graders and judges, and the agent eval that grades whole trajectories. They only protect you if they run. And the moment they matter most, a teammate's "tiny wording fix" to the system prompt on a Friday afternoon, is exactly when nobody remembers to run them.
Continuous integration (CI) fixes that by running your evals automatically on every change and blocking the merge when quality drops. This lesson turns your harness into a regression gate: a threshold file that says what "good enough" means, a check script that compares each run against those thresholds and a committed baseline, and a GitHub Actions workflow that runs everything on pull requests and every night. You'll also learn how to set thresholds that catch real regressions without crying wolf on random noise.
What a regression gate needs
A unit test passes or fails the same way every time. An eval doesn't. Run the same suite twice and the pass rate might move a few points with no change at all. So an eval gate can't be "everything must pass." It needs four pieces:
- A trigger. Run on pull requests that touch prompts, tools, cases, or graders; run nightly to catch drift you didn't cause (a dependency update, a change in an API your tools call); and run on demand before a release.
- Thresholds. Written-down minimums: overall pass rate, per-grader rates, and cases that must pass every time.
- A baseline. The summary of a run you've reviewed and accepted, committed to the repository, so each new run can say what got better and what got worse.
- A report. A short summary in the pull request's checks, plus the full results as a downloadable artifact, so a red build tells you why within a minute.
You already have the hardest part: a harness that saves every run as JSON. Everything in this lesson reads those files.