Why Evals Are Your Superpower
Evals turn 'it seems better' into numbers. Learn the types of evals and the minimum viable eval for any agent.
Here's a story every agent builder lives through. You tweak the system prompt to fix a complaint, try three examples, and the outputs look better. You ship it. Two days later, a different kind of request starts failing in a way it never did before, and you have no idea whether the change you made caused it, or whether it was always broken and nobody noticed.
The fix is not more careful eyeballing. It's evals: small, repeatable tests that turn "it seems better" into "it passes 46 of 50 cases, up from 41." This lesson explains what an eval actually is, the main types and when each fits, how to read results without fooling yourself, and the minimum viable eval you can set up for any agent this week. The rest of the module builds one for real.
Vibes don't scale
Checking a few outputs by hand ("vibe checking") is a fine way to start. It stops working quickly, for three reasons.
Models are non-deterministic. The same input can produce different outputs on different runs. One good result tells you the agent can succeed, not that it will. A change that looks like an improvement on three tries might be noise.
Changes have side effects. Prompts and tools interact in ways that are hard to predict. Telling an agent to be more concise can make it skip a verification step. Adding a tool can make it stop using an older one. You only find these regressions if you test cases you weren't thinking about.
Everything around the agent keeps moving. You'll swap models, edit prompts, add tools, and change retrieval settings. Without a fixed yardstick, every change is a fresh judgment call, and the decisions start to depend on who complains loudest.
An eval gives you a yardstick. It doesn't need to be big or fancy. Even a modest one changes how you work: you stop arguing about whether a change helped and start looking at which cases it fixed and which it broke.