Evals
Systematic tests for AI systems — fixed inputs, graded outputs — that measure whether a change made things better or worse. The safety net under everything else in this glossary.
01What it means
Evals (evaluations) are to AI systems what test suites are to software: fixed scenarios run against the system, with outputs graded against expected behavior, so that a change — new model, new prompt, new context source — can be judged better-or-worse instead of vibes-or-anecdotes. Because model output is sampled rather than retrieved, the same prompt in fresh conversations can produce different answers; a citation that appears in one run and not another deserves suspicion. Single trials decide nothing. Evals, plural, do.
The other half of the case is drift. Context changes silently — a data source shifts format, a tool's response changes, a policy is updated upstream — and the system degrades without any deploy to blame. Evaluations are the tripwire: when context changes, evals are what catch it. Context engineering without evals is just hope with extra steps.
A practitioner pattern worth stealing for getting started: instead of hand-building a test suite, have the model interview you about how the application should behave, then scaffold the test cases and grading from your answers. You keep the judgment; it does the typing.
02Why it matters for buyers
- —It is how claims get proven: "the new model is better" is answerable with evals and unanswerable without them. Any AI vendor should be able to say what their evals showed before shipping a change.
- —It guards regressions: agents and prompts drift as the systems around them change — evals are the mechanism that notices.
- —It is cheap to start: a handful of golden scenarios with known right answers beats a public benchmark you do not control.
03Common questions
What is the difference between evals and benchmarks?
Benchmarks are public, generic, and optimized to be gamed; evals are yours — your tasks, your data, your grading. A good benchmark score says a model is capable; a good eval run says your system still works.
How many test cases do we need to start?
Fewer than you think: the golden handful — the recurring scenarios your team already knows the right answer to — outperforms a large suite of unexamined cases. Grow from what breaks.
Can a model grade its own outputs?
Model-graded evals are common and useful for scale, but calibrate the grader against human judgment on a sample first — the grader is part of the system under test, not an oracle.