Evals for AI products — why you build the test before the feature

For an AI product, the evaluation suite is not quality assurance you bolt on at the end. It is the product specification, the regression safety net, and a real part of the moat. For an AI-first product especially, you build the evals before — or alongside — the feature, not after it. This is one of the genuine differences between building AI products and building conventional software.

Why AI products need evals more than other software

Conventional software is deterministic. Given the same input, it produces the same output, and a test either passes or fails. You can look at a feature and, with enough care, know it is correct.

An AI product is not like that. The core behaves probabilistically: the same input can produce different outputs, and “different” is not the same as “wrong”. You cannot tell by looking whether an AI feature is good — “it seems fine on the few examples I tried” is not a measurement, it is a hope. The only honest way to know whether an AI product works is to measure it systematically against a body of real cases. That measurement is what an eval suite is, and it is why AI products need evals in a way ordinary software simply does not.

What an eval suite actually is

An eval suite has four parts. A golden dataset: a curated set of real tasks the product must handle, each with a known-good answer or a clear definition of what a good answer looks like. A scoring method: a way to judge whether the AI’s output is acceptable — often a judge model, sometimes an exact check, sometimes a human rubric. A score: the percentage of the golden set the product handles acceptably. And a gate: a rule in continuous integration that blocks a release if the score drops below a floor. Together, these turn AI quality from a feeling into a number.

Evals as the product specification

Here is the shift that matters. For an AI product, writing the eval set is writing the spec. When you sit down and assemble two hundred real tasks with their known-good answers, you are not preparing to test the product — you are defining, concretely and unambiguously, what the product is supposed to do. Every hard case you include is a decision about scope. Every known-good answer is a decision about quality.

This is why the eval set comes first. A vague product description — “it summarises documents well” — is not buildable, because “well” is undefined. Two hundred documents with their ideal summaries and a target score is buildable, because now “well” has a meaning everyone can see. The eval set is the most precise product spec an AI product can have.

Evals as the regression net

AI products are unusually easy to break invisibly. A prompt tweak to fix one case quietly degrades ten others. A model provider updates their model and behaviour shifts overnight. A new feature changes the context the core AI sees. In conventional software a regression usually throws an error; in an AI product it just makes the output a little worse, and nobody notices until customers do.

The eval suite is the net. Run it in continuous integration, and every prompt change, every model swap, every refactor is scored against the whole golden set before it can ship. A change that improves one case but drops the overall score is caught automatically. Without this, an AI product slowly and silently gets worse; with it, quality only moves in the direction you choose.

Building the eval set

Start small and real. Twenty genuine cases with carefully decided answers are worth more than two hundred invented ones, because invented cases test an imagined product and real ones test the actual job. Grow the set continuously: every time the product fails in production, that failure — with its correct answer — becomes a new eval case, so the same mistake can never ship twice. Over time the golden set becomes a precise, hard-won record of everything the product must get right. Keep it honest, keep it real, and never let it drift away from what users actually send. A practical rhythm is a short weekly review: triage the production failures from the past week, decide the correct answer for each, and add them as new cases. It takes about an hour, and it is reliably the most valuable hour an AI product team spends — because it is the mechanism by which the product’s quality compounds instead of stalling.

The judge model, and its limits

Much eval scoring is done by a judge model — using a language model to decide whether an output meets the standard. It is a powerful technique and it scales, but it is not free of judgment. A judge model can be wrong, can be lenient, can drift. The discipline is to validate the judge against human ratings periodically, to keep its scoring criteria sharp and specific, and to treat a judge-model score as a strong signal rather than an absolute truth. Used carefully, it is what makes evaluating thousands of cases practical.

Evals and the moat

There is a strategic point that founders miss. A genuinely good eval suite — built from real cases, grown from real failures, encoding a real understanding of what quality means in your domain — is itself a competitive advantage. A rival can call the same model you do. They cannot quickly reproduce two years of accumulated golden cases and the quality bar that comes with them. The eval set, and the proprietary data and domain knowledge baked into it, is one of the few things in an AI-first product that a competitor cannot simply rent. Treating evaluation as core product work, not as testing overhead, is how an AI product compounds a lead instead of staying a wrapper.

Common questions