The eval loop for an AI feature

Evaluation is framed once, then run as a loop that starts by reading real outputs and starts again whenever the model, the prompt or the traffic changes, or an incident happens. Each step produces an artifact the next step reads. The loop rests on what each kind of evaluation can and cannot certify and on the four measurable ways an eval number can mislead.

Start here.

• Settle the evaluation tier and the bar once, before results exist: pick the tier, then freeze the acceptance criteria.
• Start the loop by reading real outputs: open the error-analysis workbench.
• When the model changes, follow the model-change playbook, then read fresh outputs.

Framing happens once

The quality owner names what good means, the costlier mistake and the floor. The builder says what can be measured. Together they produce an evaluation tier and a frozen set of acceptance criteria before any result exists. Decide the evaluation tier, then freeze the criteria with the sample sizes each rate threshold needs.

Each step has a page that does the work

1. Read outputs. The quality owner and the builder pull a random sample and read together, one verdict per trace. The error-analysis workbench runs the session; your first error-analysis session walks through the method.
2. Write criteria. Grading outputs changes what people count as good, so criteria come after reading¹. The criteria and rubric builder structures each criterion with examples.
3. Label a sample of cases. A domain expert labels under the criteria. The labeling handbook builder structures the task; inter-rater reliability measures agreement between two labelers, and multi-rater agreement among three or more.
4. Build the test set. The quality owner confirms the cases look like what real users send; the builder splits and deduplicates. The eval-set report checks for leakage; the golden-set size planner says how many cases a claim needs.
5. Choose graders and validate the model judge. The quality owner writes the pass definition; the builder writes the judge prompt. The evaluator and metric navigator picks the grader type; judge validation measures where the judge disagrees with your experts, and judge bias correction corrects the pass rate for it.
6. Run. The quality owner writes down the decision, the bar and the deadline; the builder sets repeats and wires the pipeline. The results importer reads files into a common shape; the CI eval gate contract carries the bar into CI; evals in CI covers pull-request, nightly and release runs.
7. Audit the results file. The eval results auditor checks denominators, rows that ended without a verdict and duplicate rows. Five known-answer checks test whether your graders can fail a broken system.
8. Read the results. Each rate carries uncertainty from the cases, the reruns and the grader, so the quality owner reads the interval and the verdict; the builder computes intervals and paired differences. The eval A/B comparator compares two runs case by case and says whether the difference is bigger than run-to-run noise; the before-you-quote checklist lists what to verify before citing a number.
9. Decide and write it up. The release gate designer gives the verdict; the eval readout builder assembles the numbers; the claim-evidence matrix names what was not tested.
10. Operate. The quality owner sets thresholds; the builder samples production and turns failures into new cases. The monitoring plan builder records the review cadence; the thumbs-down rate page bounds the rate when most conversations go unrated.

A model, prompt or traffic change, or an incident, restarts the loop

• Any model change (a retirement, upgrade, alias move, reroute, host change or vendor switch): follow the model-change playbook and the model upgrade checklist.
• A prompt change: rerun the eval and compare on the eval A/B comparator.
• A traffic shift: read a fresh sample on the error-analysis workbench.
• An incident: open the reproducibility checklist validator and choose “Capture an incident”.

For agents, four practices change; the agent evals guide covers them.