Prove your eval suite can fail

A suite that cannot go red is not evidence that the system works. A passing number and a valid number are separate claims: our map of the four ways an eval number lies shows each one corrupting a score while it still reads green. The five checks on this page each have a right answer you know before the check starts, a demo file you can download and a tool that reads it. The checks rest on our analysis of agents that report success and are wrong, which reports that a model judge rewards surface proxies such as confident closing language and sheer action volume. The check that judges the same outputs twice draws on our review of how judge biases drift.

Run these before you trust a green suite.

• Run a constant output and an empty run through every grader, and read each grader’s pass rate on the results auditor.
• For each bug you have fixed, check that it shows under “Fixed” in the “Same items, joined on id” mode of the eval A/B comparator.
• Judge one set of outputs twice and expect no clear change.
• Rerun all of them after any change to a grader, a judge or a model. See when the model changes.

A bug you already fixed should show as fixed

Keep the outputs of the version that had the bug, and whenever the suite changes, run old and new versions through today’s suite paired and check that every known fix appears under “Fixed.”

On the eval A/B comparator, choose the “Same items, joined on id” mode and press “Load the returns-desk example”, which loads the returns-desk two-version file. A working suite shows every known fix under “Fixed.” In the returns-desk example (40 cases, the overall verdict, paired), the comparator lists 6 cases as fixed. Three fixes are known: on rd-007 and rd-023 the person who reviewed the replies (the example’s human review labels) failed version 1 and passed version 2, and on rd-034 version 1 wrote the wrong label. Only rd-023 and rd-034 are on that list; rd-007 sits with the cases both versions pass. Version 1 handled a stove fire as a defective item and the person failed it, yet the overall verdict passed it because version 1 wrote the right label and the judge passed the reply. The suite cannot see that fix, so it would not catch the bug coming back. The other 4 listed as fixed are judge flips the person does not confirm, and the scorecard on judge bias correction measures how often a judge passes what a person fails.

A fix that lands where both versions pass needs a check that fails it: a judge criterion or a code check. Then check the judge on the judge validation report.

A constant output should score near the floor

Add a version that returns the same output for every case, run it through every grader and read each grader on its own.

Download the returns-desk constant-baseline file and open it in the “Same items, joined on id” mode on the eval A/B comparator. Choose each grader in turn. A working suite scores the constant version near the floor: the most common label’s share on a label check, near zero on a judge. In the returns-desk example (40 cases, comparing version 2 against the constant version), label_match passes 8 of 40 for the constant version, that floor, so that grader works. But policy_judge passes 36 of 40 for the constant version against 33 of 40 for version 2: +7.50 points, Newcombe 95% [-6.88, +22.15], two-sided, an interval that spans zero, McNemar mid-p 0.289. The judge rewards how a reply sounds. The suite as a whole still goes red only because label_match holds.

A crafted constant response reached an 86.5% length-controlled win rate on AlpacaEval 2.0 by gaming the judge.¹ A judge that scores the constant version close to the real version needs validation against people: judge bias correction and judge validation report.

A run that does nothing should fail almost every case

Run the agent with every tool call removed and an empty final message, then read each grader on its own.

Download the returns-desk null-agent file and open it in the same mode on the eval A/B comparator. A working suite goes red on a null agent. In the returns-desk example (40 cases, comparing the agent against the null version), end_state passes 12 of 40 for the null version: the 12 cases whose right outcome changes nothing. reply_judge passes 0 of 40. completed passes 40 of 40 for both, because doing nothing counts as completing. The overall verdict is 0 of 40 for the null version. Read an outcome check against its own floor: the cases where doing nothing is right, 12 of 40 here. A completed grader that cannot fail measures nothing about the task.²

A completed check that passes an empty run needs pairing with an outcome or reply check, or dropping. The same failure inside an agent is a task verification failure.

Answers moved to the wrong case should fail

Give each case the output the same version wrote for a different case, so that fluency, format and policy wording stay real and only the fit is wrong.

Download the returns-desk wrong-case-answers file and open it in the same mode on the eval A/B comparator. Choose each grader in turn. A working suite’s label check scores a shuffled version near chance: about 5 of 40 for the example’s label mix when each case takes another case’s output at random. In the returns-desk example (40 cases, each answer moved one case along, comparing version 2 against the shuffled version), label_match passes 1 of 40 for the shuffled version. policy_judge passes 28 of 40 against version 2’s 33: -12.50 points, Newcombe 95% [-29.80, +5.70], an interval that spans zero, mid-p 0.18. The judge cannot tell an answer to another customer from the right one.

Give the judge the case (ticket and reference) in its prompt, then re-validate it: judge bias correction and judge validation report.

The same outputs judged twice should read the same

Judge one set of outputs twice with the same judge configuration and compare the two sets of verdicts paired. The repeated-run variance planner measures rerun noise across repeats; here the outputs stay fixed and only the judge varies.

Download the returns-desk same-outputs-judged-twice file and open it in the same mode on the eval A/B comparator. A working suite shows no clear change. In the returns-desk example (40 cases, 32 both pass, 1 broke, 1 fixed, 6 both fail), the first file shows 0.00 points, paired Newcombe 95% [-9.91, +9.91], mid-p 1, churn 5.0% [1.4, 16.5]. The two flips are the judge’s noise, which any later difference must clear.

Now download the same outputs judged again after the judge’s alias moved. In the returns-desk example (12 broke, 1 fixed) it reads -27.50 points [-42.47, -10.63], wholly below zero, mid-p 0.002, churn 32.5% [20.1, 48.0]. The file reads like a 27.5-point regression in a system that never changed.

A clear change means the judge itself moved. For a pairwise judge, the position bias calculator reads the swap counts. Pin the judge’s model as the evals in CI guide describes, freeze its prompt, then rerun.

A wrong reading points at the judge or the check

A judge that rewards the wrong thing, or cannot tell a shuffled answer from the right one, needs validation against people. A grader that cannot fail an empty run needs pairing with an outcome check. A fix the suite cannot see needs a check that fails it. To audit the grader’s code itself, see how benchmarks get gamed. The error-analysis workbench shows the outputs one at a time so that you can record what went wrong in each.

Rerun all five after any change to a grader, a judge, a normalizer, a harness or a model. When the model changes covers the model case step by step.

Footnotes

1. Zheng et al., “Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates”, ICLR 2025, https://arxiv.org/abs/2410.07137 (as of 2026-10-17): a constant response crafted to game the judge reached an 86.5% length-controlled win rate on AlpacaEval 2.0. ↩

2. Zhu et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks”, arXiv preprint, 2025, https://arxiv.org/abs/2507.02825 (as of 2026-10-17): TAU-bench counted empty responses as successful, and flaws of this kind can under- or overestimate agent performance by up to 100% in relative terms. ↩