Statistical power (eval design)

Statistical power is the probability that an eval will report a significant difference when a regression of a stated size is genuinely present in the system under test. It is a property of the design rather than of any result, settled before the first call goes out by four choices: the smallest pass-rate drop you would act on, how many items the eval scores, how much those scores bounce between runs, and the false-positive rate you agree to test at. A p-value is what the run gives back. Power is what the run was capable of giving back.

Power is a conditional probability, and the condition is a specific effect size. One suite is powered differently against a 10-point drop and against a 2-point drop, and an item count that catches the first nine times in ten can catch the second barely more often than a coin flip. So a power figure travels with its effect size, its item count and its threshold, the way an interval carries its coverage level. A bare claim of 80% power describes no eval in particular.

Two of our own runs supply the evidence this page is built on. The reliability benchmark across Anthropic’s Fable 5, Opus 5 and Opus 4.8 fixed a 2.0-point win threshold before any call went out, and when the top two came back at 93.8 and 92.7 it published no order inside the tie, because the gap never cleared the size the design had committed to. The second of those two figures was computed over 77.3% of the index, and on the full index the run can place it only between 71.6 and 94.3, which is the wider reason no order was published. The pre-registered routing study set its separation bar in advance at 15 percentage points with disjoint intervals, cleared it at neither of the two effort levels it tested on machine-checkable work, and published the outcome as inconclusive. Naming the detectable size first is what makes a null readable afterwards, and the number of runs behind a reliable eval works that decision end to end.

Sizing an eval for the effect you would act on

Start from the rollback decision, because the effect size is the one input the data can’t hand you. Name the smallest pass-rate drop that would make you hold a release, then take the baseline pass rate, the item count, and the false-positive rate you will test at. Power is the share of hypothetical repeats of that eval, run against a system worse by exactly that much, in which the test would clear the threshold. The arithmetic usually runs backwards in practice, since the item count is the part you choose: fix power where you want it and read off the items it demands, which is what the sample size and power calculator does across two-arm, fixed-baseline and paired McNemar designs. Run it when the eval set gets fixed, alongside the pre-flight checks that gate a release.

Evan Miller’s Adding Error Bars to Evals (arXiv preprint, November 2024) writes that formula in the form eval work needs, carrying the paired variance term through. His worked example uses variances he labels fictional but reasonable, and the count it lands on is large: detecting an absolute difference of 3 percentage points 80% of the time at a 5% false-positive rate takes about 969 independent questions, which is why the paper recommends new evals carry at least 1,000.

One lever here belongs to model evals specifically. Because a nondeterministic system can be asked the same question more than once, sampling K answers per question shrinks the per-question variance term without commissioning a new item. In Miller’s second worked example, again on stated schematic variances, raising K from 1 to 10 across 198 questions moves the minimum detectable effect from 13.2 to 7.5 percentage points. Report the design with the number, since one eval at 80% power with K set to 10 is a different instrument from the same eval at K set to 1. An unpinned seed or a drifting judge pushes that variance term the other way, which puts reproducibility inside the sizing calculation.

Size against the denominator you will actually score, too: refusals and timeouts pull items out of a run, so the share that came back answered is the count the power was really bought with.

Statistical power vs sample size

Sample size is a count of how many items, runs or paired comparisons the eval scores. Power is a probability read off that count together with the effect size, the score variance and the threshold, so a count quoted on its own settles nothing.

Each of the two moves without dragging the other with it. Adding items raises power only when the new items can discriminate: in a paired design scored with McNemar’s test on the disagreeing pairs, only the items where the two systems differ carry information, so a thousand tasks both systems always pass leave the effective count where it was. Clustered items do it more quietly, since several questions drawn from one document share their noise and the nominal n grows while the effective n barely moves. Power also rises at a fixed item count, through pairing the comparison, resampling answers per item, and cutting judge noise. Quote the count and the power side by side, so that a reader can see which one did the work, and read the drop-by-baseline arithmetic off the runs-needed table.

Statistical power vs effect size

Effect size is the magnitude of a difference on a scale that outlives the sample it came from: a 7-point gap in pass rate, a standardized mean difference, a ratio. Power is the probability of detecting a difference of a magnitude you name in advance. One formula produces both, which is where the confusion starts, and they face opposite ways: effect size asks how big, power asks how likely you are to see it.

An underpowered run can still report a large effect, and it tends to. Low power filters which results reach significance, so the ones that get through are the draws where noise ran with the difference, and the published gap sits above the truth. The reverse case is quieter: at a large enough item count a difference of half a point clears the threshold while changing nothing about whether you ship. Pre-commit to the magnitude that would change your decision, and hold the design to it. The magnitude side is worked on effect size in eval deltas.

Observed power cannot rescue a null eval result

After an eval comes back with no significant difference, the tempting move is to compute power from the effect that was observed and quote it as evidence the eval was adequate. Hoenig and Heisey named this the abuse of power in The American Statistician in 2001, and their objection is arithmetic: the observed significance level of a test also determines the observed power, so for any test the observed power is a 1:1 function of the p value. It restates the result in a second notation and carries no news.

The replacement is a confidence interval on the difference. An interval running from a 1-point drop to a 1-point gain rules out a large regression; one running from a 12-point drop to a 10-point gain rules out very little, and it says so plainly instead of dressing a shrug as a calculation. Testing whether a delta is real works the after-the-run side, interval included.

The 80% default was set for a different cost structure

The 0.80 figure arrives from experimental design in the behavioral sciences with a rationale attached. Jacob Cohen fixed it as the working level in his power primer in Psychological Bulletin in 1992, calling it “a convention proposed for general use”: a materially smaller value would incur too great a risk of a Type II error, and a materially larger one would demand a sample beyond the investigator’s resources. Taken with the conventional 5% threshold, he notes, it puts the two kinds of risk in a 4:1 ratio, treating a false positive as four times as serious as a missed effect.

An eval sitting on a release gate keeps a different ledger. Re-running a suite costs compute and a delay; shipping a regression the eval was too small to see costs whatever it costs in production, for as long as it takes someone to notice. Where the miss is dearer than the false alarm, inheriting the 4:1 ratio is a choice nobody made. Pick the pair deliberately: state the false-positive rate and the power you settled on, then size to that.

Between them, power and effect size decide whether an eval could ever have answered the question it was built for. The eval-statistics index holds the rest of that machinery. The wider map of what an eval number certifies, and the four measurable ways one lies, sits on LLM evals. Size is the part of that map to settle first, because a suite nobody sized against a named drop won’t report the regression it was too small to see.