Eval confidence interval

An eval confidence interval is the range around an eval metric, say a pass rate, that a procedure produces to bracket the metric’s true value in a stated fraction of repeated runs of the same suite (say 95% of such intervals cover it). You estimate it from two components: a closed-form binomial interval over the finite task set (or a bootstrap when a closed form is awkward), widened by the run-to-run spread you get from re-running under independent seeds. Run the suite once and you get a point estimate with no width. The closed form assumes distinct items, not merely independent runs, so rows that repeat inside the set narrow the interval without adding any evidence to it.

A point estimate presents one draw from a distribution as a constant. Two agents scoring 82% and 79% on a single run each are statistically tied when either score swings several points on a re-run.

The variance has two sources: the finite set of tasks the suite samples, and the stochastic run itself (temperature, tool nondeterminism). Seeds expose the second; the first has a closed form. For a pass rate of 0.80 over 100 tasks, the normal-approximation 95% interval is 0.80 ± 0.078 (1.96 × sqrt(0.8 × 0.2 / 100); computed from stated assumptions, not measured), and seed variance widens it further. That normal approximation runs rough near 0 or 1 and at small task counts, where a boundary-respecting interval (Wilson or exact) is the safer default.

Three closed-form methods turn that pass count into the task-set interval, and they disagree most exactly where evals live, on small suites and rates near 0 or 1. The Wilson score interval inverts the score test, so its bounds stay inside [0, 1] and hold their coverage down to a few dozen tasks; it is the sane default and the first thing a Wilson score interval calculator should show. The normal (Wald) form, the 0.80 ± 0.078 shown above, is the quick mental estimate and the first to break near the boundaries. The Clopper-Pearson interval, the exact-binomial method a Clopper-Pearson confidence interval calculator implements, never undershoots its stated coverage, at the cost of running conservative and wide. Reporting all three side by side shows the disagreement instead of inheriting one method’s blind spot. An interval on the difference between two rates is a separate construction from an interval on either one, and when the two rates were measured on different case sets the independent two-proportion test is what reports it, alongside the test of whether the gap clears zero.

That task-set term is the shared primitive under the other eval calculators: the sample-size and power calculator inverts it to ask how many tasks buy a target width, the McNemar calculator and the reliability@k estimator build on the same interval, and the judge-bias-correction calculator widens it for a fallible judge. None of them recover the seed-to-seed spread from one run: re-run the identical suite under independent seeds and combine the seed-to-seed standard deviation of the pass rate with the task-set term in quadrature (variances add; widths do not). That second component is the piece a single score sets to zero; a defensible eval interval carries it, a textbook binomial does not.

Report the interval next to the score, as the containment rate already does and as reliability@k numbers still require. The general convention for reporting a rate with its interval lives in how to measure agent reliability. When an LLM judge produces the score, the interval also has to absorb the judge’s measured error: reporting LLM-as-a-judge evaluations coins a more specific “eval-CI” for that bias-corrected, calibration-aware interval, a specialized variant of the general interval defined here. Companion explainers cover whether an eval result is statistically significant and how many runs a reliable eval needs.