INSTRUMENT · EVAL
Eval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
Showing your last valid result. Update the inputs above to recompute.
Runs per arm
686
1,372 total across two fresh arms to detect a 5-point drop from a baseline of 90% at 80% power (two-sided alpha 0.05).
Paired (McNemar) · matched items
0
Runs by design
The default sizes two fresh arms; use the fixed-baseline count only when the baseline is a large, trusted reference you are not re-running.
| Design | Runs needed | What it assumes |
|---|---|---|
| Two fresh arms (pooled) | 686 per arm · 1,372 total | Candidate and baseline both re-run; neither treated as fixed. |
| Two fresh arms, continuity-corrected | 726 per arm · 1,452 total | Fleiss (Casagrande, Pike, Smith) correction for the discrete test. |
| Fixed baseline (one-sample) | 316 runs | Baseline is a large, trusted historical reference held constant. |
Runs per arm by drop size
Halving the drop you want to catch roughly quadruples the runs, because the count scales with the inverse square of the effect size.
| Drop (pts) | Runs per arm | Total runs | With continuity |
|---|---|---|---|
| 1 | 14,751 | 29,502 | 14,951 |
| 2 | 3,841 | 7,682 | 3,941 |
| 3 | 1,774 | 3,548 | 1,841 |
| 5 | 686 | 1,372 | 726 |
| 10 | 199 | 398 | 219 |
| 15 | 100 | 200 | 113 |
| 20 | 62 | 124 | 72 |
n per arm = (z(1-alpha/2)*sqrt(2*pbar*(1-pbar)) + z(power)*sqrt(p0*(1-p0)+p1*(1-p1)))^2 / delta^2How?
How this is calculated
The default design compares two builds run fresh: a candidate against a
baseline, both measured with the same harness. Sizing it is a
two-proportion power calculation. Write p0 for the baseline
pass rate, p1 = p0 - delta for the degraded rate you want to
flag, and pbar = (p0 + p1) / 2 for the pooled proportion under
the null. With z(1-alpha/2) the two-sided significance
quantile and z(power) the power quantile, the runs needed per
arm are the formula above, the uncorrected two-sample approximation given
by Casagrande, Pike and Smith (1978). The result is rounded up.
Worked example (the defaults). Baseline 90%, drop 5
points, so p0 = 0.90, p1 = 0.85,
delta = 0.05, pbar = 0.875. Two-sided alpha 0.05
gives z = 1.95996; power 80% gives z = 0.84162.
The null spread is sqrt(2 * 0.875 * 0.125) = 0.46771 and the
alternative spread is sqrt(0.9*0.1 + 0.85*0.15) = 0.46637. So
n = (1.95996*0.46771 + 0.84162*0.46637)^2 / 0.05^2 = 685.6,
which rounds up to 686 runs per arm, or 1,372 total.
Fixed baseline (one-sample). When the baseline is a large,
trusted historical reference held constant, only the candidate arm carries
noise, the single-proportion form in the NIST/SEMATECH e-Handbook 7.2.4.2:
n = (z(1-alpha/2)*sqrt(p0*(1-p0)) + z(power)*sqrt(p1*(1-p1)))^2
/ delta^2. On the same inputs that is
(1.95996*0.3 + 0.84162*0.35707)^2 / 0.0025 = 315.8, or
316 runs. It is cheaper because it does not pay for a
second noisy arm.
Continuity correction. The Fleiss
(Casagrande, Pike and Smith 1978) adjustment nudges the count up to keep
the discrete pass/fail test at its stated false-alarm rate:
n' = (n/4)*(1 + sqrt(1 + 4/(n*delta)))^2. On the worked
example that turns 686 into 726 per arm.
Paired (McNemar) design. If the same items are scored
under both conditions, size the number of matched items with Connor's 1987
normal approximation. With psi the discordant proportion
(items that flip) and delta the marginal difference:
pairs = (z(1-alpha/2)*sqrt(psi) + z(power)*sqrt(psi - delta^2))^2 /
delta^2. At a 20% discordance rate on the worked example that is
626 matched items. The estimate needs
psi >= delta; below that it is dropped rather than shown as
a nonsensical count.
Assumptions. All four designs use the normal approximation, which holds when the counts are moderate and the pass rate is not near 0% or 100%; near the ceiling, prefer an exact interval. Runs are assumed independent, one minimum detectable drop is chosen up front, and the two-arm test uses the pooled variance under the null. The default test here is two-sided. The companion analysis at how many runs a reliable eval needs tables the one-sided fixed-baseline case, whose counts run lower for the same inputs, and explains why a single green re-run answers a different question than "would this eval catch the regression you care about."
Formula: n per arm = (z(1-alpha/2)*sqrt(2*pbar*(1-pbar)) + z(power)*sqrt(p0*(1-p0)+p1*(1-p1)))^2 / delta^2
Questions
How many eval runs do I need to catch a regression?
Enough to see the smallest drop you would roll back a release for, at the power you want. Fix three inputs, the baseline pass rate, the drop to detect, and the target power, and the runs needed is determined. Catching a 5-point drop from a 90% baseline at 80% power takes 686 runs per arm on the two-proportion design, or 316 against a fixed baseline. Sub-point drops need four-figure counts and are effectively out of reach on a normal CI budget.
Why does the count balloon for small drops?
Runs scale with the inverse square of the drop you want to catch. Halving the effect size roughly quadruples the runs. Raising power from 80% to 90% adds about a third more runs across the board. The table above makes both curves concrete for a 90% baseline.
Should I use the two-arm or the fixed-baseline number?
Use two arms when you re-run the baseline fresh each time, so both sides carry sampling noise. Use the fixed-baseline count only when the baseline is a large, trusted historical reference you hold constant. The two-arm count is roughly twice the fixed-baseline count because it pays for a second noisy arm.
When is a paired design worth it?
When you can score the same items under both conditions. A paired McNemar test spends only on the items that flip pass or fail, so it usually needs far fewer items than two independent arms. It requires a discordance rate at least as large as the drop you are testing for.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.2 Sample sizes required
- Casagrande, Pike & Smith (1978), An improved approximate formula for calculating sample sizes for comparing two binomial distributions, Biometrics 34(3):483-486
- Connor (1987), Sample size for testing differences in proportions for the paired-sample design, Biometrics 43(1):207-211
- Wichura (1988), Algorithm AS 241: The Percentage Points of the Normal Distribution, Applied Statistics 37(3):477-484
- Miller (2024), Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations