Eval Sample Size and Power Calculator

The default design compares two builds run fresh: a candidate against a baseline, both measured with the same harness. Sizing it is a two-proportion power calculation. Write p0 for the baseline pass rate, p1 = p0 - delta for the degraded rate you want to flag, and pbar = (p0 + p1) / 2 for the pooled proportion under the null. With z(1-alpha/2) the two-sided significance quantile and z(power) the power quantile, the runs needed per arm are the formula above, the uncorrected two-sample approximation given by Casagrande, Pike and Smith (1978). The result is rounded up.

Worked example (the defaults). Baseline 90%, drop 5 points, so p0 = 0.90, p1 = 0.85, delta = 0.05, pbar = 0.875. Two-sided alpha 0.05 gives z = 1.95996; power 80% gives z = 0.84162. The null spread is sqrt(2 * 0.875 * 0.125) = 0.46771 and the alternative spread is sqrt(0.9*0.1 + 0.85*0.15) = 0.46637. So n = (1.95996*0.46771 + 0.84162*0.46637)^2 / 0.05^2 = 685.6, which rounds up to 686 runs per arm, or 1,372 total.

Fixed baseline (one-sample). When the baseline is a large, trusted historical reference held constant, only the candidate arm carries noise, the single-proportion form in the NIST/SEMATECH e-Handbook 7.2.4.2: n = (z(1-alpha/2)*sqrt(p0*(1-p0)) + z(power)*sqrt(p1*(1-p1)))^2 / delta^2. On the same inputs that is (1.95996*0.3 + 0.84162*0.35707)^2 / 0.0025 = 315.8, or 316 runs. It is cheaper because it does not pay for a second noisy arm.

Continuity correction. The Fleiss (Casagrande, Pike and Smith 1978) adjustment nudges the count up to keep the discrete pass/fail test at its stated false-alarm rate: n' = (n/4)*(1 + sqrt(1 + 4/(n*delta)))^2. On the worked example that turns 686 into 726 per arm.

Paired (McNemar) design. If the same items are scored under both conditions, size the number of matched items with Connor's 1987 normal approximation. With psi the discordant proportion (items that flip) and delta the marginal difference: pairs = (z(1-alpha/2)*sqrt(psi) + z(power)*sqrt(psi - delta^2))^2 / delta^2. At a 20% discordance rate on the worked example that is 626 matched items. The estimate needs psi >= delta; below that it is dropped rather than shown as a nonsensical count.

Assumptions. All four designs use the normal approximation, which holds when the counts are moderate and the pass rate is not near 0% or 100%; near the ceiling, prefer an exact interval. Runs are assumed independent, one minimum detectable drop is chosen up front, and the two-arm test uses the pooled variance under the null. The default test here is two-sided. The companion analysis at how many runs a reliable eval needs tables the one-sided fixed-baseline case, whose counts run lower for the same inputs, and explains why a single green re-run answers a different question than "would this eval catch the regression you care about."

The detectable change at a given number of items is the Connor 1987 formula inverted by bisection. With z(1-alpha/2) (or z(1-alpha) one-sided) and z(power) the normal quantiles and psi the discordance, the Connor count at a change d is n(d) = (z(1-alpha/2)*sqrt(psi) + z(power)*sqrt(psi - d^2))^2 / d^2, the paired formula above. The detectable change is the d in (0, psi] for which n(d) equals your item count. The range caps at psi because a paired change larger than the discordance needs more one-way flips than there are flips. The shortcut form (z(1-alpha/2) + z(power))*sqrt(psi / n) is printed beside it. On the returns-desk example (40 items, 22.5% discordance), the Connor formula gives a detectable change of 20.4 points and the shortcut gives 21.0.

When runs per side K exceeds 1, the formula follows Miller Equation 9. Each version's run-to-run variance is half the rerun disagreement, so together they equal the rerun disagreement, and the version-to-version part is the discordance minus the rerun disagreement, floored at zero. The planning variance is V(K) = (psi - rerun) + rerun / K, where rerun is the rerun disagreement, and the detectable change is (z(1-alpha/2) + z(power))*sqrt(V(K) / n). The K = 1 row prints the Connor count and the Equation 9 count side by side. On the example at K = 3 with 11.5% rerun disagreement, the detectable change drops to 17.1 points.

A judge with Youden J below 1 attenuates the observed change. The true detectable change is the observed figure divided by J, and the item count scales by at least about 1 / J squared. When the judge J differs between the two systems, the page prints both values and warns that one J cannot correct the comparison.

The unpaired contrast shows what the same budget detects on different item sets: the two-arm detectable change at n items per arm at the stated baseline. On the example at a baseline of 70%, the unpaired figure is 30.7 points, compared with 20.4 paired.

A discordance that cannot occur at the stated baseline is refused: the share of items flipping one way cannot exceed the baseline passing or failing share. The smallest set that can detect any paired change at a given discordance is the count where the Connor formula returns the discordance itself as the detectable change; below it, no paired change is detectable. On the example, that minimum is 33 items.

Value of the next block. The mode asks whether a fixed block of extra runs is worth its price. It treats the block as fresh evidence sized on the entered run count alone; everything the reader already knows goes into the prior belief. Power at the entered block size is the standard two-proportion or fixed-baseline calculation inverted: instead of solving for the count at a target power, the mode computes the power at a given count.

The decision has two possible errors, each with a cost the reader enters. Shipping a real regression costs the entered regression stake; holding a release that was fine costs the entered hold stake. At any belief that a regression is present, the expected loss of the better action is the smaller of the two products: the belief times the regression stake, or one minus the belief times the hold stake. Without running the block, decide now and accept that loss. With the block, the expected loss falls to a weighted mix of the loss after a flag and the loss after a clear, where the flag probability and the two posteriors come from Bayes' rule on the block's power and the entered prior. The value of the block is the difference: the expected reduction in loss from having the evidence before deciding.

Under the null hypothesis (no real difference), a two-sided test at level alpha reports a drop with probability alpha/2. The mode uses alpha/2 as the null flag rate and alpha/2 as the uninformative threshold. A block whose power does not exceed that threshold cannot separate signal from the test's own false-flag rate.

When the better action is the same at the prior and at both posteriors, the value is exactly zero at this block size, because no outcome the block can produce changes anything. A larger block raises the power, which moves the posteriors, and can reach a belief where the better action flips.

No confidence interval appears on this mode's figures, and none is owed. Nothing here is estimated from data: the prior is entered by the reader, power is computed from entered rates, and both stakes and the cost are entered.

Formula: n per arm = (z(1-alpha/2)*sqrt(2*pbar*(1-pbar)) + z(power)*sqrt(p0*(1-p0)+p1*(1-p1)))^2 / delta^2

See Sources below