How inflated is your eval score after reusing the same set?

The standard error. On a pass/fail set of n items with observed rate p, the sampling noise in one score is sigma = sqrt(p(1 - p) / n). This is the ordinary binomial standard error, expressed as a share of items. A set of 300 items at an 80% rate gives a standard error of about 2.3 percentage points, which means a single score on this set will typically land about 2.3 points from the true rate even with no selection at all.

Expected optimism. Score k candidates on the same set. Even if every candidate performs identically, the best-of-k score runs above the true rate by sigma times the expected maximum of k standard normals. For k = 1 that maximum is zero: one candidate, no selection, no inflation. For k = 10 it is about 1.55, so the best-of-10 score overshoots by roughly 1.55 times sigma. The calculator uses Blom's approximation to the expected maximum: m(k) = inverse-normal-CDF((k - 0.375) / (k + 0.25)), which matches the exact expected maximum to about two decimal places above k = 3. The approximation is labeled as one on the page.

The look budget. Blom's formula inverts in closed form. The largest k whose expected optimism stays within your tolerance is the look budget, which equals floor((0.375 + 0.25q) / (1 - q)), where q = normal-CDF(tolerance / sigma), clamped to at least 1. At n = 1,000, p = 0.80, and a tolerance of 2 pp the budget is 10 looks. At 5 pp on the same set the budget jumps past 1,000, because the tolerance is wide relative to the set's own noise. Past that point the page reports "more than 1,000" rather than printing the raw inverse, which steepens near vertically and implies a precision the model does not carry.

The adaptive ceiling. When you edit the system in response to what the holdout said, the selection bound becomes a floor. The worst-case bound from the adaptive data analysis literature (Dwork, Feldman, Hardt, Pitassi, Reingold and Roth, 2015) grows as sqrt(k / n). This is an order-of-magnitude ceiling. The page reports it beside the selection figures, never as the headline and never inverted into a budget. At n = 1,000 and a tolerance of 2 pp the inverse gives less than one look, which says nothing useful. The ceiling exists so a reader who admits they are tuning adaptively can see the gap between the best case and the worst case.

The corrected interval. The ordinary Wilson interval treats the observed rate as a single measurement. To correct for selection, the calculator applies a Bonferroni correction: the confidence level becomes 1 minus the error rate divided by k, so the resulting lower bound holds simultaneously over all k candidates. At large k the corrected level hits the library ceiling of 99.99% and stops tightening; the page notes this when it happens. The bias-corrected point estimate (observed rate minus the expected optimism, floored at zero) is shown separately, without an interval of its own, because shifting both Wilson endpoints down by the bias would invent coverage the construction does not have.

The fresh-set size. A reader whose budget is spent can ask how big a clean set would need to be: the fresh-set size needed equals ceil(p(1 - p)(m(k) / tolerance)^2). At p = 0.80 and a tolerance of 2 pp, a 10-look history needs about 957 items; 30 looks needs about 1,666. The number rises slowly because the fresh-set size needed grows roughly as log(k) through the m(k) squared term.

Formula: expected optimism = σ · m(k), m(k) = Φ⁻¹((k − 0.375) / (k + 0.25))

See Sources below