Golden Set Size Planner: How Many Examples an Eval Set Needs

The precision count. The half-width of a Wilson interval at rate p over n items is z / (1 + z²/n) times the square root of p(1-p)/n + z²/(4n²). It falls as n grows, so there is a smallest n whose half-width reaches the margin you asked for, and that is the count returned. At a 90% baseline and a 5-point margin the half-width is 0.050176 at 140 items and 0.049995 at 141, so the answer is 141.

The shortcut, shown beside it. The widely cited planning formula is n = z² p(1-p) / margin². At an 80% expected score and a 5-point margin it gives 245.85, so 246, which is the number the practical-evaluation guide states. It is printed here as the comparison and never as the answer: at this page's own defaults it returns 139 items, whose Wilson half-width is 5.04 points, over the 5 that were requested. A page that sizes with one rule and grades with another can reject the count it just prescribed.

The detection count. Against a held reference, n = (z_alpha sqrt(p₀(1-p₀)) + z_beta sqrt(p₁(1-p₁)))² / d², with p₁ = p₀ - d. At a 90% baseline, a 10-point drop, 5% significance and 80% power that is 85.4955, so 86. Against two independent arms the null variance is pooled across both, which at a 5-point drop gives 685.5969 per arm, so 686. Both are the standard handbook forms. That per-arm count is what the arms are each scored on, so the set itself holds twice it: 1,372 distinct items, which is the number the design chooser prices for the same comparison.

Slices and the correction. Each gated slice is its own test, so a Bonferroni correction divides the significance level by the number of slices. Five slices at 5% become 1% each, z rises from 1.959964 to 2.575829, and the precision count per slice rises from 141 to 244. Under equal allocation the total is the per-slice count times the slice count, so 1,220. Under traffic-share allocation the smallest slice governs: the total is the per-slice count divided by that share, so 2,440 at a 10% smallest slice. Bonferroni is conservative, and the page says so rather than hiding it.

Grading a set you already own. The number you enter is read as a total across every slice, and the smallest slice it implies is what gets graded: 695 items across five equal slices is 139 per slice. At the corrected level that slice carries a 6.66-point half-width and can catch a 9.38-point drop, so it clears a 10-point drop and misses a 5-point margin. The items to add are what the prescribed total is short by, so 1,220 - 695 = 525: what you own plus what you are told to add is the number at the top of the page. When the verdict clears, the items to add are zero.

The floor. No count below 16 items is ever returned. The best case Wilson width at n, reached only at a perfect or a zero score, is 19.4 points at 16 items and 20.4 at 15, so 16 is the smallest count that can place a rate inside a 20-point window at all. At a 90% baseline the real width at 16 items is 30.6 points. It is a floor on what is possible, never a recommendation.

Formula: n = min{ n : Wilson half-width(p, n) <= margin }

See Sources below