Benchmark Rank Uncertainty Calculator: Score Intervals and Rank Ranges

A leaderboard prints one number per system and orders them. This calculator attaches to each number the uncertainty it carries, then asks two questions of the ordering: which neighboring pairs are statistically separated, and how far each system's rank could plausibly move.

Score intervals. When a row is given as correct-of-total counts, or as a score with an item count, the score is a proportion of a fixed item set and its interval is the Wilson score interval at the chosen level. Wilson is used because the textbook normal interval collapses to zero width at 0% and 100% and misbehaves at small item counts, which is exactly where leaderboard extremes live. A published score times its item count is not always a whole number, because leaderboards round; in that case the same closed form is evaluated at the fractional proportion rather than snapping your score to a nearby count, and the result notes say so. When a row is given as a score with a standard error, the interval is score plus or minus z times the standard error, which suits metrics that are not a plain accuracy: averaged rubric scores, aggregate indices, or a benchmark's own published error bar.

Separated or not. For each pair of nominally adjacent systems the tool reports the gap between them with a confidence interval, and calls the pair separated when that interval excludes zero. For proportions the interval is Newcombe's hybrid score interval, assembled from the two Wilson bounds and the standard recommendation for a difference between independent proportions; for standard-error rows it is the normal difference interval, which is the same formula for symmetric bounds. This is a sharper test than the eyeball rule that two error bars must not overlap: two 95% intervals can overlap while the difference between them is still significant at 95%, because non-overlap of two intervals is closer to a 99% test. A pair the tool calls separated can therefore still show overlapping bars in a chart.

Tiers. Walking down the ranking, a new tier starts at every separated adjacent pair, so the tier count is one plus the number of separated pairs, and it is the headline because it is the honest summary of how much of the printed order the data supports. A tier is a run of neighbors none of which is separated from the next; two systems at opposite ends of a long tier can still be separated from each other, and the plausible-rank column shows that finer structure. No multiplicity adjustment is applied across the pairs: at 95%, one in twenty truly equal adjacent pairs would be called separated by chance, so a leaderboard with many adjacent tests carries that risk in proportion.

Plausible ranks. Rank uncertainty is simulated. Each system's plausible true score is drawn many times from a distribution matching its precision: for proportions, the Jeffreys posterior, a Beta distribution with parameters k plus one half and n minus k plus one half, the Bayesian interval Brown, Cai and DasGupta recommend alongside Wilson; it agrees closely with the Wilson bound at moderate item counts, drifts from it at very small ones, and stays non-degenerate at 0% and 100%. For standard-error rows, a normal distribution with the stated standard error. Every draw is ranked, draws that tie share the better rank, and the tabulated ranks give each system its probability of finishing first and a central rank range at the chosen level, read as the ranks this system holds in the central 95% of plausible worlds. The draws come from a seeded generator, so the same rows, seed, and draw count reproduce every figure exactly; a different seed moves the probabilities by ordinary Monte Carlo error, which more draws shrink.

Worked example. The preloaded leaderboard has eight systems on a 500-item benchmark, spread from 62.6% to 84.2%. Every score carries a Wilson interval between six and eight and a half points wide (the leader's is 80.7% to 87.1%), and only two of the seven adjacent gaps are separated at 95%: model-e over model-f (6.8 points, interval +1.4 to +12.2) and model-g over model-h (7.6 points, +1.7 to +13.4). The top five systems form one tier, even though the 5.8-point gap between model-a and model-d is itself separated, because no single adjacent step inside that tier clears zero. The nominal leader finishes first in about 64% of simulated draws at the default seed and its 95% rank range is 1 to 3. On this benchmark, first place is a plurality of the plausible orderings rather than a settled result.

Honest limits. Every figure treats systems as independent estimates on a fixed item set. Public leaderboards score every system on the same items, so their errors are positively correlated, and a paired comparison on the item-level results is sharper than the unpaired bounds shown here; when you have both systems' per-item outcomes, the McNemar test calculator is the right instrument for one pair, and the intervals here are the conservative reading. Item counts describe how precisely a score is measured on this item set, and say nothing about whether the item set is contaminated, saturated, or representative of your task; those questions are covered in the evals hub and the four instruments of LLM evaluation. Two audited public leaderboards, worked through in the significance walkthrough, show how much of a printed ordering can fail these checks in practice.

Formula: score CI = Wilson(k, n); gap CI = Newcombe hybrid score; ranks = draws from Beta(k + 1/2, n - k + 1/2)

See Sources below