Bootstrap resampling (eval intervals)

Bootstrap resampling estimates the uncertainty of an eval statistic by drawing repeated samples with replacement from the runs you already scored, recomputing the statistic on each draw, and reading the spread of those recomputed values as its sampling distribution. It exists for the statistics algebra will not cover: a weighted composite index, a median latency, an F1 or BLEU score, a ratio of two rates. What comes out is an eval number that carries its own uncertainty like any other, and that entry defines what such an interval claims; this one covers the machinery behind it.

This page’s worked example comes from our cross-vendor reliability benchmark on three frontier models. Its headline figure is a weighted mean of eight per-dimension scores at pre-registered weights, a compound statistic with no closed-form standard error, so the scorecard of record publishes every score with a 90% bootstrap interval: Claude Fable 5 at 66.9 [63.4, 70.4], Kimi K3 at 63.4 [59.2, 67.2], GPT-5.6 Sol at 50.9 [47.9, 54.0]. The per-dimension rows show what that width buys. Fable 5 scores 91.7 [75.0, 100.0] on consistency, and that interval covers both rivals’ scores on the same row, so the quotable point estimate settles no ranking at all.

Two different errors live inside a bootstrap interval and they behave differently. Sampling error comes from the runs you actually scored, and no number of resamples reduces it, since resampling invents no new evidence. Monte Carlo error is the wobble from having drawn a finite number of resamples rather than every possible one, and that one does shrink as the count rises. So a bootstrap figure gets reported with its resample count and its interval method attached. Two teams quoting a 95% bootstrap interval on identical data can print different bounds, depending on whether they took plain percentiles or the bias-corrected and accelerated form Efron published in 1987.

The bootstrap sits one layer beneath the metrics themselves, as machinery several of them borrow. Suite-level reliability@k averages a per-task proportion across tasks, the compound shape that resists a closed form. The flagship profiler is designed to resample its injected-fault trials and report a containment rate with a bootstrap interval on it, so the estimate arrives carrying its own width. The profiler has not shipped, so the containment case is still ahead of us, and the bootstrap intervals we publish today are the per-dimension ones on the benchmark above. Reproducibility cuts the other way, since a bootstrap consumes pseudo-random numbers: the recomputation audit on that benchmark flagged its RNG as seed-pinned across all dimensions, which is why the intervals reproduce byte-identically only from the exact committed verdict files.

Pin the seed and record it, or your interval is unreproducible by construction.

How to calculate a bootstrap interval

Treat the scored runs as the population, draw a resample of the same size with replacement, recompute the statistic on that resample, and repeat some thousands of times; then sort the recomputed values and read the 2.5th and 97.5th percentiles for a 95% percentile interval. How many resamples to draw is a compute decision more than a statistical one, and open-source practice sets it high. EleutherAI’s lm-evaluation-harness defaults to 100,000 iterations and routes median, F1, Matthews correlation, perplexity, BLEU, chrF and TER through them, with a comment in its own source conceding that the bootstrap standard error it returns is biased low, equivalent to dropping Bessel’s correction from the standard deviation. Plain accuracy takes a closed-form standard error instead, and that one does apply the correction. Report the count, the interval type, and the unit you resampled.

That last one does the most damage when it goes wrong. Resampling individual rows assumes the rows are independent, and eval items routinely are not: several questions cut from one document, several seeds of one task, several turns of one conversation. Adding Error Bars to Evals, Evan Miller’s 2024 arXiv paper on eval statistics, reports a worked case where clustered standard errors run over three times the naive ones. When items arrive in groups, resample the group.

Instrument at the scored-run record, keeping the per-item outcome and its cluster key, because an averaged aggregate cannot be resampled afterwards. For a plain k-of-n pass rate none of this is needed, and the pass-rate confidence interval calculator returns Wilson and Clopper-Pearson bounds from the two counts.

Bootstrap resampling vs a closed-form binomial interval

A closed-form binomial interval derives its bounds algebraically from a pass count and a trial count, on the assumption that each trial is an independent Bernoulli draw: Wilson inverts the score test, Clopper-Pearson inverts the exact binomial. Bootstrap resampling assumes no distribution at all, and pays for that freedom with compute and with total dependence on the sample it was handed.

Which of the two wins depends entirely on the statistic in front of you. On a plain pass rate the closed form wins outright: it is deterministic, cheap, reproducible without a seed, and it holds its coverage at small trial counts where the bootstrap degrades badly. A suite of 20 runs with 20 passes hands every resample the same 20 passes, and the percentile interval collapses to [1.0, 1.0]. On a weighted composite, a median, or a ratio of two rates, no closed form exists, and resampling is what supplies a width at all. Miller’s paper draws the boundary in one sentence: “we regard bootstrapping as unnecessary unless a complicated sampling scheme or estimator is being used.” Use the closed form where it applies and the bootstrap where it does not, then name which one produced the number you published.

Bootstrap resampling vs a permutation test

A permutation test shuffles the labels that separate two conditions, recomputes the test statistic on each shuffle, and counts how often a shuffled value matches or beats the observed one; that count estimates a p-value under the null that the label carries no information. Bootstrap resampling holds the labels fixed and resamples the observations instead, describing how far the statistic would move under repeated sampling from the same source.

They answer different questions, and a serious report usually wants both. A permutation test says whether a delta could plausibly be noise and nothing about its size; a bootstrap gives you the size with a width and tests no hypothesis. Resampling reached language-model evaluation through this door: Philipp Koehn’s EMNLP 2004 paper on significance tests for machine translation worked it on BLEU and reported that test sets of only 300 sentences could give assurance a difference was real. For pass/fail labels scored on the same items across two runs, McNemar’s exact test is sharper, and our walkthrough of whether an eval delta is real runs it beside the paired McNemar calculator. A report quoting only the interval has said how big the gap is without saying whether it survives a shuffle.

Bootstrapping in machine learning means something different

The word holds two other jobs in the machine-learning literature, and neither of them produces an interval on a measurement. Bootstrap aggregating, the bagging Leo Breiman introduced in Machine Learning in 1996, resamples a training set to fit many predictors and average them: variance reduction for a model. In reinforcement learning, bootstrapping names updating a value estimate from another estimate instead of from a completed return.

So “we bootstrapped the interval” identifies very little on its own. A defensible sentence names the statistic resampled, the unit resampled, how many resamples were drawn, and which interval type was read off them. Answer coverage, the answers returned against the requests sent, belongs there too whenever refusals or timeouts removed runs before scoring, since a bootstrap faithfully reproduces whatever denominator you hand it.

Between them the bootstrap and the closed form cover only the task-set half of an eval’s uncertainty, and the seed-to-seed half needs variance decomposition across the eval’s facets before either width reads as complete. Working out the eval sample size a given regression demands is the arithmetic that runs ahead of the eval, and the wider vocabulary these terms sit inside is collected in the statistics an eval report has to carry. If you can name the statistic, the unit and the resample count, then another team can rebuild your width from the same scored runs; short of all three, the width is decoration.