Variance decomposition (eval runs)

Variance decomposition is the practice of attributing the run-to-run spread in an eval score to the separate facets of the design that produced it, so that one number for uncertainty becomes a list of named contributors you can act on one at a time. It answers a different question from the interval around an eval score, which reports how wide the uncertainty is; the decomposition reports where that width came from.

Components add on the variance scale rather than the scale you read the score in. A facet holding a quarter of the total variance supplies half of the standard deviation, since the square root of 0.25 is 0.5, so a share that looks small in the table is larger in the margin you publish. Each component is itself an estimate, drawn from however many levels the design contained: five seeds, two judges, one prompt template. Report every share with its level count, since a judge term built on two judges has very little underneath it.

A decomposition is only as good as the design feeding it, and eval designs lose cells. Refusals, timeouts and blocks delete observations in a pattern that tracks difficulty, which conditions every rate computed on the remainder, and a facet with no observations left contributes no estimated variance and plenty of real uncertainty. The evidence this page rests on for that is our own benchmark, where the provider blocked every request Claude Opus 5 was sent on one task. All 10 of those calls fell on the one task deep enough to separate it from Claude Fable 5, so its index rests on 77.3% of the instrument weight, and the unscored remainder admits any index from 71.6 to 94.3.

Say what the harness pinned as well as what it sampled, because a facet you held fixed contributes zero to the estimate and an unknown amount to the truth.

Separating the four variance components of an eval run

Vary one facet at a time while holding the others fixed, then compare the spread within a facet against the spread across it. Repeat every task at your production sampling settings and the spread among those repeats is the model’s own sampling term. Score two disjoint halves of the suite separately and the gap between them is the task-sampling term. Re-score one frozen set of outputs with a second judge, or the same judge under a swapped presentation order, and the movement is the judge term. Pin the seed and the decoding parameters, re-run, and whatever still moves is the residual the harness contributes, which is the territory eval reproducibility covers.

The classical name for this design is generalizability theory, set out by Cronbach, Gleser, Nanda and Rajaratnam in 1972, which replaces a single error term with a separately estimated component for every facet of a measurement. Estimating those components does not require closed-form mean squares. Resampling whole runs gives each component an interval, which is what bootstrap resampling is for on a statistic with no tidy formula.

Instrument it where the harness already writes rows: one per task, per repeat, per judge, per seed, with the configuration stamped on the row. The reliability@k and pass^k estimator does the two-facet version, and its suite-level t interval captures spread across tasks, treating each task’s pass^k as one observed value, so sampling noise inside a task sits outside it. More runs per task shrink the inside term and leave the across-task term alone.

Seed variance and task variance have both been measured

Two of the four facets are characterized in published work. Madaan and colleagues measured seed variance across initializations on open models and on models pretrained from scratch, and found that reframing a multiple-choice task such as MMLU as a completion task often reduces variance at around the 7B scale, while methods borrowed from the human testing literature struggled to move it (arXiv preprint, June 2024). Miller’s treatment of eval questions as a sample from a larger population gives the task facet an error bar of its own and formulas for planning the run in advance (arXiv preprint, November 2024). Neither gives the judge facet the same treatment, and a judge’s agreement, calibration and bias each want their own test before its scores enter a decomposition.

Variance decomposition vs eval reproducibility

Eval reproducibility is a property of a re-run: the same harness, the same data and the same parameters land on the same result, in the computational sense the National Academies uses. Variance decomposition describes structure, naming how much of the spread each facet supplies. Either one can hold while the other fails. A suite can be perfectly reproducible and still hold most of its variance in the task facet, because pinning a seed says nothing about how a different fifty tasks would have scored. A suite can be irreproducible while its decomposition stays stable, when the seed term is large but consistent, so every re-run lands somewhere new and the shape of the variance holds. Report the two separately: reproducibility says whether a re-run lands where the last one did, and the decomposition says which facet you would have to change to move the number at all.

Variance decomposition vs a confidence interval

A confidence interval is an output, one range from a procedure that brackets the true value a stated fraction of the time, and a pass-rate interval calculator will hand you one from a single k-of-n result without asking where the variation came from. Variance decomposition is the structure underneath that width. They diverge when the interval looks acceptable and the structure is wrong, which happens when a narrow interval was bought by re-running one small suite many times: the width shrinks while the task facet it under-counts stays where it was. They diverge the other way when a wide interval is exactly right, because the design sampled tasks broadly and the width reports the truth about a heterogeneous suite. The interval goes in the report, and the decomposition tells you whether narrowing it would mean anything.

Where the ANOVA and finance senses stop applying

Variance decomposition is an established phrase in two adjacent fields, and both meanings are legitimate. In classical analysis of variance it partitions a total sum of squares into between-group and within-group pieces for a designed experiment with balanced cells and independent errors. In portfolio analytics it splits return variance into systematic factor exposures and an idiosyncratic remainder. An eval run breaks the first set of assumptions in ways that are easy to miss: the same tasks are re-measured under every condition, so the errors are correlated by construction; the judge is often nested inside the prompt template rather than crossed with it; and refusals, blocks and timeouts empty cells in a pattern tied to task difficulty, which is why the report has to state the base a published rate actually rests on.

The premise to refuse is the request for one variance number for the eval. A single figure blends facets you can fix cheaply, such as running each task more times, with facets that need a new instrument, such as writing more tasks or replacing the judge. A decomposition worth reporting names every facet and says which ones were held fixed rather than sampled, so a reader can see the uncertainty the estimate never had access to.

Both numbers belong in the same eval report, and they answer to different readers: the interval to anyone deciding whether to trust a delta, the decomposition to whoever has to make the next run better. Sizing the run comes first, because which facets you can afford to sample is settled before any of them is estimated, and the runs a regression of a given size demands and the probability your design would catch it settle that between them. Afterwards, read a measured delta against the spread you decomposed instead of against zero, which is what a paired significance test does once you know how large the difference is. Seed variance turns up again on the other side of that work, as one of four measurable ways an eval number lies in the map of LLM evals and where they mislead, and the rest of the machinery is indexed in the eval statistics this all sits inside. Write the facet list into the same report as the number, since a variance share with no level count behind it can’t be checked.