LLM eval sample size and planning calculators
Everything here is decided before the run: whether both models see the same items, how many items the set needs, how many repeats each item gets, and what that adds up to in tokens and dollars.
What these numbers mean
Every number on this page is settled before a single case is scored. A plan that skips them still produces a result, and that result carries whatever precision the plan happened to buy, which is usually less than whoever reads it assumes. The four calculators here turn the four planning questions into counts you can defend: which design the comparison uses, how many cases the set holds, how many times each case runs, and what the whole thing costs to execute.
Work across the table in that order. The design choice comes first because it changes what every later count means. The case count and the repeat count multiply together, so neither one is readable on its own. Cost sits last because it takes both of the others as inputs, and it is the row that most often sends people back to revise the first three. One more card sits in the grid below, cross-listed from pass-rate-statistics: the live sample size and power calculator, which sizes the same run count from a classical two-arm or paired-design formula for a reader who wants that approach instead of this family’s variance split.
| What you are deciding | The instrument |
|---|---|
| Whether the two systems being compared see the same cases | settle the design before anything else |
| How many cases the set has to hold | size the set from the drop worth catching |
| How many times each case should be run | split the variance, then set the repeats |
| Cases in the pilot that were not all run the same number of times | weight the ragged pilot instead of trimming it |
| What the plan will cost in tokens and in review hours | price it before you commit to it |
| A run that already happened and a number that needs defending | take it to the after-the-run instruments |
The repeat guidance here rests on our own measurement of how far a suite’s score moves between runs that changed nothing: how many times a suite has to run before its number holds still. The wider map of eval statistics picks the thread up after the run finishes, which is exactly where this page stops.
The choice people get stuck on is whether to spend on more cases or on more repeats. More cases buy down case-to-case variance, more repeats buy down run-to-run variance, and the variance split is what tells you which of the two you are actually paying for. Budget is what turns that into a real decision rather than a preference, so price both plans before committing to either one.
Tools in this topic
Paired vs Independent: Choosing an Eval Comparison Design
Six questions give you the design, the test that fits it, the rules that fired, and what the branch you did not pick costs in cases and model runs.
Instrument | LLM eval sample size and planningGolden Set Size Planner: How Many Examples an Eval Set Needs
Size a golden set from a baseline pass rate, a margin, and the drop worth catching: items per slice, a total, and a verdict on the set you already have.
Instrument | LLM eval sample size and planningRepeated-Run Variance Planner: How Many Trials per Eval Case
Split your eval's spread into run-to-run and case-to-case variance, then read the trials per case, the case count that implies, and the protocol to freeze.
Instrument | LLM eval sample size and planningEval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
Instrument | LLM eval sample size and planningEval Budget Calculator: What an LLM Evaluation Run Will Cost
Price an eval plan before you run it: cases by variants by trials by judge calls, plus the human review hours it needs, from your own rates.
Analyses that use these calculators
- Multi-agent failures
Limitations of multi-agent debate: what the paper found
What arXiv 2509.05396 found about the limitations of multi-agent debate: accuracy fell below a first-answer vote, models flipped correct answers to agree, and a correctness reward made it worse.
- Eval statistics
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
- Reliability testing
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Reliability testing
Agentic AI testing beyond a single eval run
Single-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.
- Eval statistics
Is your eval difference statistically significant?
Two eval runs a few points apart. Separate a real gain from run-to-run noise with a paired McNemar test on the same items: a p-value and a confidence interval on the pass-rate delta.