Cases at that trial count
401
INSTRUMENT | Eval statistics
Repeated-Run Variance Planner: How Many Trials per Eval Case
3 cited sources
Split your eval's spread into run-to-run and case-to-case variance, then read the trials per case, the case count that implies, and the protocol to freeze.
A score that moves between runs holds two spreads: the bounce within one case, and the real gap between cases. Repeating a case shrinks only the first.
It measures the split from a pilot and sizes the runs, resting on how many reruns a suite needs and on what the two components mean. Then read reliability with the pass^k estimator.
Cases with different trial counts are weighted here rather than refused, and the page names the weighting.
Showing your last valid result. Update the inputs above to recompute.
Trials per case
3
Cheapest plan for detecting a 5.0-point drop at 20% missed-detection risk. 401 cases at 3 trials on each of 2 versions is 2,406 runs. Fewest runs is always 1 trial per case, which answers a different question.
Total runs
2,406
Case-count floor
65
Repeats stop paying at
141
Where the spread comes from
| Part of the spread | Variance | Repeating cases shrinks it |
|---|---|---|
| Run to run, within one case, baseline arm | 0.1667 | Yes, by the trial count |
| Case to case | 0.09119 | No |
| Case-to-case share of a single run | 35.4% | No |
| Difference between the two versions, measured | 0.0206 | No |
Trials against cases, at the same detectable drop
| Trials per case | Planning variance | Cases | Total runs | Cost, n (R + aK), run-equivalents |
|---|---|---|---|---|
| 1 | 0.3414 | 1,072 | 2,144 | 3,216 |
| 2 | 0.181 | 569 | 2,276 | 2,845 |
| 3 · recommended | 0.1275 | 401 | 2,406 | 2,807 |
| 4 | 0.1008 | 317 | 2,536 | 2,853 |
| 5 | 0.08476 | 267 | 2,670 | 2,937 |
| 8 | 0.0607 | 191 | 3,056 | 3,247 |
| 10 | 0.05268 | 166 | 3,320 | 3,486 |
| 20 | 0.03664 | 116 | 4,640 | 4,756 |
The case counts above follow from the trials per case you just chose. How many cases a set needs in its own right is a different question, and sizing a golden set is where it is answered. Whether the two systems even see the same cases is a design choice that comes before either. To put a price on this plan, carry the trials figure and the number of versions into the budget calculator, which costs a run in tokens and dollars. After the runs, read the rate with the pass-rate interval calculator, and size a single-run comparison with the sample size and power calculator this page extends.
The run protocol
What to freeze, what to resample, and what to report
Filled from the plan above on 2026-08-31. Temperature 0 is not determinism: identical requests can take different reduction orders depending on what else is in the batch, so a protocol that freezes decoding alone still moves. Freeze the serving side too, and resample only what you mean to average over.
| What the run fixes | Record this | Frozen or resampled |
|---|---|---|
| Model snapshot | Exact version or checkpoint string | Frozen |
| Serving stack | Inference server and kernel versions, hardware | Frozen |
| Batch size and concurrency | The values in force during the run | Frozen |
| Temperature and decoding | Temperature, top-p, max tokens, stop rules | Frozen |
| Seed | One seed per trial, all of them written down | Resampled per trial |
| Case order | The order cases ran in | Resampled per trial |
| Judge prompt version | Judge prompt hash and judge model version | Frozen |
| Judge repeats | How many times the judge scored each item, and the agreement between those runs | Frozen |
| Retry policy | Whether retries are on, and whether a retry counts as a trial | Frozen |
| Isolation | That each trial ran with no state carried from the last one | Frozen |
| Trials per case and cases | K = 3, n = 401 | Frozen |
| Significance and power | Two-sided alpha 0.05, power 0.8 | Frozen |
| Report with the result | Trials per case, cases, run-to-run variance, case-to-case variance, and the case-to-case share | Reported |
n = (z_alpha + z_beta)^2 (V + U/K) / d^2How?
How this is calculated
The two variances. Score one case k times and its pass rate is q. Across cases, the spread of q has two sources: real difficulty differences between cases, and the run-to-run noise inside each case. Write them w2 and s2. From a balanced pilot, s2 is the mean of k/(k-1) q(1-q) over the cases and w2 is the sample variance of q minus s2/k. That subtraction is what makes it an estimate of the real spread rather than the observed one. When it lands below zero on a small pilot, the page clamps it to 0 and says so instead of printing a negative variance.
From summary numbers instead. For a pass/fail score the two parts add up to p(1-p) exactly, so one observed number fixes both: if whole- suite repeats bounce with standard deviation sd across n cases, then s2 = n sd2 and w2 = p(1-p) - s2. That conversion holds only when each repeat ran every case exactly once. Independent per-case noise cannot bounce a suite mean by more than sqrt(p(1-p)/n); a larger spread has a correlated cause, and the page stops rather than splitting numbers the model cannot produce.
What K buys. The variance a plan has to fight is V + U/K, where K is trials per case. U is the part repeating divides down and V is the part it cannot touch. Against a fixed reference, V is w2 and U is s2. Against a second system run on the same cases, V is the variance of the per-case difference, which the pilot measures directly, and U is the within-case terms of both arms added. Cases follow from Miller Equation 9: n = (zalpha + zbeta)2 (V + U/K) / d2, with zalpha = 1.959964 and zbeta = 0.841621 at the defaults. Dropping K out of it leaves the case-count floor: the number no amount of repeating can lower.
Choosing K. More trials per case cost runs and save cases; one new case costs R runs to write. Total price is proportional to n (R + aK) for a arms, and that is smallest near K = sqrt((U / aV) R). The page prices both whole numbers either side of it, because rounding up alone can land on the dearer one. Separately, the point where the run-to-run term falls to t of the planning variance is (U/V)(1-t)/t; past there, more repeats barely move the standard error.
Worked example, the seeded view. Eight cases, ten trials each, on two versions. The baseline arm gives s2 = 0.1667 and w2 = 0.09119, so 35.4% of a single run's variance is real case-to-case difference. The per-case differences have variance 0.05268; taking out the measurement noise the ten trials leave inside them gives a difference variance of 0.02059, far below the two arms case-to-case variances added together, which is the whole point of running both versions on the same cases. With U = 0.3208 and V = 0.02059 the cost optimum sits at 2.79 trials, and 3 prices cheaper than 2. Detecting a 5-point drop then needs 401 cases, 2,406 runs in total, against a floor of 65 cases that no repeating can beat.
Worked example, summary numbers. A 70% baseline over 100 cases, bouncing 4 points between whole-suite repeats, splits into s2 = 0.16 and w2 = 0.05, a 23.8% case-to-case share. Against a fixed reference that gives 2 trials per case, 409 cases and 818 runs, with the floor at 157. Five repeats measure a spread loosely, though: one standard error either side runs from 2.59 to 5.41 points, and at the top of that band the spread reaches the correlated-cause ceiling, where the trials figure has no upper bound. It would take 25 whole-suite repeats to pin the spread under that ceiling.
Formula: n = (z_alpha + z_beta)^2 (V + U/K) / d^2
Questions
Why is the answer a small number of trials?
Because trials and cases buy the same thing and cases usually buy more of it. Repeating a case only shrinks the noise inside that case; it does nothing about the fact that a suite of 50 cases samples a narrow slice of the work. Once the run-to-run term is small next to the part repeating cannot touch, further trials are close to free of effect, which is the figure shown beside the floor. Small trial counts with more cases is the usual shape of a cheap plan.
My rows have different trial counts. Can this page read them?
Yes, once you turn on "Cases were run different numbers of times" above the pilot table. With it off the estimator assumes a balanced pilot: one trial count shared by every row. Run over ragged rows it still returns a number, and the number is wrong in a direction that depends on which cases got the extra trials, so the page refuses rather than answering confidently. With the checkbox on, a weighted moment estimator reads the rows and the page names the weighting it used. Trimming every row to your lowest trial count, or summarizing the run and using the summary mode, are still the two ways to get an answer without a weighting.
The two weightings give different answers. Which one is right?
What each one does. Weighting every case equally averages the per-case variances flat and takes out the noise each case's own trial count leaves: vi = ni/(ni-1) qi(1-qi), s2 = mean(vi), and w2 = var(qi) minus mean(vi/ni). Weighting by trial count is the one-way random-effects ANOVA split instead: the within-case mean square is s2, and the between-case mean square is turned into a variance by dividing by a constant g = nbar(1 - cv2/(a-1)), where nbar is the mean trial count, cv the coefficient of variation of the trial counts and a the number of cases. Because cv2 is never negative, g is never above nbar: on that branch raggedness can only cost precision, and g is exactly how much it cost. The constant is the one Stata's loneway Methods and formulas publishes for this design, read 2026-08-30; the form above is algebraically identical and says in one line why raggedness costs precision.
Why they can disagree. On a balanced pilot they are the same estimator and return the same numbers, which is what the checkbox shows you if you turn it on over an even table. They part when the trial counts carry information. A trial count can depend on how hard a case turned out to be: you kept rerunning the ones that kept failing. That dependence makes the trial-weighted answer let those cases speak loudest, and a pilot stopped early on the hard cases is not missing at random. Neither estimator can repair that, and nothing on this page can detect it from pass counts alone. Weighting every case equally is the default here because it matches the population the plan draws from: every case gets the same trial count going forward.
Under the paired comparator. Pairing here is at the case level, not the run level. The difference variance takes the per-case difference in pass rates and subtracts the run-to-run noise both versions leave inside it, which assumes the two arms' run-level noise is independent given the case. A harness that pairs runs by seed leaves a positive covariance that marginal pass counts cannot recover; the subtraction then takes out too much, the difference variance prints too small, and the plan under-sizes the study. The ANOVA machinery does not extend to a difference at all, so under this comparator the plan is sized from the case-weighted branch whichever weighting you pick.
Effective trials is not effective cases. The effective-trials row answers how many trials per case a balanced pilot would have needed to leave the noise this ragged one leaves. That is a trial count per case, not a case count: an effective sample size of the n/deff kind counts cases, and the two figures are not comparable.
Is setting temperature to 0 enough to make runs repeat?
No. Identical requests at temperature 0 can still produce different text, because the reduction order inside the inference kernels depends on how requests are batched together, and batching depends on what else is in flight. A run that pins only decoding still moves. That is why the protocol block asks for batch size, concurrency and the serving versions, and treats seeds and case order as the things you deliberately resample.
The page says my spread has a correlated cause. What now?
A suite mean built from independent per-case noise cannot bounce further than sqrt(p(1-p)/n). A larger bounce means whole runs moved together: a serving or model change between repeats, a judge that drifted, a shared resource, or an ordering effect. Repeating cases does not shrink that, so there is no trials figure to give. Freeze the stack, rerun the suite, and read the spread again.
Does the trials figure replace a sample size?
No. It sets how many times each case runs. The case counts here are what that trial count implies for the drop you asked to detect, at your significance and power. Sizing a set on its own terms, with slices and coverage in the picture, is a separate calculation, and pricing the whole plan is another one again.
Do retries count as trials?
No, and mixing them biases the pass rate upward. A trial is a fresh, isolated attempt scheduled by the plan. A retry is an extra attempt after a failure, so counting retries as trials quietly reports the best of several attempts as if it were a typical one. If the system under test retries internally, that is part of the system and belongs in the protocol as a frozen setting.