Per-run spend, PR tier
$0.80
TOOL | Evaluation design
How long your eval suite adds to CI, and what a PR subset detects
Enter cases, concurrency and cadence across three tiers. The calculator reports wall-clock minutes under parallelism, monthly eval spend, and how much detection the per-PR subset gives up.
The power block inverts the sizing formula from our eval sample-sizing research to report what each tier can still detect, built on the evidence in the pre-ship testing checklist. The eval budget calculator prices one run; this page prices a cadence and shows what statistical significance costs in CI minutes. The sample-size power calculator sizes one arm; this page inverts that formula for each tier. Where each check belongs in the pipeline is the companion CI guide.
Showing your last valid result. Update the inputs above to recompute.
Wall-clock minutes, per-PR check
2.6
PR tier, concurrency-bound: per-run minutes under the binding constraint, plus fixed CI overhead.
Per-run minutes under the binding constraint, plus fixed CI overhead. A lower bound: no queueing, no tail latency, no scheduler wait.
| Tier | Cases | Items | Minutes | Bottleneck |
|---|---|---|---|---|
| PR | 40 | 40 | 2.6 | concurrency |
| Nightly | 400 | 400 | 7.8 | concurrency |
| Release gate | 400 | 1,200 | 19.5 | concurrency |
Developer wait (min/month)
310.0
The monthly total is the sum across three tiers, each multiplied by its own cadence.
| Tier | Runs/month | Minutes/month | Spend/month |
|---|---|---|---|
| PR | 120 | 310.0 | $96.00 |
| Nightly | 30 | 235.0 | $240.00 |
| Release gate | 4 | 78.0 | $96.00 |
| All tiers | 154 | 623.0 | $432.00 |
The power figure uses distinct cases, not graded items. Repeats reduce run-to-run variance; the repeated-run variance planner covers that.
| Tier | Cases | Detectable drop (pp) | Power at your drop |
|---|---|---|---|
| PR | 40 | 15.1 | 22% |
| Nightly | 400 | 4.4 | 88% |
| Release gate | 400 | 4.4 | 88% |
Cases for target power
316 cases
The PR subset size that would reach the target power at the drop entered, and the minutes that subset would add: 4.0 on top of the 2.6 the PR check runs today, for 6.6 in total.
These figures are a worked illustration at round defaults. Replace every cell with your own numbers.
minutes = overhead + max(items · latency / concurrency, calls · 60 / rate limit, one item) / 60How?
How this is calculated
The wall-clock estimate is a lower bound. It divides the total latency-seconds of work by the concurrency, compares that to the rate-limit ceiling, and takes whichever is larger. When both are smaller than the time for a single item, that single item is the floor. The number includes no queueing variance, no tail latency, no retries and no scheduler wait for a runner. A real run will be at least this long and usually longer.
Each item carries a system call and zero or more judge calls. The total latency for a tier is the item count times the sum of the system latency and the judge latency per call. Dividing by concurrency gives one bound; dividing the total API calls into the rate-limit window gives another. The bottleneck label beside the minutes figure names which bound won.
The money block multiplies items by the cost per item, adds runner-time cost when a runner rate is entered, and then multiplies each tier's per-run spend by its monthly cadence. The per-item cost is one number carried from the eval budget calculator, which breaks it down by tokens, model and judge. This page does not re-derive that arithmetic.
The power block counts distinct cases, not graded items. Repeats of one case are correlated, and the power math assumes independent observations. Wiring cases times trials into the sample size would inflate every power figure on the page. What repeats buy instead is variance reduction, which the repeated-run variance planner owns.
For each tier, the calculator inverts the standard one-proportion sizing formula to report the power that the tier's case count achieves at the entered drop and significance level. The inversion is exact: it solves the published sizing equation for the z-score on the power side and converts that to a probability through the normal CDF. The fixed-baseline design compares one arm against a recorded rate; the two-arm design compares two fresh runs, which needs a pooled variance and a larger sample for the same power.
The detectable drop is the complement: the largest effect each tier can catch at the target power. When the PR tier runs fewer cases than the suite, the page reports how much power the subset gave up and the probability that a real drop of the entered size passes the PR check unseen. The "cases for target power" line inverts the same formula the other way, telling the reader how large a PR subset would need to be to reach the power target, and how many minutes that subset would add.
Every rate on this page is typed by the reader, so no confidence interval applies. A confidence interval on the baseline belongs on the pass-rate CI calculator.
Formula: minutes = overhead + max(items · latency / concurrency, calls · 60 / rate limit, one item) / 60
Questions
Why is this different from the eval budget calculator?
The eval budget calculator prices one eval run from token counts and model rates. This page prices a cadence: three tiers of runs across a month, with wall-clock minutes under your concurrency and the detection power the per-PR tier gives up by running a subset. If you need the per-item cost this page asks for, start there.
Why does the power block ignore my trials?
Repeats of the same case are correlated. The power math assumes independent observations, so it counts distinct cases only. Trials help with variance across repeated runs of the same item; the repeated-run variance planner is the page for that question.
The PR tier shows less than 50% power. Should I run more cases?
Not necessarily. A per-PR tier at low power is still useful as a breakage check: it catches a prompt template that broke, a parser that stopped working, or an output that is no longer valid. Catching a 3-point quality regression takes a much larger set, and that is what the nightly and release tiers are for. The companion CI guide walks through which checks belong at which tier.
What does the bottleneck label mean?
It names the constraint that set the wall-clock floor. Concurrency means more parallel workers would help. Rate limit means more workers would not, because the provider cap already binds. Single-item means the tier has so few items that concurrency is already more than the work; only per-call latency can shorten it.
How accurate is the minutes estimate?
It is a lower bound. Real runs add queueing delays, tail latency, retries, warm-up time and scheduler waits that this page does not model. Plan for the number to be optimistic. The change-impact worksheet can narrow which suites a specific change needs to rerun, reducing the subset to fit the PR budget.