Topic
Eval statistics
The statistics under an eval result: sample size, confidence intervals, the tests that separate a real difference from noise, and what an eval can certify.
Research
- Analysis
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
- Study
Claude Fable 5 vs Opus 5 vs Opus 4.8 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, Claude Opus 5 and Claude Opus 4.8 on identical tasks, seven areas scored, every count and caveat published.
- Study
Model routing and the refusal tax: a pre-registered study
On short, checkable tasks we measured no Opus 4.8-to-Fable 5 capability separation at either effort we ran, and the premium tier's refusal 'rescue' silently served the cheaper model on 20 of 28 calls.
- Analysis
Is your eval difference statistically significant?
Two eval runs a few points apart. Separate a real gain from run-to-run noise with a paired McNemar test on the same items: a p-value and a confidence interval on the pass-rate delta.
- Analysis
How many runs a reliable eval needs to catch a regression
How many runs a reliable eval needs is a power calculation set by the regression you must catch, your target power, and the baseline pass rate. Includes a runs-needed table and the formula behind it.
- Analysis
AI agent evaluation that follows the whole trajectory
AI agent evaluation breaks when it scores the final answer and skips the path. Evaluate the trajectory, catch early-step corruption, and report pass rates with intervals.
- Analysis
What LLM evals are, and what each type can certify
LLM eval covers four instruments: offline benchmark, LLM-as-judge, human, and online, each answering a different question, plus the benchmark-vs-product line and the rigor behind a trustworthy score.
- Analysis
LLM evals: which methods to trust and where they lie
LLM evals report whether a model passed. Whether that score is valid is a separate question. This hub maps the eval methods and the four ways an eval number lies, each routed to its fix.
Guides
- Guide
OpenAI Evals is winding down. The alternatives skip the statistics.
OpenAI is deprecating its hosted Evals platform and steering users to Promptfoo, which it now owns. What OpenAI Evals, DeepEval, Ragas, TruLens, and Promptfoo each do, and what porting costs you.
- Guide
The prompt wording is a hyperparameter you never swept.
Rewording the same task swings a model's pass rate: format, option order, even a 'please'. A one-phrasing eval samples one point from a spread you never measured. Pin the prompt and measure it.
Terms
-
Answer coverage
Answer coverage is answers returned over requests sent. Publishing it beside any rate computed on those answers lets a reader see how much of the intended sample the rate actually rests on.
-
Bootstrap resampling (eval intervals)
Bootstrap resampling estimates the uncertainty of an eval statistic by resampling the scored runs with replacement, recomputing the statistic on each draw, and reading the spread of those values as its sampling distribution. It supplies an interval where no closed-form standard error exists.
-
Capability tier (model routing)
Capability tier is the band a router sorts a model into, ordered by how much task competence its vendor claims it delivers. The ordering is published as a product hierarchy, so whether a given boundary changes your results is a question only a paired eval on your own tasks can settle.
-
Construct validity (benchmarks)
Construct validity is the degree to which a benchmark measures the specific capability it claims rather than a proxy a system can score high on without having it; a benchmark is construct-valid only when its top score cannot be earned without the capability it advertises.
-
Coverage conditioning
Coverage conditioning is the dependence of a published rate on which requests came back with an answer, and it bites when membership of that answered subset correlates with the property the rate is meant to measure.
-
Effect size (eval deltas)
Effect size is the magnitude of a difference between two eval results, measured on a scale that holds still when the run count changes: on a pass/fail suite, the gap between two pass rates in percentage points, reported with an interval on the delta itself.
-
Eval confidence interval
An eval confidence interval is the range a procedure produces that, across repeated runs of a suite, brackets a metric's true value a stated fraction of the time (say 95%); its width combines a task-set term (closed-form binomial, or bootstrap) with the seed-to-seed spread, which one run omits.
-
Eval reproducibility
Eval reproducibility is getting the same result from an evaluation re-run on the same data and the same parameters; it breaks when uncontrolled non-determinism such as sampling temperature, an unpinned seed, or a drifting judge model moves the score while the declared inputs stay fixed.
-
Fallback chain (model routing)
A fallback chain is the ordered list of models a router tries for a single request, moving to the next entry each time the one before it declines or fails, and ending at the first model that returns an answer or at the end of the list.
-
Model router
A model router is the component that decides which model handles each incoming request, choosing once per request and before the call is dispatched, so one application can spread its traffic across an expensive tier and a cheap one.
-
Refusal rate (LLM models)
Refusal rate is the share of requests a model declines to answer on policy grounds, measured over requests sent rather than answers returned. A refusal arrives as a normal response with stop_reason set to refusal, so it never touches an error rate.
-
Statistical power (eval design)
Statistical power is the probability that an eval reports a significant difference when a regression of a stated size is genuinely present, settled before the run by the drop you would act on, the item count, the score variance, and the false-positive rate.
-
Variance decomposition (eval runs)
Variance decomposition splits the spread in an eval score into the sources that produced it: which tasks the suite happened to contain, how the model sampled tokens on each attempt, which judge scored the output, and what the harness held fixed between runs.
Calculators
- Calculator
reliability@k and pass^k Estimator
Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t intervals and the unbiased pass@k counterpart.
- Calculator
Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
Turn an eval pass rate (k of n) into a defensible confidence interval: Wilson score, Clopper-Pearson exact, and Wald normal bounds side by side.
- Calculator
Eval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
- Calculator
McNemar Test Calculator for Paired Eval Runs
Run McNemar's paired significance test on two eval runs scored on the same items, with exact and chi-square p-values and a CI on the pass-rate difference.