Evidence and reproducibility calculators
What has to be written down for a number to be checkable later: the card behind a result, the manifest behind a re-run, and the claim each piece of evidence is actually evidence for.
What these numbers mean
An eval number that nobody can re-run is a claim, not evidence. The instruments below are about the part that survives the run: what was recorded, whether someone else could reproduce it, and which specific claim each artifact is actually evidence for.
| What you have | The instrument |
|---|---|
| An eval number about to go into a deck, with no check on whether it is quotable | run the checklist before citing it |
| A result and no record of the conditions it was measured under | write the evaluation card |
| A card or a manifest, and a doubt about whether anyone could re-run it | check it against the reproducibility profile |
| A set of claims going into a deployment decision, and a pile of artifacts beside them | register each claim against its evidence |
| A launch decision, with evidence to assemble into a readout for sign-off | build the launch readout |
The method these instruments apply is written out in the evidence quality framework, which defines the per-claim status vocabulary and the dimensions each claim is assessed against. This family rests on our own work on the four ways an eval number lies and how to report a judge-scored evaluation honestly.
The fork here is whether a gap is absent or recorded. A card with a null temperature and a stated reason is a different object from a card that never had the field. Only the first one tells a reader what to go and find out.
Tools in this topic
Evaluation Card Generator: Markdown and JSON on a Published Schema
Write one evaluation down so a second person can read the number correctly, or run it again. Fill the form, export a Markdown card and a JSON card that validates against a schema published here.
Tool | Evidence and reproducibilityClaim-Evidence Matrix: What Backs Each Deployment Claim
One row per deployment claim, eleven evidence dimensions from relevance and representativeness to independence and applicability, with named rate-down reasons and residual deficits.
Tool | Evidence and reproducibilityIs this eval number ready to quote?
A short check before an eval number goes into a deck or readout. The page returns quotable, quotable with caveats, or not yet, with the sentences to paste beside the number.
Tool | Evidence and reproducibilityWrite the eval readout for a launch decision
Record a launch decision with its evidence: each figure's interval and source, the segments, residual risk and what was not tested. Export it for sign-off or paste a summary.
Tool | Evidence and reproducibilityReproducibility Checklist and Manifest Validator
Check whether your eval write-up has what someone else needs to rerun it, or capture evidence from a production failure before it is lost. Three modes: checklist, manifest validator, incident record.
Tool | Evidence and reproducibilityEval results auditor: find errored, empty and unreadable rows
Reads a results file or pasted rows, classifies each row that got no verdict (errored, NaN, empty, unparsed grade), and recomputes the pass rate with unscored rows counted as failures.
Analyses that use these calculators
- Evaluation design
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Eval statistics
How many cases a reliable eval needs to catch a regression
How many cases a reliable eval needs is a power calculation set by the regression you must catch, your target power, and the baseline pass rate. Includes a case-count table and the formula behind it.
- Judge reliability
Bias-correct your LLM-as-a-judge eval before reporting it
An LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.
Where next
- Directory | 92 calculators
Evaluation and reliability calculators
Calculators for AI agent eval statistics: confidence intervals, paired significance, repeated-run reliability, judge calibration, agreement and bias, prompt robustness, and RAG.
- Reference
Glossary
The metrics these calculators implement, defined in plain language with their assumptions.
- Glossary
Eval reproducibility
Eval reproducibility is getting the same result from an evaluation re-run on the same data and the same parameters; it breaks when uncontrolled non-determinism such as sampling temperature, an unpinned seed, or a drifting judge model moves the score while the declared inputs stay fixed.
- Glossary
Exit criteria (LLM features)
Exit criteria are the conditions that must hold to call a test activity done. For an LLM feature, each criterion names a rate, an interval method, a confidence level and which side it protects, written before the run.
- Glossary
Flaky test (LLM features)
A flaky test is the tester's word for a case whose single run cannot settle a verdict. On an LLM feature, the case has a pass rate across reruns of the same version and input, and the job is to measure that rate.
- Analysis
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Analysis
How many cases a reliable eval needs to catch a regression
How many cases a reliable eval needs is a power calculation set by the regression you must catch, your target power, and the baseline pass rate. Includes a case-count table and the formula behind it.
- Analysis
Bias-correct your LLM-as-a-judge eval before reporting it
An LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.