Tools
Interactive instruments that turn evaluation runs into reliability numbers you can defend. Each one states its formula, its assumptions, and how far a given sample size lets you trust the result.
Evaluation and reliability calculators
Calculators for measuring AI agent reliability from evaluation runs: pass-rate confidence intervals, sample sizes, and repeated-trial success.
reliability@k and pass^k Estimator
Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t intervals and the unbiased pass@k counterpart.
Instrument · evalWilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
Turn an eval pass rate (k of n) into a defensible confidence interval: Wilson score, Clopper-Pearson exact, and Wald normal bounds side by side.
Instrument · evalEval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
Instrument · evalMcNemar Test Calculator for Paired Eval Runs
Run McNemar's paired significance test on two eval runs scored on the same items, with exact and chi-square p-values and a CI on the pass-rate difference.
Instrument · evalSystem Reliability Calculator for Multi-Step Agents
Compute the end-to-end success rate of a multi-step agent chain from its per-step reliability, and the per-step budget your target reliability demands.
Instrument · evalInter-Rater Reliability Calculator (Cohen's Kappa)
Measure how well an LLM judge agrees with a human reference on pass/fail labels: Cohen's kappa from a 2x2 table, with its confidence interval and reading.
Instrument · evalLLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity, then report a calibration-aware confidence interval on the true rate.
Instrument · evalLLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.
Instrument · evalRAG Chunk Size Calculator: Index Size and Context Budget
Size a RAG index from its chunking plan: from chunk size, overlap, and top-k, get the vector count, overlap storage inflation, and context-budget share.
The metrics these calculators implement are defined in the glossary.