Evaluation and reliability calculators

Calculators for measuring AI agent reliability from evaluation runs: pass-rate confidence intervals, sample sizes, and repeated-trial success.

Instrument · eval

reliability@k and pass^k Estimator

Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t intervals and the unbiased pass@k counterpart.

Instrument · eval

Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator

Turn an eval pass rate (k of n) into a defensible confidence interval: Wilson score, Clopper-Pearson exact, and Wald normal bounds side by side.

Instrument · eval

Eval Sample Size and Power Calculator

Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.

Instrument · eval

McNemar Test Calculator for Paired Eval Runs

Run McNemar's paired significance test on two eval runs scored on the same items, with exact and chi-square p-values and a CI on the pass-rate difference.

Instrument · eval

System Reliability Calculator for Multi-Step Agents

Compute the end-to-end success rate of a multi-step agent chain from its per-step reliability, and the per-step budget your target reliability demands.

Instrument · eval

Inter-Rater Reliability Calculator (Cohen's Kappa)

Measure how well an LLM judge agrees with a human reference on pass/fail labels: Cohen's kappa from a 2x2 table, with its confidence interval and reading.

Instrument · eval

LLM Judge Bias Correction Calculator

Correct an LLM judge's raw pass rate for its measured sensitivity and specificity, then report a calibration-aware confidence interval on the true rate.

Instrument · eval

LLM Judge Position Bias Calculator (Swap-Consistency Test)

Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.

Instrument · eval

RAG Chunk Size Calculator: Index Size and Context Budget

Size a RAG index from its chunking plan: from chunk size, overlap, and top-k, get the vector count, overlap storage inflation, and context-budget share.