Evaluation and reliability calculators
92 calculators | 10 topics
Calculators for AI agent eval statistics: confidence intervals, paired significance, repeated-run reliability, judge calibration, agreement and bias, prompt robustness, and RAG.
How to read these
The tools
Agent Loop Cap Calculator
Paste your agent's step counts. Get the iteration cap your coverage target implies, the share of work the cap would cut, and a dollar ceiling for your unattended window.
Tool | Evaluation designAcceptable Error Rate Calculator: The Reliability Bar Automation Needs
Invert your own manual cost, automation cost, and consequence cost into the minimum reliability that makes automating pay, and see if a measured rate clears it.
Tool | Evaluation designEval Run Register: Preregistration and Run Cards for LLM Evals
Keep one row per eval run in this browser: freeze the plan before results exist, log what happened, and export a run card or a preregistration record for a PR.
Tool | Propagation & containmentExpected Cost of Failure Calculator for AI Agents
Price a failure rate in money: what gets caught, what escapes, and what correction and rework savings are worth, split by failure class.
Tool | Eval statisticsReproducibility Checklist and Manifest Validator
Check whether your eval write-up has what someone else needs to rerun it, or capture evidence from a production failure before it is lost. Three modes: checklist, manifest validator, incident record.
Tool | Propagation & containmentAgent Action Risk Matrix: Which Actions Need an Approval Gate
Place every action your agent can take on two axes: whether anything undoes it, and its blast radius, the field's name for propagation radius. Each row returns a written control level, never a score.
Tool | Eval statisticsPaired vs Independent: Choosing an Eval Comparison Design
Six questions give you the design, the test that fits it, the rules that fired, and what the branch you did not pick costs in cases and model runs.
Tool | Eval statisticspass^k, pass@k and reliability@k Estimator: Chance That All k Runs Pass
Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t confidence intervals and the unbiased pass@k capability counterpart.
Tool | Evaluation designIrreversible Action Inventory: What Your Agent Cannot Take Back
List every action your agent can take, mark what reverses it, and name the point of no return. A written verdict per row, never a score or a budget.
Tool | Eval statisticsGolden Set Size Planner: How Many Examples an Eval Set Needs
Size a golden set from a baseline pass rate, a margin, and the drop worth catching: items per slice, a total, and a verdict on the set you already have.
Tool | Eval statisticsWilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
Turn an eval pass rate (k of n) into a defensible confidence interval: Wilson score, Clopper-Pearson exact, and Wald normal bounds side by side.
Tool | Eval statisticsRepeated-Run Variance Planner: How Many Trials per Eval Case
Split your eval's spread into run-to-run and case-to-case variance, then read the trials per case, the case count that implies, and the protocol to freeze.
Tool | Eval statisticsEval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
Tool | Eval statisticsJudge Validation Report Builder: Report, Version Card, Drift Plan
Record how your LLM judge was checked against people. One form builds a validation report, a version card, a drift plan and a checklist.
Tool | Eval statisticsEval Budget Calculator: What an LLM Evaluation Run Will Cost
Price an eval plan before you run it: cases by variants by trials by judge calls, plus the human review hours it needs, from your own rates.
Tool | Eval statisticsMcNemar Test Calculator for Paired Eval Runs
Run McNemar's paired significance test on two eval runs scored on the same items, with exact and chi-square p-values and a CI on the pass-rate difference.
Tool | Eval statisticsHarness Migration Score Continuity Checker
Separate what the harness moved from what your system moved. Feed the overlap bridge and your eval results to see the harness offset, the system-attributable change and a restated baseline.
Tool | Propagation & containmentMulti-Step Agent Reliability Calculator (Independent Steps)
Compute the end-to-end success rate of a multi-step agent chain from its per-step reliability, and the per-step budget your target reliability demands.
Tool | Judge reliabilityCohen's Kappa Calculator (Two Raters): Inter-Rater Reliability
Cohen's kappa from the four counts of a 2x2 table: chance-corrected agreement between two raters, with a confidence interval and an interpretation band. Built for an LLM judge against human labels.
Tool | Judge reliabilityLLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity (the Rogan-Gladen correction), then report a calibration-aware confidence interval on the true rate.
Tool | Judge reliabilityLLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.
Tool | RAG & retrievalRAG Chunk Size Calculator: Index Size and Context Budget
Size a RAG index from its chunking plan: chunk size, overlap, and top-k give the vector count, storage inflation, and context-budget share. Which size retrieves best stays a measured question.
Tool | Evaluation designLLM Token Counter: Context Budget and Cost per Attempt
Count tokens for a named model, see what the request reserves in the context window once output is set aside, and price one attempt at your own rates. Exact where we bundle the encoding.
Tool | Judge reliabilityLLM Judge Calibration Calculator: Brier Score and ECE
Score an LLM judge's stated confidence against observed correctness: Brier score, expected and maximum calibration error, a reliability diagram, and the error rate behind its confident verdicts.
Tool | Eval statisticsLLM Eval A/B Comparator: Paired Difference with Bootstrap CIs
Run an LLM eval A/B test on paired graded scores: mean score difference with a bootstrap confidence interval, paired effect size, and win/tie/loss evidence.
Tool | Judge reliabilityPairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
Turn pairwise LLM judge comparisons between models or prompts into win rates with Wilson intervals, Bradley-Terry strengths on the Elo scale, and a plain read on whether the advantage is real.
Tool | Evaluation designHuman Review Threshold Optimizer for Confidence-Gated Automation
Derive the human review threshold from your own records: paste stated confidence and observed correctness, price one review against one missed failure, and read the cut that costs least.
Tool | Judge reliabilityFleiss' Kappa, Krippendorff's Alpha and ICC Calculator
Agreement among three or more raters or LLM judges: Fleiss' kappa, Krippendorff's alpha, Gwet's AC1 and intraclass correlation, with percent agreement, bootstrap intervals, and per-rater diagnostics.
Tool | Eval statisticsJudge Agreement Tracker: Log Kappa Over Time
Keep a dated log of one judge's agreement with human labels: the value, the prompt version and set behind it, the published band it falls in, the change since the last run, and the next check date.
Tool | Propagation & containmentRetry, Verification, and Fallback Reliability Optimizer for LLM Agents
How many times should you retry a failed agent step? Compare retry, majority vote, verifier gate, fallback, and escalation on correctness and cost, from your own rates, and price a circuit breaker.
Tool | Judge reliabilityYour judge disagrees with your humans. Which bias explains it?
Paste judge and human labels. See which bias mode explains the gap, ranked by how much each one accounts for, then follow the link to the calculator that tests it.
Tool | Evaluation designAI SLO Error Budget Calculator: Split a Failure Budget Across Classes
Turn a completion-rate target and a task volume into a failure budget split by class, with a Wilson interval on each class observed rate against its allowance.
Tool | Eval statisticsBenchmark Rank Uncertainty Calculator: Score Intervals and Rank Ranges
Put error bars on an LLM benchmark leaderboard. Paste scores with item counts, correct-of-total, or standard errors and see which adjacent ranks are a statistical tie.
Tool | Evaluation designPrompt Robustness Analyzer: Perturbation Stability for LLM Evals
Is a prompt's pass-rate drop real or noise? Paste pass/fail results for a baseline prompt and its semantically equivalent variants and read each change with a paired interval and an exact McNemar p.
Tool | RAG & retrievalRAG Retrieval Quality Calculator: Precision@K, Recall@K, nDCG, MRR, MAP
Score a retrieval run per query and across the suite: Precision@K, Recall@K, Hit@K, nDCG, mean reciprocal rank (MRR), and mean average precision (MAP), with a K sweep and intervals.
Tool | Propagation & containmentMulti-Step Agent Reliability with Shared Failures (Cascade Simulator)
Price a multi-step agent chain when its steps share a cause: end-to-end reliability under common-cause failure beside the independent figure, plus the retry ceiling that shared failures impose.
Tool | Eval statisticsEval Dataset Schema Validator: Golden-Set Checks and Export Profiles
Check an eval golden set against a neutral schema in your browser: findings by row and line, duplicate ids, field completeness, plus a template, exports, and a converter for eval harness formats.
Tool | Eval statisticsExact Duplicate Row Checker for Eval and Golden Datasets
Find byte-identical rows in an eval or golden dataset without uploading it. Duplicate groups with row and line numbers, an identity key you choose, and the sample size left after a dedupe.
Tool | Eval statisticsNear-Duplicate Row Detector: Find Eval Rows That Differ Only in Formatting
Find rows in an eval set that are the same test written twice. Normalize the columns you pick, group rows whose normalized text matches, and read the exact transform, the raw variants, and a CSV.
Tool | Eval statisticsCross-Split Overlap Checker: Find Eval Items in Both Train and Test
Compare two eval splits in your browser and see which items sit on both sides, with a separate overlap rate against the row count of each split, exact and normalized matching, and the lines to open.
Tool | Eval statisticsN-Gram Overlap Checker: Find Contamination Between Two Eval Files
Check a test set against a training file for shared word sequences. Pick n, read the per-row dirty-token share, see the matching text in both files, and export a CSV. Nothing is uploaded.
Tool | Eval statisticsSlice and Class Balance Analyzer: See What an Eval Set Contains
Read what an eval set holds before you quote a pass rate. Count rows per slice, per class, and per control flag, put an interval on every share, and see the smallest slice named.
Tool | Propagation & containmentCost per Successful Task Calculator: What One Acceptable Result Costs
Turn a per-attempt price and a pass rate into the cost of one acceptable result, with retries, failed attempts, review and the fallback path all priced into the total.
Tool | Eval statisticsEvaluation Card Generator: Markdown and JSON on a Published Schema
Write one evaluation down so a second person can read the number correctly, or run it again. Fill the form, export a Markdown card and a JSON card that validates against a schema published here.
Tool | Evaluation designGolden Set Version Tracker: Dataset Diff and Changelog
Diff two versions of your golden eval set in the browser, keep a dated changelog of what changed and why, and freeze the baseline metric values the next run is judged against.
Tool | Evaluation designRelease Gate Designer: Go/No-Go Rules and Decision Log
Set the go/no-go rule for your eval figures, get the call and save each decision to a dated log. Export the latest as a memo.
Tool | Evaluation designChange Impact Worksheet: What to Rerun Before You Ship
Describe one change to a shipped AI system and get the rerun list it invalidates: which suites, slices and graders to run again, why each is on the list, and a change record for the ticket.
Tool | Evaluation designAI Model Deprecation Dates, Replacements and Breaking Changes
Retirement dates, vendor replacements and what breaks in your eval when you swap.
Tool | Eval statisticsIndependent Two-Proportion Test Calculator for Eval Arms
Compare two pass rates measured on different case sets: a pooled two-proportion z test, Fisher exact when an arm is small, and an interval on the difference.
Tool | Evaluation designContext-of-Use Profiler: Turn a System Description into Evaluation Consequences
Describe what your AI system does, who it affects and how it fails, and get the failure families, case classes, slices and evidence expectations those answers demand.
Tool | Evaluation designAgent Trace Analyzer: Step Counts, Repeated Calls, Handoff Edges
Upload an agent run trace and get step counts, repeated-call detection with index ranges, handoff edges, and per-class run tallies. Deterministic checks on the file you already have.
Tool | Evaluation designLLM Latency Budget Calculator: TTFT and Token Time Across a Chain
Compose per-step time-to-first-token and token-stream latency into a chain deadline. See which step spends the budget, what kind of time it spends, and the timeout each layer should hold.
Tool | Evaluation designEvaluation Blueprint Builder: Write the Argument Before You Run the Eval
State each claim your evaluation will make, name the evidence that would support or refute it, and write the inference step connecting the two, all before any results exist.
Tool | Evaluation and reliability calculatorsMCP Tool Surface Auditor
Audit an MCP tool surface or diff two manifests for breaking changes. Paste or open your tools/list output. Nothing is uploaded.
Tool | Evaluation designRisk-to-Test Mapper: Which Failures Need Which Cases
Name the failures you care about, then see the test-case shapes, slices, evaluators and metrics each one needs. Every mapping says what it cannot establish. The mapping is a LatentEval synthesis.
Tool | Eval statisticsEvaluation Coverage Matrix: Which Cells Have No Cases
Put what you test on one axis and what you test it across on the other. The grid shows which cells have cases, which are empty, which are too small to carry an interval, and what each gap costs you.
Tool | Eval statisticsEvaluator and Metric Navigator: Which Grader, Which Number
Answer four questions about oracle type, grader, system count, and failure mode. Get the admissible evaluator, inadmissible ones and why, the metric family, and a route to the tool that runs it.
Tool | Eval statisticsWhat Does This Evaluation Prove?
Select the evidence your evaluation produced, see which claims it supports, which it cannot, and the inference error behind each gap. Each unsupported claim names the instrument that would close it.
Tool | Evaluation designAcceptance Criteria Builder: Set the Bar Before the Run
State each pass/fail criterion, the risk axes it covers, the reasoning behind the threshold and who set it. Get the sample size each one needs and export the rules for the release gate designer.
Tool | Eval statisticsGolden Set Blueprint Builder: Seed a Test Set From a Task Family
Choose a task family, load its case taxonomy and starter rows, then edit, add and export. Every row carries an oracle type and a provenance tag, and the export validates against the eval-set schema.
Tool | Eval statisticsClaim-Evidence Matrix: What Backs Each Deployment Claim
One row per deployment claim, eleven evidence dimensions from relevance and representativeness to independence and applicability, with named rate-down reasons and residual deficits.
Tool | Propagation & containmentPer-correct-answer cost of a two-tier router, misroutes included
Does the router still save money once its own mistakes count? Enter per-tier prices, pass rates and misroute rates to see cost per correct answer, the quality gap, and when routing stops paying.
Tool | Evaluation and reliability calculatorsAt what load does serving your own model cost less than the API?
Enter your instance throughput, fleet parameters and API token prices. The calculator sizes a fleet for the peak, compares both monthly bills and finds the request volume where they cross.
Tool | Eval statisticsHow long to run a canary release before you know it is worse
Enter a pass rate, the drop to catch and your canary traffic share. See how many hours the watch needs and a stopping rule you can check whenever you like without inflating false alarms.
Tool | Propagation & containmentWhich agent topology holds up when a step can fail
Enter a per-agent success rate and a team size. See end-to-end success, caught and escaped failures for five computed topologies and a single-agent baseline, ranked with intervals from sample counts.
Tool | Eval statisticsHow inflated is your eval score after reusing the same set?
Enter your set size, how many candidates you scored, and the gap you can live with. See how much selection inflated the winner and how many more looks the set supports.
Tool | Evaluation designHow long your eval suite adds to CI, and what a PR subset detects
Enter cases, concurrency and cadence across three tiers. The calculator reports wall-clock minutes under parallelism, monthly eval spend, and how much detection the per-PR subset gives up.
Tool | Evaluation designWhat is your structured output failure rate?
Paste or upload a file of model outputs and see your failure rate by class. Each rate carries a Wilson interval, and the page produces the two numbers the retry optimizer needs.
Tool | Propagation & containmentHow many reviewers your human gate needs to hold the wait target
Enter the reviewed share, handling time and shift size. The calculator returns the wait, the backlog, the service level against your target, and the minimum staffing to hold it.
Tool | Evaluation designShould this task be an agent, a workflow, or ordinary software?
Ten questions about the task you are about to automate produce one verdict: build an agent, wire a fixed workflow, or skip the model. See which single answer you would change to flip it.
Tool | Propagation & containmentHow long to wait on a 429, and how many requests to keep in flight
Enter your rate limits, backoff settings and current load. The planner computes every wait in the schedule, the concurrency your quota allows, and the time a backlog takes to clear.
Tool | Eval statisticsFind eval rows that mean the same thing in different words
Paste an eval set and pick the text column. An in-browser model groups rows that ask the same question in different words, reports how many collapse, and exports the pairs with a run stamp.
Tool | Evaluation designHow much evaluation does this feature need?
Six questions about your feature, one of four evaluation tiers, the minimum setup for that tier, and the single answers that would change the verdict.
Tool | Evaluation designError analysis workbench for LLM traces and eval results
Review traces or eval results one case at a time: mark pass or fail, code each failure, note its first failing step, and build a failure taxonomy. Your work saves to a session file you download.
Tool | Eval statisticsReview sample drawer: random and stratified sampling for eval review
Take a reproducible random, stratified or fixed-rate review sample from a large CSV or JSONL, with each row's inclusion probability and a manifest of the settings. Your file stays in this browser tab.
Tool | Evaluation designCriteria and rubric builder for LLM eval grading
Turn failure categories into pass or fail criteria with must and must-not statements and examples, lint vague wording, and version the rubric as a file for labelers and whoever builds the checks.
Tool | Judge reliabilityBuild a labeling handbook from criteria, with rules, examples and a quiz
Turn criteria into a labeling handbook with rules, examples, precedence and a cannot-judge policy, plus a quiz and a precedent log. The page exports the handbook, quiz, answer key and log as files.
Tool | Judge reliabilityHow many human labels to trust the judge?
Enter the catch rate and pass accuracy you expect from a model judge, and see how many human labels to collect, per class, with random and stratified designs and the bias trap in a naive reading.
Tool | Eval statisticsCheck an eval set for duplicates, leakage and personal data
Drop a test file and get a report on its schema, blank cells, duplicate and near-duplicate cases, cross-split leakage, class balance, and personal data, in one pass.
Tool | Evaluation designCode grader playground: score one expected and actual pair
Paste an expected answer and the model's output, pick code graders, and see each score with its reason and near-miss hints.
Tool | Evaluation designScore a results file with code graders
Load a results file, choose code graders, see pass, fail, near-miss and disagreement counts per grader, and download a scored file, all in your browser.
Tool | Evaluation designBuild a normalization pipeline and check it against your harness
Choose normalization steps in order, load a sample file, and see which rows each step flips, whether the pipeline reproduces your harness, and where the gaps are.
Tool | Eval statisticsImport eval results from promptfoo, Inspect, lm-eval or OpenAI
Paste six-field result rows or load a promptfoo, Inspect, lm-eval or OpenAI results file in your browser, and download per-case rows with errored cases kept.
Tool | Eval statisticsEval results auditor: find errored, empty and unreadable rows
Reads a results file or pasted rows, classifies each row that got no verdict (errored, NaN, empty, unparsed grade), and recomputes the pass rate with unscored rows counted as failures.
Tool | Evaluation designWrite a CI eval gate with three verdicts and a false-block forecast
Set pass-rate floors and regression margins for PR, nightly and release tiers, pin a baseline by hash, quarantine flaky cases, and see how often the gate will block an unchanged version.
Tool | Eval statisticsNo-regression verdict against a pre-set margin
Set a margin before the run and check paired pass/fail results against it: not worse, worse, or inconclusive, with one-sided bounds from a Newcombe paired interval and next-run sizing.
Tool | Eval statisticsSlice explorer with false-alarm control
Test each slice of one run against the rest, with Holm and Benjamini-Hochberg corrections across all tested slices, intervals per slice, a floor for small slices, and a Simpson's paradox check.
Tool | Eval statisticsIs this eval number ready to quote?
A short check before an eval number goes into a deck or readout. The page returns quotable, quotable with caveats, or not yet, with the sentences to paste beside the number.
Tool | Eval statisticsWrite the eval readout for a launch decision
Record a launch decision with its evidence: each figure's interval and source, the segments, residual risk and what was not tested. Export it for sign-off or paste a summary.
Tool | Evaluation designModel upgrade checklist
Track sixteen steps for a model or vendor switch, eight of which block it. Record each step's status, export a dated record, and re-import it to pick up where you left off.
Tool | Evaluation designBuild a monitoring plan for your AI feature
Set traffic, review share, judge budget and alert thresholds. See review hours, false pages with a suggested threshold, and a canary noise floor, then export the plan as a file. No live data is read.
Tool | Eval statisticsThumbs-down rate: what does 2% really mean?
Enter thumbs-up and thumbs-down counts beside total conversations, see the range consistent with silent conversations, and size a review that narrows it.