Agent regression testing calculators
What happens once an eval run has to become a ship-or-hold decision: record runs, track the golden set version, decide whether a candidate clears the bar, and know what to rerun after a change.
What these numbers mean
An eval number turns into a release decision at some point, and that step needs machinery of its own. The instruments below cover every surface between a locked criterion and a shipped change, in the order a release actually reaches them.
| What you have | The instrument |
|---|---|
| A result almost ready, and no written rule for what counts as ship or hold | set the acceptance criteria before the run |
| Eval numbers in hand and a ship-or-hold call still to make | write the gate rule and log the verdict |
| One thing changed in a shipped system, and it isn't clear what evidence just went stale | work out what has to rerun before you ship |
| The eval set moved since the last run, with nothing written down about why | diff the versions and freeze the new baseline |
| An eval run about to start, with no record of the plan before the results come in | preregister the run and log it as a card |
| A model retirement announced, with no record of the date, the replacement or what breaks | look up the dates, replacements and break classes |
| A finished agent run as a trace file, with no count of what happened inside it | count steps, loops and handoffs from the trace |
| An MCP tool surface that grew or changed, with no record of what moved | audit the surface or diff two manifests |
This family rests on our own work on whether an eval difference is real or just noise, what a single pass rate hides once you look past it, and what to check before a change ships.
The gate and the worksheet are meant to run together: the worksheet says what a given change puts back in question, and the gate is where that evidence turns into ship or hold. The version tracker and the run register sit underneath both: one flags when the set itself moved, the other keeps the record neither of the other two invents from memory.
Tools in this topic
Golden Set Version Tracker: Dataset Diff and Changelog
Diff two versions of your golden eval set in the browser, keep a dated changelog of what changed and why, and freeze the baseline metric values the next run is judged against.
Instrument | Agent regression testingRelease Gate Designer: Go/No-Go Rules and Decision Log
Write the rule that turns your eval figures into PROMOTE, HOLD or ROLLBACK, get the verdict plus the rule that produced it, and keep a dated decision log and memo in your browser.
Instrument | Agent regression testingChange Impact Worksheet: What to Rerun Before You Ship
Describe one change to a shipped AI system and get the rerun list it invalidates: which suites, slices and graders to run again, why each is on the list, and a change record for the ticket.
Instrument | Agent regression testingAI Model Deprecation Dates, Replacements and Breaking Changes
Retirement dates, vendor replacements and what breaks in your eval when you swap.
Instrument | Agent regression testingAgent Trace Analyzer: Step Counts, Repeated Calls, Handoff Edges
Upload an agent run trace and get step counts, repeated-call detection with index ranges, handoff edges, and per-class run tallies. Deterministic checks on the file you already have.
Instrument | Agent regression testingMCP Tool Surface Auditor
Audit an MCP tool surface or diff two manifests for breaking changes. Paste your tools/list output. Nothing from the manifest you paste is sent anywhere.
Instrument | Agent regression testingAcceptance Criteria Builder: Set the Bar Before the Run
State each pass/fail criterion, the risk axes it covers, the reasoning behind the threshold and who set it. Get the sample size each one needs and export the rules for the release gate designer.
Instrument | Agent regression testingEval Run Register: Preregistration and Run Cards for LLM Evals
Keep one row per eval run in this browser: freeze the plan before results exist, log what happened, and export a run card or a preregistration record for a PR.
Instrument | Agent regression testingJudge Agreement Tracker: Log Kappa Over Time
Keep a dated log of one judge's agreement with human labels: the value, the prompt version and set behind it, the published band it falls in, the change since the last run, and the next check date.
Instrument | Agent regression testingEvaluation Blueprint Builder: Write the Argument Before You Run the Eval
State each claim your evaluation will make, name the evidence that would support or refute it, and write the inference step connecting the two, all before any results exist.
Analyses that use these calculators
- Judge reliability
Kappa thresholds for LLM judges, and who published each one
Five published kappa bands from four sources, side by side, each with the author who wrote it, the date we read it, and the coefficient it was written for. They are conventions, and they disagree.
- Eval statistics
Claude Fable 5 vs Opus 5 vs Opus 4.8 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, Claude Opus 5 and Claude Opus 4.8 on identical tasks, seven areas scored, every count and caveat published.
- Reliability testing
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
- Reliability testing
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Judge reliability
The LLM-judge bias checklist that gates your ranking
The LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.