Agent regression testing calculators
What happens once an eval run has to become a ship-or-hold decision: record runs, track the golden set version, decide whether a candidate clears the bar, and know what to rerun after a change.
What these numbers mean
An eval number turns into a release decision at some point, and that step needs machinery of its own: something that logs what actually ran, something that notices when the set underneath a score moved, a written rule for what counts as ship or hold, and a way to tell what a given change puts back in question. These four instruments are that machinery, in the order a release actually reaches them.
| What you have | The instrument |
|---|---|
| Eval numbers in hand and a ship-or-hold call still to make | write the gate rule and log the verdict |
| One thing changed in a shipped system, and it isn't clear what evidence just went stale | work out what has to rerun before you ship |
| The eval set moved since the last run, with nothing written down about why | diff the versions and freeze the new baseline |
| An eval run about to start, with no record of the plan before the results come in | preregister the run and log it as a card |
This family rests on our own work on whether an eval difference is real or just noise, what a single pass rate hides once you look past it, and what to check before a change ships.
The gate and the worksheet are meant to run together: the worksheet says what a given change puts back in question, and the gate is where that evidence turns into ship or hold. The version tracker and the run register sit underneath both: one flags when the set itself moved, the other keeps the record neither of the other two invents from memory.
Tools in this topic
Judge Agreement Tracker: Log Kappa Over Time
Keep a dated log of one judge's agreement with human labels: the value, the prompt version and set behind it, the published band it falls in, the change since the last run, and the next check date.
Instrument | Agent regression testingGolden Set Version Tracker: Dataset Diff and Changelog
Diff two versions of your golden eval set in the browser, keep a dated changelog of what changed and why, and freeze the baseline metric values the next run is judged against.
Instrument | Agent regression testingRelease Gate Designer: Go/No-Go Rules and Decision Log
Write the rule that turns your eval figures into PROMOTE, HOLD or ROLLBACK, get the verdict plus the rule that produced it, and keep a dated decision log and memo in your browser.
Instrument | Agent regression testingChange Impact Worksheet: What to Rerun Before You Ship
Describe one change to a shipped AI system and get the rerun list it invalidates: which suites, slices and graders to run again, why each is on the list, and a change record for the ticket.
Instrument | Agent regression testingEval Run Register: Preregistration and Run Cards for LLM Evals
Keep one row per eval run in this browser: freeze the plan before results exist, log what happened, and export a run card or a preregistration record for a PR.
Analyses that use these calculators
- Judge reliability
Kappa thresholds for LLM judges, and who published each one
Five published kappa bands from four sources, side by side, each with the author who wrote it, the date we read it, and the coefficient it was written for. They are conventions, and they disagree.
- Eval statistics
Claude Fable 5 vs Opus 5 vs Opus 4.8 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, Claude Opus 5 and Claude Opus 4.8 on identical tasks, seven areas scored, every count and caveat published.
- Reliability testing
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
- Reliability testing
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Judge reliability
The LLM-judge bias checklist that gates your ranking
The LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.