INSTRUMENT | Reliability testing
Change Impact Worksheet: What to Rerun Before You Ship
1 cited source
Describe one change to a shipped AI system and get the rerun list it invalidates: which suites, slices and graders to run again, why each is on the list, and a change record for the ticket.
Change one thing in a shipped system and a different part of your evidence goes stale. Answer two short steps here and read back which suites, slices, and graders have to run again before it ships, with the reason on every line and somewhere to run it.
It rests on our pre-ship reliability checks, and the record it fills hands off to the release gate.
Showing your last valid result. Update the inputs above to recompute.
Reruns before this ships
12 required
Model or provider swap, all traffic. 12 required reruns, 4 suggested. Nothing is blocked.
What to run again, and why. The last column is where to run it.
| Rerun | Your suite | Band | Why it is on the list | Where to run it |
|---|---|---|---|---|
| Core task suite on the frozen golden set | required | A model or provider swap changes the generator, so the headline pass rate is no longer comparable until it is measured again. | pass-rate interval | |
| Paired comparison against the current version, same cases | required | A model or provider swap changes the generator, so only the same cases run twice can separate the change from ordinary run-to-run noise. | prompt robustness , paired significance , A/B comparison | |
Per-slice pass rates for the named slices
| required | A model or provider swap changes the generator, so an average can hold steady while one slice moves underneath it. | not yet live | |
| Output-contract and parse conformance | required | A model or provider swap changes the generator, so a shape that stops parsing fails downstream long before anyone reads a number. | schema conformance | |
| Safety and refusal suite | required | A model or provider swap changes the generator, so refusal behavior moves for reasons nobody intended and nobody watched. | not yet live | |
| Prompt-injection and adversarial suite | suggested | A model or provider swap changes the generator, so the attack surface moves whenever the instructions or the retrieved content move. | not yet live | |
| Tool-call correctness: schema, arguments, wrong-tool rate | required | A model or provider swap changes the generator, so a call with the wrong arguments looks like a working call until it lands. | not yet live | |
| Judge agreement against human labels | suggested | A model or provider swap changes the generator, so an automated judge is only usable while it still agrees with a person. | rater agreement | |
| Repeated-run variance at the new setting | required | A model or provider swap changes the generator, so a difference smaller than the spread between two identical runs is not a difference. | run-to-run variance | |
| Cost per successful task | required | A model or provider swap changes the generator, so per-call price is not the bill; retries and failures are where it goes. | cost per success | |
| Latency distribution | required | A model or provider swap changes the generator, so the mean moves last, and the tail is what people actually feel. | error budget and watch window | |
| Baseline invalidation notice | required | A model or provider swap changes the generator, so anyone still comparing against the old baseline needs telling that it no longer holds. | release gate | |
| Golden-set changelog entry | suggested | A model or provider swap changes the generator, so an undated set change makes every later movement in the numbers unreadable. | golden-set changelog | |
| End-to-end multi-step run | suggested | A model or provider swap changes the generator, so each step can pass on its own while the chain still fails. | not yet live | |
| Context-window and long-input behavior | required | A model or provider swap changes the generator, so long inputs fail differently, and usually later, than the short ones anyone tests. | not yet live | |
| Error paths, rate limits, and timeouts | required | A model or provider swap changes the generator, so the failure path is the part nobody rehearses and everybody eventually meets. | not yet live |
The change record
- Change
- Model or provider swap: Route borderline prompts to a stricter model
- Why
- The current generator is slower than the latency target allows at peak.
- From
- model-a 2026-06-01
- To
- model-b 2026-06-01
- Owner
- A. Lead
- Planned
- 2026-09-01
- Scope
- all traffic
- Reversibility
- redeploy required
- Risks remaining
- Behavior under sustained load is untested.
- Rollback
- Point the router back at the previous model and redeploy.
- Rollback trigger
- Any safety case fails, or task success drops past the gate.
- Rollback rehearsed
- no
- Watch window
- 48 hours after the change
Paste this into the ticket unedited.
Which rules fired, and which did not
- column MD read
- M02 fired: No retrieval, so the retrieval items do not apply.
- M07 fired: Named slices were given.
- M08 fired: Something downstream parses the output.
- M09 fired: Cost or latency is sensitive here.
- M09 fired: Cost or latency is sensitive here.
- M11 fired: This goes to all traffic.
- M01: not fired (hasGoldenSet is no)
- M02: fired
- M03: not fired (usesTools is no)
- M04: not fired (usesJudge is no)
- M05: not fired (hasSafetySuite is no)
- M06: not fired (rerankerChanged is checked)
- M07: fired
- M08: fired
- M09: fired
- M10: not fired (criticalPath is checked)
- M11: fired
- M12: not fired (reversibility is irreversible side effects)
- M13: not fired (oneThingOnly is unchecked)
- M14: not fired (changeClass is emergency)
Bring a record back
The only file this page reads is a record it exported itself. Nothing else is uploaded, and nothing leaves your browser.
JSON. Drag one here, or use the box below.
Check this value.
How this is calculated
Two ordered passes, and nothing in either one is a score. The first pass reads one column of a fixed table: the seven change types are the columns, the rerun items are the rows, and each cell says required, suggested, or not on the list. A custom change reads whichever base column you say it behaves like.
The second pass runs fourteen rules in a fixed order over your answers about the system. Each one can add a row, lift a row to required, remove a row, or leave a note. Every rule reads your raw answers rather than the running list, and every removal is applied after every addition, so a row that came off never comes back and the order of the table never changes which rows survive. The drawer above prints all fourteen, fired or not.
The counts are counts. Nothing is divided, averaged, weighted, or turned into a grade: a twelve-row list is longer than a six-row list and that is all a number here means. Bands belong to a row, never to the list.
One case is worth knowing because it looks like a bug. A retrieval change on a system with no callable tool keeps the prompt-injection row, because retrieved content is an injection channel on its own. That row leaves only when the system has neither retrieval nor tools.
Where to run every item, in one place
The list above shows only the items your change puts on it, so the tools for the rest are not on screen. Here is the whole index. An item with nothing beside it has no page here yet.
- Core task suite on the frozen golden set pass-rate interval
- Paired comparison against the current version, same cases prompt robustness , paired significance , A/B comparison
- Per-slice pass rates for the named slices not yet live
- Output-contract and parse conformance schema conformance
- Safety and refusal suite not yet live
- Prompt-injection and adversarial suite not yet live
- Retrieval recall at k recall at k
- Retrieval relevance at the new chunk size and top-k chunk size and top-k
- Reranker precision reranker precision
- Grounding and faithfulness not yet live
- Tool-call correctness: schema, arguments, wrong-tool rate not yet live
- Permission and irreversible-action gates action risk
- Judge agreement against human labels rater agreement
- Judge stability across repeated runs judge drift
- Rubric back-compatibility on a frozen sample not yet live
- Repeated-run variance at the new setting run-to-run variance
- Cost per successful task cost per success
- Latency distribution error budget and watch window
- Baseline invalidation notice release gate
- Golden-set changelog entry golden-set changelog
- End-to-end multi-step run not yet live
- Context-window and long-input behavior not yet live
- Error paths, rate limits, and timeouts not yet live
- Rollback rehearsal on a copy not yet live
Questions
Questions
Why is there no risk score?
Because a number cannot tell you what to run. The list is the answer, each line carries its own reason, and a low, medium or high band over a count would be the same missing answer wearing more confidence.
What if my change is not one of the seven types?
Name it, then say which of the seven it behaves like. The select opens empty and nothing computes until you pick, because a guessed base would quietly answer a question you never looked at.
Nothing is saved. Where does my record go?
Into the export, and into your ticket. This page keeps nothing between visits, which is why the export is the save path and why a record can be read back in.
Can I change two things at once?
You can, and the worksheet says so rather than stopping you. A difference you measure afterwards will not attribute to a cause, and that note goes into the record.
What does this hand to the release gate?
The filled record, as an attachment. The gate never reads a figure out of it: a number moves between the two pages only because you typed or pasted it.