TOOL | Evaluation design
Model upgrade checklist
Track sixteen steps for a model or vendor switch. Eight block the switch: snapshots pinned, retirement date known, baseline captured, a paired frozen-set comparison, output contract, judge re-validated, safety set, and a canary plan with rollback. Mark each step OK, problem found or not applicable. When every blocking step clears, the checklist reads ready. The no-regression verdict answers whether the new model is worse. Export a model upgrade checklist for the launch review.
Mark each step as you go. The verdict appears when at least one step has a status.
Load a checklist you saved from this page as JSON, version 1.x. The page restores the form.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
Stored on this device only. Export the checklist as JSON to keep a durable copy.
This browser is not keeping data for this page, so your checklist is lost when you leave. Export it as JSON to keep a copy.
How this is calculated
The gate rule Every step is either applicable or not applicable. Among the applicable steps, the blocking steps must all be OK for the checklist to read ready. A blocking step that is blank or that records a problem holds the checklist at not ready. When every blocking step is OK or marked not applicable and a non-blocking step is still open, the verdict reads ready, with open steps. Not applicable requires a reason; without one, the step counts as not done.
Date arithmetic Days to retirement and days to switch are whole calendar days from today. A past date shows no count; the line says the date has passed. The retirement date is the provider's announced date; the target switch date is the team's plan. When the switch date falls after the retirement date, the page warns that the plan switches after the old model stops answering.
The host check API providers can serve a different model under the same id, and checks that read only outputs detect the swap unreliably (Cai et al., 2025). Versions, defaults and retirement dates can also differ by host. The host check compares the same model on each host the team uses.
What "problem" means A step marked "Problem found" means the check was done and found something the team should resolve. It blocks the switch on the eight blocking steps and stays visible on the rest. A problem on a non-blocking step stays in the exported record and holds the verdict below ready: with every blocking step clear, it reads ready, with open steps. Record what was found in the note field. A team that switches with a found problem records that in its release decision, and the step stays a problem here.
Questions
What does "ready" mean?
Ready means every blocking step is OK or marked not applicable. Whether the new model is worse than the old by more than your margin is the no-regression verdict's question. Non-blocking steps that are still open appear in the verdict as open steps.
Why do these eight steps block?
Each one guards against a failure that is hard to recover from after the switch. Unpinned snapshots let a model change silently. A missed retirement date leaves no time for the other steps. A baseline not captured before shutdown leaves nothing to compare against. A comparison without a frozen set or with a harness change mixed in answers two questions at once. An unchecked output contract lets parse failures into production. An unvalidated judge shifts the bar. A skipped safety set leaves the hardest requirements untested. A switch with no rollback path has no exit.
The old model is already gone. What now?
If the old model's retirement date has passed and the baseline has not been captured, the page warns that the old outputs cannot be regenerated from the provider. Capture what remains from logs or cached outputs, mark the baseline step not applicable with the reason, and the comparison runs against recorded history only. Every other step still applies.
How does this page relate to the rerun list on the change impact worksheet?
The change impact worksheet lists which eval suites to re-run for a model or provider swap and keeps a change record with rollback fields. This checklist tracks whether each step has been done, including the steps that sit outside a rerun: pinning snapshots, the retirement date, the baseline, the changelog, repricing tokens and checking each host. Use both together.
Why separate a harness change from a model change?
Running the old and new models through the same harness version, paired by case, isolates the model change. A harness change mixed in (a new grader, a changed prompt template, a different retry policy) makes the comparison answer two questions at once. If both changed, the harness migration page measures the harness offset separately.
When does the judge need re-validation?
Re-validate the judge when its prompt, criteria or model changed, or when its model now shares a family with the model it evaluates. A shared family means the judge may favor the evaluated model's style. If the judge and its model are unchanged and share no family, the step can be marked not applicable with a reason.
Why check each host?
A host can serve a cheaper or compressed model under the same id, and output-only checks detect the swap unreliably. Versions, defaults and retirement dates can also differ by host. If only one host is used, mark the step not applicable with that reason.
Where are answers kept?
Answers are kept in this browser on this device, for this page only, until you clear the data or press Start over. Export the checklist as JSON for a dated record, and import it on another device or after clearing your browser data. The file carries the form state; the page recomputes the verdict and dates.
Why compare cost per successful task?
A model with a lower price per token may cost more per successful task if its pass rate is lower or if it uses more tokens per output. Comparing cost per successful task shows whether the saving is real after accounting for retries and failures. The cost per successful task page computes it.