Criteria
3
TOOL | Evaluation design
Write each acceptance criterion with the risk it covers (reversibility and blast radius), why that number, who set it, and, for a rate bound, its sample size. Freeze the criteria before results arrive. When a threshold rests on cost arithmetic, derive the tolerable failure rate. Built on our statistical significance and judge threshold research. Pairs with the release gate designer. Tighten a gate where the action risk matrix shows wider reach, and split the allowed failures from the SLO error budget.
UNRESOLVED
1
1 unresolved item across 3 criteria.
Criteria
3
Gate rules
3
| Criterion | Form | Reversibility | Blast radius | Gate rule | On fail |
|---|---|---|---|---|---|
| Severe failure rate under 0.1 percent | Bounded tolerance (rate upper bound) | R3: Point of no return. Nothing undoes it. | B3: Outside the organization. Customers, partners, the public, regulated populations. | Floor | Hold |
| Core pass rate above 92 percent | Threshold (mandatory floor) | R1: The actor can undo it. Version history, a restorable delete, an idempotent re-apply. | B1: One team, one internal system, one department process. | Floor | Hold |
| Human review of every irreversible case | Review requirement (evidence) | R3: Point of no return. Nothing undoes it. | B3: Outside the organization. Customers, partners, the public, regulated populations. | Evidence | Hold |
Freezing records the date, who froze it, and a fingerprint of the policy text. It cannot prove when anyone first saw the data.
Reads an acceptance criteria policy saved from this page as JSON, version 1.x. Restores the form, the freeze and the frozen copy.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
The required sample size is the smallest n such that the upper bound at zero failures is at most the target rate r, under the criterion's own interval method, sidedness and confidence level. Alpha is 1 minus the confidence level for a one-sided bound, and half of that for a two-sided interval, so it is 0.05 at the default. For Clopper-Pearson, n equals the ceiling of the natural log of alpha divided by the natural log of (1 minus r). At the default settings, the Clopper-Pearson n sits just under the "3 divided by r" mnemonic (2995 against 3000 at a target rate of 0.001), which is slightly conservative. For Wilson, n equals the ceiling of z-squared times (1 minus r) divided by r, where z is the normal quantile at 1 minus alpha. A criterion that names its own interval method shows no required n; the gate reads whatever interval you bring. Each criterion's card and both exports name its sidedness, level and method as one phrase, such as "one-sided 95% Clopper-Pearson".
Each of the ten criterion forms maps to one of the five rule kinds the release gate designer reads (must pass, floor, regression, budget, evidence) with a fixed on-fail action and a set of fields the gate rule reads. The two risk axes (reversibility and blast radius) are never multiplied or averaged.
| Criterion form | Rule kind | Mandatory | On fail | Fields read |
|---|---|---|---|---|
| Blocker (zero tolerance, rollback) | Must pass | Yes | Roll back | none |
| Must-pass (zero tolerance, hold) | Must pass | Yes | Hold | none |
| Threshold (mandatory floor) | Floor | Yes | Hold | direction, threshold |
| Target (advisory) | Floor | No | Hold | direction, threshold |
| Slice minimum | Floor | Yes | Hold | direction, threshold, suite |
| Guardrail (budget cap) | Budget | Yes | Hold | threshold |
| Bounded tolerance (rate upper bound) | Floor | Yes | Hold | direction, threshold |
| Statistical condition (regression test) | Regression | Yes | Hold | tolerance, alpha, baseline |
| Review requirement (evidence) | Evidence | Yes | Hold | artifact |
| Rollback trigger | Must pass | Yes | Roll back | none |
The freeze fingerprint is a SHA-256 hash of a canonical text. The canonical text starts with the schema line, then the JSON body with keys sorted, criteria sorted by id, whitespace collapsed and trimmed, undefined written as null, and the freeze block itself excluded. The first 12 hex characters of the hash are shown on the page. Loading a saved policy re-hashes the frozen copy against the stored fingerprint; if they disagree, the stamp is not trusted.
The release gate designer runs the rule against numbers. This tool captures why the rule is set to that number, who decided it, and how many cases the claim needs. A threshold without a stated reason is a threshold nobody can argue with or update.
It lands in the unresolved list at the top. The tool still exports the rule, but the policy document records the gap.
The JSON export writes a release gate spec document with a data payload shaped for the release gate designer's input. Save the file, open the release gate designer, and import it there. The import message records that the thresholds came from this policy, and when the policy was frozen, the decision memo also carries its freeze date and fingerprint; review each threshold before locking it in.
The gate reads the target rate as a floor with "at most" direction, and evaluates it from your count of cases. With the criterion's interval method and sidedness, the gate builds the interval and tests whether the bound clears the bar. Without a count, the rule is not evaluated and the gate holds. The exported rule carries the method, sidedness and confidence level so the gate applies the same test you chose here.
Floor criteria (threshold, target, slice-minimum) carry a per-criterion interval method, sidedness and confidence level. When the gate receives a count, it builds the interval and tests the bound against the bar. A value typed alone on the gate is a point check.
The comparison of a candidate's pass rate against a baseline's is a paired test when both runs score the same cases, or a two-proportion test when the case sets differ. Choose which design fits before exporting.
A one-sided 95% bound is one edge of the two-sided 90% interval, and a two-sided 95% interval is wider. On 12 failures in 400 against a bar of 0.05, the one-sided 95% Clopper-Pearson upper bound is 0.048, which clears, while the two-sided 95% upper bound is 0.052, which does not. The choice decides the verdict whenever the rate sits close to the bar.
Freezing records a date from this device's clock, the name in "Frozen by", and a SHA-256 fingerprint of the policy text. If the frozen copy in a saved file is edited later and its fingerprint is left as it was, the two no longer match, and the page says the stamp is not trusted. The date comes from this device, nobody else signs it, and anyone with the file can recompute the fingerprint, so freezing cannot prove when anyone first saw the eval data. An edited-after-freeze flag appears if any field changed after the stamp.
It records who may object after the numbers are in, what evidence they must bring, who rules on it, and by when. An upheld objection holds the release on the gate page, whatever the verdict. It exists because criteria written before the run cannot cover every failure the run reveals, and a team member who sees something the criteria did not cover needs a path to say so.