LatentEval

TOOL | Evaluation design

Acceptance Criteria Builder: Set the Bar Before the Run

Write each acceptance criterion with the risk it covers (reversibility and blast radius), why that number, who set it, and, for a rate bound, its sample size. Freeze the criteria before results arrive. When a threshold rests on cost arithmetic, derive the tolerable failure rate. Built on our statistical significance and judge threshold research. Pairs with the release gate designer. Tighten a gate where the action risk matrix shows wider reach, and split the allowed failures from the SLO error budget.

The form chosen for a criterion decides which of the five gate rules it becomes and whether a failure holds or rolls back.

UNRESOLVED

1

1 unresolved item across 3 criteria.

Criteria

3

Gate rules

3

Policy

Check this value.

Criterion 1

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

A one-sided 95% bound is one edge of the two-sided 90% interval. Two-sided reads the wider two-sided interval at the level you pick, so it needs more cases to clear the same bar.

Check this value.

Clopper-Pearson is exact and errs wide, so its interval covers the true rate at least as often as the level says. Wilson is narrower. "Own interval" is for a mean or a method this page does not run: give its name in "Method name" and bring the interval to the gate.

Check this value.

Sets the required sample size here, and the level the gate uses when it builds the interval from the counts you bring to the release gate designer.

Check this value.

Required sample size: 2995 cases (one-sided 95% Clopper-Pearson upper bound)

Rule-of-three mnemonic (one-sided 95% bound): 3/0.001 = 3000

One-sided 95% upper bound at n = 2995: 0.1%

The 3/n mnemonic uses the same one-sided 95% convention as this tool. At the mnemonic's n = 3000 the bound is 0.0998%, at or below the 0.1% target.

Criterion 2

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

A one-sided 95% bound is one edge of the two-sided 90% interval. Two-sided reads the wider two-sided interval at the level you pick, so it needs more cases to clear the same bar.

Check this value.

Clopper-Pearson is exact and errs wide, so its interval covers the true rate at least as often as the level says. Wilson is narrower. "Own interval" is for a mean or a method this page does not run: give its name in "Method name" and bring the interval to the gate.

Check this value.

Sets the required sample size here, and the level the gate uses when it builds the interval from the counts you bring to the release gate designer.

Check this value.

The release gate designer reads this floor from a count of cases out of a total, with the one-sided 95% Clopper-Pearson test. An observed value typed there alone is a point check with no interval.

Criterion 3

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Check this value.

Sign-off

The person or role who signs off on the release decision. Required before freezing.

Check this value.

Who may grant an exception. The release gate designer asks for this name before it saves an override on a ROLLBACK.

Check this value.

Who stamped the freeze. Required before freezing.

Check this value.

Objection rule

Write down, before the run, what happens when the numbers pass and someone says the result feels wrong: who may raise an objection, what evidence they must bring, who rules on it, and by when. An upheld objection holds the release; the gate records the outcome.

Roles or individuals allowed to object after the numbers are in.

Check this value.

What the objector must bring. A name and a concrete observation, or a worked counter-example.

Check this value.

The person or role who upholds or dismisses it.

Check this value.

How long the ruling may take, counted from when the objection is raised.

Check this value.

Check this value.

Criteria mapping

Criterion Form Reversibility Blast radius Gate rule On fail
Severe failure rate under 0.1 percent Bounded tolerance (rate upper bound) R3: Point of no return. Nothing undoes it. B3: Outside the organization. Customers, partners, the public, regulated populations. Floor Hold
Core pass rate above 92 percent Threshold (mandatory floor) R1: The actor can undo it. Version history, a restorable delete, an idempotent re-apply. B1: One team, one internal system, one department process. Floor Hold
Human review of every irreversible case Review requirement (evidence) R3: Point of no return. Nothing undoes it. B3: Outside the organization. Customers, partners, the public, regulated populations. Evidence Hold

Freezing records the date, who froze it, and a fingerprint of the policy text. It cannot prove when anyone first saw the data.

Export the policy

Reads an acceptance criteria policy saved from this page as JSON, version 1.x. Restores the form, the freeze and the frozen copy.

Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.

Export rules as a release gate spec
How this is calculated

Sample-size calculation

The required sample size is the smallest n such that the upper bound at zero failures is at most the target rate r, under the criterion's own interval method, sidedness and confidence level. Alpha is 1 minus the confidence level for a one-sided bound, and half of that for a two-sided interval, so it is 0.05 at the default. For Clopper-Pearson, n equals the ceiling of the natural log of alpha divided by the natural log of (1 minus r). At the default settings, the Clopper-Pearson n sits just under the "3 divided by r" mnemonic (2995 against 3000 at a target rate of 0.001), which is slightly conservative. For Wilson, n equals the ceiling of z-squared times (1 minus r) divided by r, where z is the normal quantile at 1 minus alpha. A criterion that names its own interval method shows no required n; the gate reads whatever interval you bring. Each criterion's card and both exports name its sidedness, level and method as one phrase, such as "one-sided 95% Clopper-Pearson".

Form-to-rule mapping

Each of the ten criterion forms maps to one of the five rule kinds the release gate designer reads (must pass, floor, regression, budget, evidence) with a fixed on-fail action and a set of fields the gate rule reads. The two risk axes (reversibility and blast radius) are never multiplied or averaged.

Criterion form Rule kind Mandatory On fail Fields read
Blocker (zero tolerance, rollback) Must pass Yes Roll back none
Must-pass (zero tolerance, hold) Must pass Yes Hold none
Threshold (mandatory floor) Floor Yes Hold direction, threshold
Target (advisory) Floor No Hold direction, threshold
Slice minimum Floor Yes Hold direction, threshold, suite
Guardrail (budget cap) Budget Yes Hold threshold
Bounded tolerance (rate upper bound) Floor Yes Hold direction, threshold
Statistical condition (regression test) Regression Yes Hold tolerance, alpha, baseline
Review requirement (evidence) Evidence Yes Hold artifact
Rollback trigger Must pass Yes Roll back none

Freeze fingerprint

The freeze fingerprint is a SHA-256 hash of a canonical text. The canonical text starts with the schema line, then the JSON body with keys sorted, criteria sorted by id, whitespace collapsed and trimmed, undefined written as null, and the freeze block itself excluded. The first 12 hex characters of the hash are shown on the page. Loading a saved policy re-hashes the frozen copy against the stored fingerprint; if they disagree, the stamp is not trusted.

Questions

Why does this tool exist alongside the release gate?

The release gate designer runs the rule against numbers. This tool captures why the rule is set to that number, who decided it, and how many cases the claim needs. A threshold without a stated reason is a threshold nobody can argue with or update.

What happens when a criterion has no sample size?

It lands in the unresolved list at the top. The tool still exports the rule, but the policy document records the gap.

How does the release gate spec export work?

The JSON export writes a release gate spec document with a data payload shaped for the release gate designer's input. Save the file, open the release gate designer, and import it there. The import message records that the thresholds came from this policy, and when the policy was frozen, the decision memo also carries its freeze date and fingerprint; review each threshold before locking it in.

How does the gate handle a bounded-tolerance criterion?

The gate reads the target rate as a floor with "at most" direction, and evaluates it from your count of cases. With the criterion's interval method and sidedness, the gate builds the interval and tests whether the bound clears the bar. Without a count, the rule is not evaluated and the gate holds. The exported rule carries the method, sidedness and confidence level so the gate applies the same test you chose here.

What about floor and threshold criteria?

Floor criteria (threshold, target, slice-minimum) carry a per-criterion interval method, sidedness and confidence level. When the gate receives a count, it builds the interval and tests the bound against the bar. A value typed alone on the gate is a point check.

The comparison of a candidate's pass rate against a baseline's is a paired test when both runs score the same cases, or a two-proportion test when the case sets differ. Choose which design fits before exporting.

Why do one-sided and two-sided disagree on the same numbers?

A one-sided 95% bound is one edge of the two-sided 90% interval, and a two-sided 95% interval is wider. On 12 failures in 400 against a bar of 0.05, the one-sided 95% Clopper-Pearson upper bound is 0.048, which clears, while the two-sided 95% upper bound is 0.052, which does not. The choice decides the verdict whenever the rate sits close to the bar.

What does freezing prove?

Freezing records a date from this device's clock, the name in "Frozen by", and a SHA-256 fingerprint of the policy text. If the frozen copy in a saved file is edited later and its fingerprint is left as it was, the two no longer match, and the page says the stamp is not trusted. The date comes from this device, nobody else signs it, and anyone with the file can recompute the fingerprint, so freezing cannot prove when anyone first saw the eval data. An edited-after-freeze flag appears if any field changed after the stamp.

What is the objection rule for?

It records who may object after the numbers are in, what evidence they must bring, who rules on it, and by when. An upheld objection holds the release on the gate page, whatever the verdict. It exists because criteria written before the run cannot cover every failure the run reveals, and a team member who sees something the criteria did not cover needs a path to say so.