Glossary
Exit criteria (LLM features)
Exit criteria are the conditions that must hold to call a test activity done. For an LLM feature, each criterion names a rate, an interval method, a confidence level and which side it protects, written before the run.
Agile teams call them the Definition of Done; release meetings call them go/no-go criteria. For an LLM feature, the criterion names a rate (the pass rate under a stated grader), an interval method, a confidence level and which side the criterion protects, written down before the test run starts. The acceptance criteria builder is where you write them.
Two ways to judge one criterion
ISTQB’s AI testing syllabus, CT-AI v2.0, section 6.1.3, works through an example: a 98% accuracy target with a margin of plus or minus 4 percentage points at 95% confidence, requiring 601 test cases and at least 589 passes. The 601 figure assumes the true rate could be anywhere from 0 to 100%, a worst case; at an observed 98% the margin is only about 1.1 points, so a 4-point margin is a conservative bound. And the result depends on how the bar is checked.
Judged on the point estimate, a system whose true pass rate is exactly 98% passes about 57% of the time: almost a coin flip. Judged on the one-sided 95% lower bound, it needs 595 of 601 passes and gets through about 4% of the time. Which side the criterion protects is the release decision. The pass-rate page’s “Runs for a claim” mode checks a run’s counts against a stated bar.
A criterion judged on an interval can land in a third state, inconclusive: the interval straddles the bar, and the true rate could sit on either side. More cases narrow the interval.
A fixed sampling plan (accept the batch with at most c failures in n checked) has no such state and carries two stated risks: rejecting a good batch (the producer’s) and accepting a bad one (the consumer’s). A sequential rule has its own error rates; the fixed-sample interval at its stopping count shows the precision of that estimate.
Turn the criterion into go/no-go rules on the release gate designer. Size the test set from the margin on the golden-set planner. In the readout builder’s test report section, the team sets each criterion’s status and types a reason; a criterion whose interval straddles the bar is recorded as “Across the bar” only when the team enters it that way.