Glossary
Flaky test (LLM features)
A flaky test is the tester's word for a case whose single run cannot settle a verdict. On an LLM feature, the case has a pass rate across reruns of the same version and input, and the job is to measure that rate.
In deterministic software a flaky test is usually a broken fixture or a race condition, and the standard move is to quarantine it and fix the test. On an LLM feature, even at settings meant to be deterministic, Atil et al. measured accuracy variations of up to 15% across 10 runs of the same inputs, and Yuan et al. found up to 9% accuracy variation from GPU count, type and evaluation batch size. A single red run is a sample of one.
Reading a flaky LLM case as a rate to measure is our synthesis. Research on how many reruns a reliable eval needs quantifies what it takes to see a real drop through that noise.
With every case at a true 99% pass rate and 200 independent cases in the suite, the suite comes back all green about 13% of the time and shows 2 red cases on average. A case that passed 7 of 10 reruns has a 95% interval of about 40 to 89%, wide enough that a bar anywhere in that range stays unsettled after just ten reruns.
What to do with a flaky test case
Rerun the case and open the repeated-run variance planner’s “Reruns of one failing case” mode to see how many reruns settle the question.
Open a run-rows file in the repeated-run variance planner’s “Run rows” mode for a variance estimate across every case in the suite at once.
To see how many regressions retries can mask, open the reliability-at-k estimator; retries that pass a case on any attempt can hide a drop in that case’s rate.
One flip alone does not make a case flaky. Keep running a quarantined case as a watched case under the CI gate contract, so you still measure its rate while it no longer blocks a release.
Errored or empty rows come from the test harness or an API. The results auditor separates harness errors from genuine failures before any rate is reported.
A regression is a drop in a case’s or suite’s rate larger than what reruns alone produce; a move between two single runs is not one. Paired runs on the same cases separate a real change from rerun noise.