Glossary
Mutation score (LLM test suites)
A mutation score is the share of deliberately planted faults a test suite catches (kills, in the field's word), counted as killed faults over faults planted, with an interval.
Testers call a caught planted fault “killed.” A suite that killed 17 of 20 planted faults scores 85%, with a 95% confidence interval of about 64 to 95%. The suite’s true kill rate could sit anywhere in that range. On the pass-rate page’s “Interval” mode, enter killed faults as “Passes (c)” and planted faults as “Total runs (n).”
Even a perfect 20 of 20 killed leaves a one-sided 95% lower bound of about 86%, which the “Runs for a claim” mode prints when you enter 20 “Clean runs observed” of 20 “Total runs observed” at 95% one-sided confidence. The “Interval” mode’s two-sided 95% bound is about 84% for 20 of 20.
Planting faults in an LLM feature
The method of breaking a system on purpose and checking whether the suite notices is described in the guide Prove your eval suite can fail. That guide’s five checks plant a fault (a known bug replayed, a constant output, an empty agent, answers moved to the wrong case) or test the grader (the same outputs judged twice). For each planted fault, record whether the suite’s tests caught it.
A fault the model works around changes nothing: in the classic literature this is an equivalent mutant, left out of the score once found. A fault killed on one run can survive a rerun of the same version, because the suite’s test may pass the rerun. Declaring a fault killed only after the test catches it on a stated number of reruns is our synthesis; the number depends on the variance the suite shows across those reruns. High variance needs more reruns before declaring a fault killed or surviving.
Mutation score and fault injection
Fault injection seeds a chosen fault at a known point and measures how far it travels before something stops it. Planting a fault for a mutation score tests whether the test suite detects it. Injection measures the system’s response to a known fault; the mutation score measures the test suite’s ability to notice the fault at all. The mutation score is a statistic reported with an interval on the defined set of planted faults. A higher count of planted faults narrows the interval.