Glossary
Coverage conditioning
Coverage conditioning is the dependence of a published rate on which requests came back with an answer, and it bites when membership of that answered subset correlates with the property the rate is meant to measure.
A system that declines the hard half of your test set can post a better score than one that attempted all of it. Coverage conditioning is that dependence: a published rate describes whichever requests survived to be scored, and it distorts most when survival correlates with the very property the rate is meant to measure. The declined requests never enter the arithmetic, so the rate that reaches the page describes the questions the system was willing to take. Provider refusals, output-cap truncations, timeouts, and rate limits all thin the pool this way, and any of them can thin it non-randomly. Our pre-registered study of routing and refusals publishes a capability rate computed over seven non-refused trials, which is this conditioning stated in numbers. Deliberate thinning behaves the same way: a confidence gate that sends the weakest items to a person conditions the published rate on whatever the gate let through. Sweeping coverage against residual error over your own records is how you see what each human review threshold withholds and what it still ships wrong.
The diagnosis needs a companion measurement. Answer coverage is the fraction that says how much of the intended sample survived, and a rate published without it hands a reader a number whose base is invisible. The reporting protocol the term was coined in prints that fraction beside every score on its scoreboard, so a reader never has to work out what a score was computed over.
It belongs beside construct validity as a question about meaning. A rate can be reproducible, carry an interval, and still describe the answered subset rather than the system. Read coverage per task family, because the ordering between systems can invert between families and a pooled figure hides the inversion. Splitting the set into its slices and counting each one is how you find the families thin enough for that to happen.
Where a family returned nothing at all, report the empty denominator with its request count and its reason, and do not read it as a zero rate.