Coverage conditioning

A system that declines the hard half of your test set can post a better score than one that attempted all of it. Coverage conditioning is that dependence: a published rate describes whichever requests survived to be scored, and it distorts most when survival correlates with the very property the rate is meant to measure. The declined requests never enter the arithmetic, so the rate that reaches the page describes the questions the system was willing to take. Provider refusals, output-cap truncations, timeouts, and rate limits all thin the pool this way, and any of them can thin it non-randomly. Our pre-registered study of routing and refusals publishes a capability rate computed over seven non-refused trials, which is this conditioning stated in numbers.

The diagnosis needs a companion measurement. Answer coverage is the fraction that says how much of the intended sample survived, and a rate published without it hands a reader a number whose base is invisible. The reporting protocol the term was coined in prints that fraction beside every score on its scoreboard, so a reader never has to work out what a score was computed over.

It belongs beside construct validity as a question about meaning. A rate can be reproducible, carry an interval, and still describe the answered subset rather than the system. Read coverage per task family, because the ordering between systems can invert between families and a pooled figure hides the inversion.

Where a family returned nothing at all, report the empty denominator with its request count and its reason, and do not read it as a zero rate.