Construct validity (benchmarks)

Construct validity asks whether a benchmark measures the capability it names, or a proxy a system can score high on without possessing it. It comes from measurement theory, which applies to any test read as a measure of an attribute that is not “operationally defined” (Cronbach & Meehl, 1955, peer-reviewed). A review of 445 LLM benchmarks found patterns that undermine their claims’ validity (Bean et al., Measuring what Matters, NeurIPS 2025, peer-reviewed).

The operative test is simple. A benchmark is construct-valid only when a top score cannot be earned without the capability it advertises.

From psychometrics to benchmark validity

Cronbach and Meehl defined construct validity for psychological tests, where the attribute under test, intelligence or anxiety, has no direct operational definition and can only be inferred from how scores behave. A benchmark inherits the same problem. “Reasoning,” “agentic capability,” and “helpfulness” are constructs in exactly this sense, so a leaderboard number is valid only when it behaves like the capability it names. When a score can be lifted by a shortcut that leaves the capability untouched, the construct and the measurement have come apart. That gap is what a gamed benchmark exploits.

Consider a reliability benchmark that ranks agents by single-run pass rate. A system that passes once by luck, then fails its next two runs, posts the same number as one that passes every time: the score credits a lucky draw and calls it reliability. pass^k counts only success held across all k runs of a task, and its suite-level aggregate, reliability@k, ties the score to the construct.

This differs from eval reproducibility, which asks only whether re-running returns the same number; a benchmark can be perfectly reproducible and still measure the wrong thing. Check what a benchmark measures before trusting its ranking; see AI agent evaluation for where this check sits among eval-rigor tests.

How to check a benchmark’s construct validity

For a reader choosing between systems, the check is one question: could a system top this benchmark without holding the capability the benchmark advertises? If it could, the ranking says less than it appears to. For a team building or auditing a benchmark, the same question becomes a procedure: enumerate the ways a system could raise the score, then confirm each one still requires the target capability. A reliability benchmark, for instance, passes this check only when its score requires success held across every run, the quantity the reliability@k estimator computes. The prompt belongs on that list too: reword the task without changing what it asks, rescore the same items, and you can check whether the score follows the capability or the phrasing. Two further entries on that list are properties of the file rather than the model. If the items already sat in training data, a top score needs memory and not capability, which is what contamination does to a benchmark’s claim and what an exact overlap check across the train and test splits settles. And a set padded with restatements of the same item measures less than its row count implies, so normalized near-duplicate detection belongs in the same pass.