Verbosity bias (LLM judges)

Verbosity bias is an LLM judge’s tendency to reward answer length on its own, scoring a padded response above a shorter one that is equally correct or better. It sits next to position bias, which turns on where a response appears in the prompt, and it survives every fix for position, because swapping two candidates into the opposite order leaves both of their lengths exactly where they were.

Verbosity bias, position bias and self-preference bias are three of the distortions a pairwise judge carries into a published ranking, and each has its own detection test in our map of judge biases and the tests that catch them. The six judge-bias terms and the defect kind each one names sit together in one table. Measure length before publishing any judge-scored ranking; a win rate that moves when you hold length fixed was measuring words.

A preference for longer answers counts as bias only against a reference the judge is meant to reproduce, which means human labels or a verifiable ground truth. Given that reference, the measurement compares two verdicts on the same content: the judge’s call on an answer, and its call on a variant that runs longer and says no more. Because both verdicts land on the same items, the quantity is a proportion over pairs, and a proportion over pairs carries a confidence interval rather than a bare percentage.

How to calculate verbosity bias

Take the pairs your judge already decided, build a longer variant of one side that adds tokens and no information, then re-run the judge and count how often the verdict moves to the inflated answer. The flip rate is that count over the number of pairs you padded. The MT-Bench paper built the construction this description follows (Zheng et al., 2023, NeurIPS Datasets and Benchmarks). They took 23 model answers containing numbered lists, had GPT-4 rephrase the list items without adding information, inserted the rephrased items at the front, and recorded how often each judge then called the inflated version better. Claude-v1 and GPT-3.5 preferred the padded answer on 91.3% of those 23, and GPT-4 preferred it on 8.7%.

Twenty-three answers is a thin denominator and the paper reports those rates without intervals, so read the ordering rather than the decimals.

The construction only holds if the padding genuinely adds nothing, since a longer answer that also covers more ground is a better answer and a judge preferring it has done its job. Report the flip rate as k of n with an interval around it, which the pass-rate interval calculator computes as a Wilson bound. The paired version asks whether the ranking itself changed: score every decisive pair raw, score it again against a length-matched rewrite, and run the paired test for eval deltas on the disagreeing items, which the McNemar calculator computes. Both constructions assume a pairwise judge, and under pointwise scoring the same experiment becomes a before-and-after on one answer’s score, where what you report is a mean score shift rather than a flip rate.

A second construction removes length statistically instead of experimentally. Length-controlled AlpacaEval fits a generalized linear model predicting the judge’s preference from the length difference, then asks that model what it would have preferred had both answers been the same length. Across the benchmark’s 805 instructions, varying only the verbosity instruction in the prompt moved the baseline model’s raw win rate from 22.9% to 64.3%, while the length-controlled win rate for that same model moved from 41.9% to 51.6% (Dubois et al., peer-reviewed, COLM 2024). The same paper reports the correlation with a human-preference leaderboard rising from 0.94 to 0.98, a point pair it gives without intervals. The place to instrument this in a running eval is the judge call itself. Log the token count of both candidates beside every verdict, and the association between length difference and winner becomes a standing monitor instead of a scheduled audit.

A measured flip rate doesn’t correct the score by itself. Correction comes either from rescoring on length-matched pairs, or from treating the judge as an imperfect classifier and adjusting its raw pass rate by measured sensitivity and specificity, which the judge bias correction calculator does. That second route holds only when the calibration set contains answers at the lengths you actually ship. Whether the judge’s confidence tracks its accuracy is a separate axis from bias, and our guide to whether an LLM judge is reliable enough to publish from sets out all three.

Verbosity bias vs position bias

Position bias is a judge preferring whichever candidate occupies a particular slot in the prompt, usually the first one it reads, independent of what either candidate says. Verbosity bias attaches to a property of the response instead, its length in tokens. Verbosity bias and position bias vary independently of each other. A judge can be perfectly swap-consistent, returning the same winner whichever order you present, and still hand that win to the longer answer nearly every time, which is why the swap-consistency test clears order and says nothing at all about length. A judge can also be length-neutral and order-driven, picking slot one at a rate its content cannot justify while ignoring how long either answer runs. Audit for each one separately, because the corrections don’t overlap: order is handled by presenting each pair twice and discarding the inconsistent verdicts, length by matching it across candidates or regressing it away afterwards.

Verbosity bias vs self-preference bias

Self-preference bias is a judge scoring output from its own model family above output from another family, with quality held constant. The confound with verbosity is real, and it runs in one direction: models differ systematically in how long they answer, so a judge grading its own family’s characteristically longer output looks self-preferring when the only thing it rewarded was length. Separating the two takes one of two moves. Hold length fixed across the candidates before you mask model identity, so the identity effect is measured on length-matched pairs. Or fit both terms in one regression, the way length-controlled scoring already fits length, and read the identity coefficient with length partialed out. Measure each one anyway, because neither fix removes the other: a judge can prefer its own family’s shorter answer, which is self-preference net of length, and it can prefer a rival’s padded answer, which is verbosity net of identity.

The same length bias costs more in a reward model than in a judge

Reinforcement learning from human feedback carries the same statistical shape under a different name. There, length bias, sometimes called length hacking, describes a reward model assigning higher reward to longer completions, and a policy trained against that reward model learns to pad. The two names sit over identical arithmetic, a preference correlated with token count rather than with quality. The artifact differs, and so does the size of the damage. A length-biased reward model changes what the system produces on every request from then on, while a length-biased judge changes what a leaderboard says and rescoring recovers it. Inherit a padded policy, then grade it with a length-biased judge, and you have a system that pads and a score that rises because it padded.

Verbosity bias and self-preference bias both attach to properties an order swap leaves untouched, so a length-matched re-score belongs in the gate order before any ranking is quotable. The runnable version of that gate order, with a pass criterion and a calculator per check, is in the checklist that gates a judge-scored ranking, and the statistics for reporting a corrected number afterwards are in bias-correcting a judge eval before you publish it. If you have budget for only one of those checks, then make it the length-matched re-score, since no amount of order swapping will surface what it catches.