Position bias (LLM judges)

Position bias in an LLM judge is a preference for whichever response occupies a particular slot in the prompt, first or last, that persists when the same two responses change places and nothing about their content changes. The measured form of it is a rate: of the pairs you scored twice, once in each order, the share whose winner changed when the order changed, together with the direction those changes lean.

The direction of those flips needs a line of its own. A judge whose flips split evenly across the two slots is unstable, one whose flips land mostly on the same slot is tilted, and the two conditions call for different corrections.

A pair either flips on the swap or holds, so the flip rate is a proportion over paired trials and inherits the sampling error of one. Report it with an interval that bounds the estimate and with the count of decisive pairs behind it. A judge that flips on 3 of 20 close calls and one that flips on 30 of 200 share a point estimate while making very different claims on your ranking.

Position bias lives inside the protocol that puts two candidates in one prompt, where one of them has to go first. It sits in the per-bias map of what moves a judge alongside verbosity bias, which keys on length, and a judge’s pull toward its own model family, which keys on authorship. The vocabulary for judge bias sets out all six terms with the test each one takes. Position bias keys on the slot alone, which is why it comes first in the runnable gate order: the test needs no extra labeling and no second rubric, only the same pairs run twice. Measure it before publishing any ranking whose top two sit close together, because a margin narrower than the flip rate’s upper bound is a margin the ordering could have produced.

How to calculate position bias

Score each decisive pair twice, once as (A, B) and once as (B, A), holding the prompt, the rubric, the judge model and the sampling temperature fixed across both passes. Divide the pairs whose winner survived the swap by the total pairs judged in both orders, which gives swap-consistency; its complement is the flip rate. Then classify the flips by direction, counting how many kept choosing the response shown first and how many kept choosing the one shown second, and divide the first-slot flips by the flips you were able to classify. A judge with no directional tilt splits those evenly, so an exact binomial test against a half tells you whether the skew you measured is larger than chance supplies at your pair count.

Report both rates with a Wilson interval on the proportion, and report each denominator separately, since the directional rate is computed over flips alone and rests on a much smaller base. The swap-consistency test for pairwise judges runs the whole computation, binomial test included. Instrument it where the ranking is actually decided rather than uniformly across a suite. The pairs worth swapping are the ones whose margin is small enough that a flipped verdict would reorder the leaderboard, and a paired test on the discordant pairs tells you whether the two orders disagree more than sampling noise accounts for.

Position bias vs verbosity bias

Verbosity bias is a judge’s preference for the longer answer when the extra length carries no extra correct content, so a padded response outscores a terse one making the same claims. Position bias is the same class of surface effect keyed on slot instead of length. Either effect can be large while the other is absent entirely. Hold length equal across a pair and swap the order, and any remaining flip is positional, because the only thing that changed was which answer the judge read first. Hold the order fixed and lengthen one candidate without adding substance, and any score change is a length effect that no amount of order rotation will remove.

The two are easy to confound on a swap set built from whatever lengths your candidates happened to have. A systematic study of 15 judges on MTBench and DevBench found position bias only weakly related to prompt length and strongly related to the quality gap between the two solutions (Shi et al., arXiv:2406.07791, peer-reviewed, AACL-IJCNLP 2025). Match lengths to keep the order measurement about order, and expect the positional effect to concentrate on the near-ties. The length-controlled pairs that isolate a preference for word count deserve a pass of their own, run separately from the swap set.

Position bias vs judge instability

Repetition instability is a judge changing its verdict on a rerun with the order left alone, which is sampling noise inside the evaluator. Position bias is the share of verdict changes attributable to order. A judge can be perfectly stable on repeats and still tilt hard toward the first slot, in which case every rerun reproduces the same wrong winner and repetition never surfaces the error. A judge can also be unstable on repeats and even-handed on order, so the flips you see across swaps are the flips you would have seen across identical reruns, and reading them as positional assigns noise to a mechanism.

Shi and colleagues define repetition stability alongside position consistency and preference fairness, which gives an order effect a stability baseline to be read against. Build the same baseline before correcting anything: run a subset of pairs twice in the same order, take that flip rate as your floor, and treat only the excess as positional. Correcting for order while the judge cannot reproduce its own verdicts gives you a stable ranking of nothing, so whether a rerun lands in the same place is the prior question, and a judge that clears both can still be miscalibrated in how confident it sounds.

Position bias in ranking and survey research

Two older fields own the phrase, and both meanings are legitimate.

In information retrieval, position bias names the effect of rank on click probability: users click higher-ranked results more often whatever the relevance, so a click log is a biased sample of preference until a click model corrects it. Craswell and colleagues found in 2008 that a cascade account, where a user reads top to bottom and stops at the first result worth stopping for, explained the early ranks best. Survey methodology uses the phrase for response-order effects, where the slot an option occupies among the choices offered shifts how often it gets picked. Krosnick and Alwin’s 1987 study of those effects found a primacy pattern, strongest among respondents lower in cognitive sophistication. Both fields describe a chooser weighting the slot over the content, which is what an LLM judge is doing too.

The single position-bias number does not carry across. The retrieval literature estimates a propensity curve over many ranks under one fixed presentation. A pairwise judge has two slots and a presentation you control, so the quantity that survives the move is a conditional flip rate: measured on the pairs whose margin is small enough to matter, at the prompt and rubric you actually ship, with the interval that bounds it. A position-bias figure quoted with no denominator and no condition describes a different experiment from the one your ranking depends on.

Position bias and swap-consistency are one measurement read from opposite ends, and both get held against your smallest win margin. Wang and colleagues showed how far the unchecked version reaches, reordering candidates until Vicuna-13B beat ChatGPT on 66 of 80 tested queries with ChatGPT itself acting as the evaluator (Large Language Models are not Fair Evaluators, arXiv:2305.17926, peer-reviewed, ACL 2024). The three axes a judge has to clear before its scores are worth publishing, agreement and calibration alongside bias, are laid out in the case for characterizing the evaluator first. Scoring every pair twice doubles the judge bill, so the swap set earns its cost where the margins are thin enough that a flipped verdict would reorder the table. If your leaderboard has one clear winner and a long tail behind it, spend the extra calls on the contested rows and leave the tail scored once.