TOOL | Judge reliability
LLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.
Position bias is a pairwise judge favoring whichever answer it saw first or last. Score every pair in both orders, enter the counts, and this page returns the swap-consistency rate with a Wilson interval and an exact binomial test on which side the flips lean. Run it before trusting a pairwise win rate. It comes from our checklist that gates a judge ranking; the wider taxonomy of judge biases sets it beside verbosity bias and self-preference.
Showing your last valid result. Update the inputs above to recompute.
Swap-consistency interval
72.0% to 86.2%
95% two-sided. Observed 80.0% (96 of 120 pairs held their verdict); 24 flipped.
Swap consistency
| Metric | Value |
|---|---|
| Swap-consistency rate | 80.0% (96 of 120) |
| 95% Wilson interval | 72.0% to 86.2% |
| Flipped pairs | 24 of 120 (20.0%) |
Position preference among flips
| Metric | Value |
|---|---|
| Classified flips (both directions) | 24 |
| First-position share | 70.8% (17 of 24) |
| 95% Wilson interval | 50.8% to 85.1% |
| Exact binomial p vs even split | 0.064 |
How this is calculated
A pairwise judge is asked which of two answers is better. Show the same two answers in the other order and a judge with no position bias returns the same verdict. This tool runs that swap experiment: each of n pairs is scored once as (A, B) and once as (B, A). A pair is consistent when the same content answer wins both times, and flipped when the winner moves with the order. It reports two numbers, each with a confidence interval: the swap-consistency rate, and, when you record which side each flip favored, the directional lean among the flips. This is the position gate from the judge-bias checklist; the wider taxonomy of judge biases sits at LLM-as-a-judge bias.
Swap-consistency rate and its interval. The point estimate is
rate = consistent / n. Around it the tool reports a
Wilson score interval at your confidence level, treating each pair
as one Bernoulli trial that either survives the swap or does not. With z the
two-sided normal quantile:
center = (p̂ + z²/2n) / (1 + z²/n)
margin = (z / (1 + z²/n)) · √(p̂(1−p̂)/n + z²/4n²)
Wilson stays inside 0% and 100% and returns a real width even at a 0% or 100% observed rate, which is why it is the default here rather than the Wald normal approximation. The same interval math backs the pass-rate confidence interval calculator.
Position preference among the flips. A flip has a direction: it
can keep picking whichever answer was shown first (primacy) or whichever was shown
second (recency). Among the m flips you classify by direction, the tool
reports the share of flips that favored the first position, out of the m
flips classified, with its Wilson interval, and tests it against an even split with an
exact two-sided binomial test. A judge whose flips are pure noise
splits them evenly, so a p-value below your alpha is evidence of a directional
lean. The tail probabilities use the exact binomial-Beta identity
P(X ≤ x) = I₀.₅(m − x, x + 1), doubled at the smaller tail and capped
at 1; because the null p = 0.5 is symmetric, doubling equals summing every outcome
at least as extreme.
Worked example. Take n = 120 pairs, 96 consistent, and of the 24 flips, 17 favor the first position and 7 favor the second, at 95% (z ≈ 1.9600, z² = 3.8415). Swap-consistency rate = 96 / 120 = 0.8000. center = (0.8000 + 3.8415/240) / (1 + 3.8415/120) = 0.816006 / 1.032013 = 0.79070; margin = (1.9600 / 1.032013) · √(0.8·0.2/120 + 3.8415/57600) = 1.89920 · √0.0014000 = 0.07107. The interval is 0.79070 ± 0.07107, or 72.0% to 86.2%. A point estimate of 80% lands in the moderate band. For direction, m = 24 and the first-position share is 17/24 = 0.7083, Wilson interval 50.8% to 85.1%. The exact two-sided binomial p-value for 17 of 24 against 0.5 is 0.064. That sits above 0.05, so at 95% the lean toward the first position is not yet significant: 24 flips is too small a sample to separate a real primacy effect from noise.
The sensitivity band, and what it assumes. The tool labels the swap-consistency point estimate as low sensitivity at 90% or above, moderate from 75% to 90%, and high below 75%. These thresholds are the lane's rule of thumb, not a standardized cutoff. The literature reports judge swap-consistency without an agreed threshold: a strong judge held its verdict on more than 60% of order-swapped pairs in the MT-Bench study (Zheng et al., 2023), and position-robustness ran from about 0.57 to 0.83 across models in the CALM framework (Ye et al., 2024, preprint). Read the interval, not the band, as the number to report. Every interval here also assumes pairs are independent draws; correlated prompts or a repeated item set make the true uncertainty wider than any bound shown.
The review sheet. For each case, the page draws one coin from the seeded stream to decide which system appears on the left in the first presentation, then builds the second presentation as the exact swap. Both presentations carry a task id assigned after the full list of 2n tasks is shuffled, so the task ids leak nothing about which case they belong to or which system sits on which side.
The tally. Join the filled sheet to the key on the task id. The sides swap between the two presentations, so the same candidate winning both times is the same as opposite sides winning both times. The counting rules are in "The blind review sheet for the swap run", the collapsed section below the Compute button.
Formula: rate = consistent / n; Wilson bound (p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)
Questions
What counts as a consistent pair?
A pair is consistent when the same content answer wins in both answer orders. If the winner changes when you swap (A, B) to (B, A), the pair flipped, and the flip is attributable to order rather than to the answers. Score every decisive pair both ways; ties or pairs you cannot classify by direction stay out of the position-preference count but still count toward n for swap-consistency.
What does the blind sheet actually hide?
The sheet strips the two system names you typed: nothing on it says which candidate produced which answer. The pasted answer references print verbatim, so they must not name or hint at a system. The order is randomized per case, so a reviewer who always picks the left slot splits the bias evenly. The two presentations of each case are shuffled apart, so they are not judged back to back. What the sheet cannot do: the reviewer sees the same two answers twice, because the swap test requires it. A shuffle spaces the repeats out, but it cannot stop a reviewer recognizing a pair and recalling their earlier verdict.
How many flips do I need to prove a directional lean?
Enough that the exact binomial test clears your alpha. A lean of 17 of 24 flips toward the first position looks strong at 71%, yet its two-sided p-value against an even split is 0.064, which does not reach the 5% level. The share can look skewed while the interval still spans an even split. Report the first-position share with its interval and the p-value together, and escalate the pairs that moved rather than a single headline percentage.
What swap-consistency rate is good enough?
There is no standardized pass line, so tie the answer to your decision. A ranking turns on its close pairs, so the useful test is whether the flip rate's upper interval bound sits below your smallest win margin. If it does, order is unlikely to decide a contested pair; if it exceeds that margin, correct for position by averaging both orders or dropping the pairs that flipped before you trust the ranking. The bands here are a reading aid, and the interval is the number that travels.
Sources
- Probable Inference, the Law of Succession, and Statistical Inference (Wilson score interval)
- The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial (exact binomial tail probabilities)
- Interval Estimation for a Binomial Proportion (why Wilson over the Wald approximation)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (position bias in pairwise judges)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (position-robustness rate range)