LLM Judge Position Bias Calculator (Swap-Consistency Test)

A pairwise judge is asked which of two answers is better. Show the same two answers in the other order and a judge with no position bias returns the same verdict. This tool runs that swap experiment: each of n pairs is scored once as (A, B) and once as (B, A). A pair is consistent when the same content answer wins both times, and flipped when the winner moves with the order. It reports two numbers, each with a confidence interval: the swap-consistency rate, and, when you record which side each flip favored, the directional lean among the flips. This is the position gate from the judge-bias checklist; the wider taxonomy of judge biases sits at LLM-as-a-judge bias.

Swap-consistency rate and its interval. The point estimate is rate = consistent / n. Around it the tool reports a Wilson score interval at your confidence level, treating each pair as one Bernoulli trial that either survives the swap or does not. With z the two-sided normal quantile:

center = (p̂ + z²/2n) / (1 + z²/n)

margin = (z / (1 + z²/n)) · √(p̂(1−p̂)/n + z²/4n²)

Wilson stays inside 0% and 100% and returns a real width even at a 0% or 100% observed rate, which is why it is the default here rather than the Wald normal approximation. The same interval math backs the pass-rate confidence interval calculator.

Position preference among the flips. A flip has a direction: it can keep picking whichever answer was shown first (primacy) or whichever was shown second (recency). Among the m flips you classify by direction, the tool reports the first-position share firstFavored / m with its Wilson interval, and tests it against an even split with an exact two-sided binomial test. A judge whose flips are pure noise splits them evenly, so a p-value below your alpha is evidence of a directional lean. The tail probabilities use the exact binomial-Beta identity P(X ≤ x) = I₀.₅(m − x, x + 1), doubled at the smaller tail and capped at 1; because the null p = 0.5 is symmetric, doubling equals summing every outcome at least as extreme.

Worked example. Take n = 120 pairs, 96 consistent, and of the 24 flips, 17 favor the first position and 7 favor the second, at 95% (z ≈ 1.9600, z² = 3.8415). Swap-consistency rate = 96 / 120 = 0.8000. center = (0.8000 + 3.8415/240) / (1 + 3.8415/120) = 0.816006 / 1.032013 = 0.79070; margin = (1.9600 / 1.032013) · √(0.8·0.2/120 + 3.8415/57600) = 1.89920 · √0.0014000 = 0.07107. The interval is 0.79070 ± 0.07107, or 72.0% to 86.2%. A point estimate of 80% lands in the moderate band. For direction, m = 24 and the first-position share is 17/24 = 0.7083, Wilson interval 50.8% to 85.1%. The exact two-sided binomial p-value for 17 of 24 against 0.5 is 0.064. That sits above 0.05, so at 95% the lean toward the first position is not yet significant: 24 flips is too small a sample to separate a real primacy effect from noise.

The sensitivity band, and what it assumes. The tool labels the swap-consistency point estimate as low sensitivity at 90% or above, moderate from 75% to 90%, and high below 75%. These thresholds are the lane's rule of thumb, not a standardized cutoff. The literature reports judge swap-consistency without an agreed threshold: a strong judge held its verdict on more than 60% of order-swapped pairs in the MT-Bench study (Zheng et al., 2023), and position-robustness ran from about 0.57 to 0.83 across models in the CALM framework (Ye et al., 2024, preprint). Read the interval, not the band, as the number to report. Every interval here also assumes pairs are independent draws; correlated prompts or a repeated item set make the true uncertainty wider than any bound shown.

Formula: rate = consistent / n; Wilson bound (p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)

See Sources below