Recommended protocol
INSTRUMENT | Judge reliability
Pairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
9 cited sources
Turn pairwise LLM judge comparisons between models or prompts into win rates with Wilson intervals, Bradley-Terry strengths on the Elo scale, and a plain read on whether the advantage is real.
A pairwise win rate is the share of head-to-head comparisons won, ties aside. Paste wins, losses and ties from a judge or human votes for that rate, a Wilson interval, Elo-scale gap and Bradley-Terry strengths; ties take half credit. Run the position bias calculator first; a judge that flips with answer order has no usable rate. Built on our catalog of judge biases. Graded scores belong in the eval A/B comparator. Unsure which? Settle the protocol first.
Showing your last valid result. Update the inputs above to recompute.
System A win rate over System B, decided comparisons
57.0%
95% interval 47.2% to 66.3%. The interval includes 50%, so this advantage is not statistically separated at 95%.
Head-to-head readout
| Metric | Estimate | Interval (95%) |
|---|---|---|
| Win rate for A, decided comparisons | 57.0% | 47.2% to 66.3% |
| Tie-inclusive score for A | 56.4% | n/a |
| Elo-scale gap, A minus B | +44.5 | -17.6 to +105.9 |
| Exact sign test p-value, two-sided | 0.193 | n/a |
Ranking by Bradley-Terry strength
| Rank | System | W-L-T | Win rate | BT rating | Separated from next? |
|---|
Direct head-to-head records
| Pairing | W-L-T | Win rate | Interval | Separated? |
|---|
win rate = wins / decided; gap = 400·log10(creditA / creditB); P(i beats j) = pi_i / (pi_i + pi_j)How?
How this is calculated
Every comparison ends in a win, a loss, or a tie, and the tool reports three readings in rising order of assumption: a win rate with its interval, a strength gap on the Elo scale, and, for three or more systems, Bradley-Terry strengths fit to the full set of records.
Win rate and interval. The headline win rate is wins over decided comparisons, with a Wilson score interval at your chosen confidence level; Wilson is used because the textbook normal approximation misbehaves near 0, near 1, and at small counts. Ties are dropped from the rate, the interval, and the exact sign test, which is the standard sign-test convention; the tie-inclusive score counts each tie as half a win and is shown alongside. "Separated" means one specific thing here: the interval on the decided win rate excludes 50% at the chosen level. The exact two-sided sign test p-value is the significance counterpart of the same question, computed from the binomial distribution at even odds.
The Elo scale, without the ordering problem. Classic Elo updates ratings one game at a time, so the same set of games in a different order produces different ratings. For a fixed batch of comparisons the order-free estimate is the Bradley-Terry maximum likelihood fit, and that is what this tool computes, reported on the familiar Elo scale: a gap of d points implies an expected score of 1 / (1 + 10^(-d/400)), so 0 points means an even match and 400 points means winning about 10 of every 11 decided games. Chatbot Arena moved from online Elo to exactly this batch Bradley-Terry fit for the same reason. Ties grant each side half a win of credit, matching the Elo scoring rule, and ratings are mean-centered at 1000. The interval printed beside the two-system gap is the Wilson interval on the decided win rate carried through that same half-tie credit, holding the observed tie count fixed the way the sign test does, so the gap and its bounds always price the same quantity.
Bradley-Terry for several systems. The model gives each system a positive strength and sets the chance that i beats j to strength i over the sum of the two strengths. The fit uses the MM algorithm of Hunter (2004) on the half-tie credit matrix, iterated until the largest per-system log change drops below 1e-12, with a hard cap of 5,000 iterations reported honestly if hit. The fit exists only when the comparison graph is strongly connected in win credit (Ford 1957): an undefeated system, a winless system, or two groups never compared against each other have no finite maximum likelihood strengths, and the tool names the offending systems instead of printing a diverging number. Because raw win rates depend on who each system happened to play, the Bradley-Terry strength is the number to compare across an uneven schedule.
Worked example, two systems. A beats B in 57 comparisons, loses 43, with 10 ties. The decided win rate is 57/100 = 57.0%, Wilson 95% interval 47.2% to 66.3%. The interval includes 50%, so 110 comparisons at this split do not separate A from B; the exact sign test reads the same (p = 0.193). The credit odds (57 + 5) to (43 + 5) put the gap at +44.5 Elo-scale points, with an interval spanning -17.6 to +105.9.
Worked example, three systems. With candidate-a over candidate-b 55-45-12, candidate-a over baseline 74-22-8, and candidate-b over baseline 68-30-6, the fit puts the ratings at 1073, 1035, and 892. Both candidates are separated from the baseline by their direct records, but the 55-45 record between the two candidates is not separated at 95%, so the top of this ranking is one statistical tier, ordered by point estimate only.
Honest limits. Every interval treats comparisons as independent draws from one judging process. Comparisons that share items, prompts, or a judge are correlated, which makes the printed intervals narrower than the truth; a preference artifact in the judge shifts every number here, and a judge favoring its own model family (self-preference bias) produces a win rate measuring kinship as much as quality (the evidence is collected in the taxonomy of LLM judge biases, and the pre-ranking bias checklist is the gate to clear first), so run the position bias gate on the same judge first. Bradley-Terry additionally assumes one scalar strength per system, which forces transitivity; real preference data can be intransitive, and the fit will average that structure away without flagging it. Separation here is a statement about sampling noise on the direct record, and about nothing else.
Formula: win rate = wins / decided; gap = 400·log10(creditA / creditB); P(i beats j) = pi_i / (pi_i + pi_j)
Which protocol should you be scoring with?
This section picks how a judge is shown the work: one output at a time against a scale, or two side by side, and what each mode means is set out in the glossary. How the experiment around it is laid out, and whether the same cases go to every candidate, is a separate question from this one, and the chooser for paired or independent runs answers it.
Showing your last valid result. Update the inputs above to recompute.
Answer the six questions. The recommendation, and the rule that produced it, appear here.
Rule that fired
All nine rules, in the order they run. The one marked Fired is the recommendation; the rest show what stopped them.
| Rule | Fires when | Your answer | Outcome |
|---|
What each design costs per test case at your candidate count.
| Design | Judgments per case | Within your budget |
|---|
The sparse figure is one paper's result on text-quality assessment sets of several hundred candidates, where a ranking converged within about ten comparisons per candidate. It prices a full ranking built from selected pairs, so it is not a floor under the anchored design, and it is not a guarantee here. The paper is in the sources below.
On the calculator above, enter the comparisons and read the win rate with its interval.For the bar, turn one pass rate into the interval you read it against.To rank the candidates by those scores, compare two score sets side by side.
The nine rules, in the order they run (the ids are names, not positions)
The list is read top to bottom and the first rule whose conditions all hold is the one that answers. It is here before you enter anything, because a chooser that hides its rules is asking to be taken on trust.
- R0 Only one candidate. Pointwise
- R1 The criterion is checkable and every case has a reference answer. Pointwise
- R3 The number has to hold as a bar across future runs. Pointwise
- R4 The criterion is checkable. Pointwise
- R5 The criterion is partly checkable and every case has a reference answer. Pointwise
- R2 The number has two jobs, and the budget covers a grading pass plus a comparison pass. Both
- R6 The number only has to pick, and there are exactly two candidates. Pairwise
- R7 The number only has to pick, there are three or more candidates, and the budget covers one comparison per candidate. Pairwise
- R8 Two or more candidates. Nothing above claimed the answer. Pointwise
Questions
My model wins 55% of the time. Is that a real advantage?
Not on this evidence alone. At 55% over 200 decided comparisons the 95% Wilson interval runs from about 48% to 62%, which still contains 50%, so the record is equally consistent with two systems of the same strength. A true 55% edge needs the better part of a thousand decided comparisons before its interval clears an even split. Paste your own counts above and read the interval beside the rate: the tool states in plain words whether the two systems are separated at your chosen level. Our account of separating a real gain from run-to-run noise sets out the wider frame.
How many comparisons do I need for a reliable win rate?
More than most eval writeups collect. A 57-43 decided record over 100 comparisons is not separated from an even split at 95%, while 60-40 over the same 100 barely is. As a working floor, a true 60% winner needs on the order of 200 decided comparisons to clear 50% reliably (at 100 it is close to a coin flip whether the interval separates), and a true 55% winner needs the better part of a thousand. The sample size and power calculator turns a target gap into the count before you run the eval, which is the statistical power question stated in eval terms.
Why the Wilson interval instead of the normal approximation?
Because the normal approximation misbehaves exactly where eval data sits. The textbook Wald interval is built on the observed rate, so it collapses to zero width at 0% and 100% and undercovers badly at small counts: a run of 12 comparisons swept clean would report certainty it has not earned. The Wilson score interval stays well behaved at the extremes and at small samples, which is why it is the default here and on every rate this shelf reports. The wider argument sits in the entry on the eval confidence interval.
What win rate does a 100-point Elo gap correspond to?
About 64%. The Elo scale is defined so a gap of d points implies an expected score of 1 / (1 + 10^(-d/400)), which puts 100 points at a 64.0% expected score for the stronger system, 200 points at about 76%, and 400 points at about 91%, or winning roughly 10 of every 11 decided games. This tool reports the gap and the win rate side by side, so the conversion is visible in both directions on your own data.
Why does this fit Bradley-Terry instead of updating Elo ratings?
Sequential Elo depends on presentation order: the same comparisons shuffled differently end at different ratings, and early games are partly forgotten. An eval batch has no meaningful order, so the order-free Bradley-Terry maximum likelihood fit is the right estimate, and its gaps live on the same 400-points-per-factor-of-10 scale Elo readers already know. This is the same choice Chatbot Arena made for its leaderboard.
How do I handle ties in pairwise comparisons?
Two conventions, each stated where it applies. The win rate, its Wilson interval, and the exact sign test drop ties, which matches the sign-test convention and keeps the question binary; other write-ups of pairwise scoring keep ties in the denominator and report the tie share separately, so say which convention a published rate used. The Elo-scale gap and the Bradley-Terry strengths give each tie half a win of credit to each side, matching the Elo scoring rule. An explicit tie model exists (the Davidson extension of Bradley-Terry) and is deliberately out of scope here; with heavy ties, read the tie-inclusive score next to the decided rate before concluding anything.
Why are there no Bradley-Terry ratings for my data?
The maximum likelihood strengths exist only when win credit flows both into and out of every part of the comparison graph. An undefeated system has a likelihood that keeps rising as its strength grows without bound, a winless system mirrors that at zero, and two groups with no comparisons between them have no common scale at all. The tool names the systems responsible. The fix is data: collect comparisons in the missing direction, or read the head-to-head intervals, which remain valid throughout.
My model is ranked first. Is the ordering real?
Rank order alone says very little: some ordering always comes out, even from noise. The separation column answers the real question pair by pair, from each direct record's interval. Adjacent systems whose interval includes 50% are one statistical tier, and reporting them as tied is what the evidence supports. For a paired significance test on one specific pair judged over the same items, the McNemar test calculator is the sharper instrument. For a published leaderboard of scores instead of head-to-head records, the benchmark rank uncertainty calculator gives each system a plausible rank range and names which adjacent ranks are a statistical tie.
Sources
- Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons (the Bradley-Terry model)
- Solution of a Ranking Problem from Binary Comparisons (when the maximum likelihood ranking exists)
- MM Algorithms for Generalized Bradley-Terry Models (the fitting algorithm used here)
- Probable Inference, the Law of Succession, and Statistical Inference (the score interval on a win rate)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Bradley-Terry over online Elo for LLM comparisons)
- Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation (the preference-flip rates)
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (the baseline-anchored design, and length bias)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (position, verbosity and self-enhancement bias)
- Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons (the quadratic comparison cost)