Observed agreement
70.0%
INSTRUMENT · EVAL
Inter-Rater Reliability Calculator (Cohen's Kappa)
Measure how well an LLM judge agrees with a human reference on pass/fail labels: Cohen's kappa from a 2x2 table, with its confidence interval and reading.
Cohen's kappa (κ)
0.40
95% CI 0.15 to 0.65 · Fair agreement
Chance agreement
50.0%
Rates behind the score
Wilson score intervals at 95% coverage · 50 paired items
| Metric | Rate | Confidence interval |
|---|---|---|
| Raw agreement · (a + d) / n | 70.0% | 56.2% to 80.9% |
| Judge pass rate · (a + b) / n | 50.0% | 36.6% to 63.4% |
| Reference pass rate · (a + c) / n | 60.0% | 46.2% to 72.4% |
κ = (po - pe) / (1 - pe)How?
How this is calculated
This tool reads one eval run as a 2x2 agreement table. Every item was scored pass or fail by the LLM judge and, independently, pass or fail by a human reference. Cell a counts items both passed, b the judge passed but the reference failed, c the reverse, and d items both failed. n is their sum.
Observed agreement is the share the two graders labeled the same way,
po = (a + d) / n. Two graders also agree on some items by chance
alone, at a rate set by how often each says pass. Chance agreement combines
the marginal rates: pe = (judge pass)(reference pass) + (judge
fail)(reference fail).
Cohen's kappa rescales observed agreement against that chance floor, so it
gives no free credit for the easy calls both graders get right:
κ = (po - pe) / (1 - pe). Kappa is 1 at perfect agreement, 0 at
chance, and negative when the graders agree less than their pass rates
predict. When pe = 1, every item sits in one category and kappa
is undefined.
The kappa confidence interval uses the large-sample standard error from
Fleiss, Cohen, and Everitt (1969), the form standard statistics packages
report. The interval is κ ± z·SE, clamped to the theoretical
range of -1 to 1. It is a normal approximation: at small n or extreme pass
rates it grows rough, so weigh its width more than its exact endpoints. The
raw agreement rate and the two marginal pass rates each carry a Wilson score
interval, which stays inside 0 to 1 and behaves on small counts.
Worked example. Take 50 items with a = 20, b = 5, c = 10, d = 15. Observed agreement is (20 + 15) / 50 = 0.70. The judge passed 25 of 50 (50%) and the reference passed 30 of 50 (60%), so chance agreement is 0.50·0.60 + 0.50·0.40 = 0.50. Kappa is (0.70 - 0.50) / (1 - 0.50) = 0.40, which the convention labels Fair. The standard error is 0.127, so the 95% interval is 0.40 ± 1.96·0.127, or 0.15 to 0.65. Raw agreement of 70% carries a 95% Wilson interval of 56% to 81%.
Agreement is one axis of judge reliability. For calibration and bias, see the LLM-as-a-judge reliability hub.
Formula: κ = (po - pe) / (1 - pe)
Questions
What counts as good agreement?
No cutoff is universal. The Landis and Koch bands call 0.61 to 0.80 substantial and above 0.80 almost perfect, and many eval teams want substantial or better before a judge gates a release. Always report the interval next to the point value so a reader sees the uncertainty.
Why not just use raw agreement?
Raw agreement counts the easy items both graders get right, which inflates the number on skewed pass rates. If a judge and a reference each pass 95% of items, they land on the same label about 90% of the time by chance alone. Kappa subtracts that floor, so the score reflects agreement beyond chance.
My kappa is negative. What does that mean?
Negative kappa means the judge and reference disagree more than their individual pass rates predict. In practice it usually points to an inverted label, a prompt that flips the judge's polarity, or a reference set scored under different instructions. Inspect the off-diagonal cells b and c.
How many labeled items do I need?
More items narrow the interval. The standard error here is a large-sample approximation, so below roughly 30 paired items the kappa interval is a rough guide. When you need a target precision, size the sample from the interval width you can tolerate on the underlying pass rate.
Is this Cohen's kappa or Fleiss' kappa?
This is Cohen's kappa for two raters on a binary label. Fleiss' kappa generalizes it to three or more raters. For an ordinal or graded label, weighted kappa credits near-misses; this tool uses the unweighted binary form.