Cohen's Kappa Calculator (Two Raters): Inter-Rater Reliability

This tool reads one eval run as a 2x2 agreement table. Every item was scored pass or fail by the LLM judge and, independently, pass or fail by a human reference. Cell a counts items both passed, b the judge passed but the reference failed, c the reverse, and d items both failed. n is their sum.

Observed agreement is the share the two graders labeled the same way, po = (a + d) / n. Two graders also agree on some items by chance alone, at a rate set by how often each says pass. Chance agreement combines the marginal rates: pe = (judge pass)(reference pass) + (judge fail)(reference fail).

Cohen's kappa rescales observed agreement against that chance floor, so it gives no free credit for the easy calls both graders get right: κ = (po - pe) / (1 - pe). Kappa is 1 at perfect agreement, 0 at chance, and negative when the graders agree less than their pass rates predict. When pe = 1, every item sits in one category and kappa is undefined.

The kappa confidence interval uses the large-sample standard error from Fleiss, Cohen, and Everitt (1969), the form standard statistics packages report. The interval is κ ± z·SE, clamped to the theoretical range of -1 to 1. It is a normal approximation: at small n or extreme pass rates it grows rough, so weigh its width more than its exact endpoints. The raw agreement rate and the two marginal pass rates each carry a Wilson score interval, which stays inside 0 to 1 and behaves on small counts.

Worked example. Take 50 items with a = 20, b = 5, c = 10, d = 15. Observed agreement is (20 + 15) / 50 = 0.70. The judge passed 25 of 50 (50%) and the reference passed 30 of 50 (60%), so chance agreement is 0.50·0.60 + 0.50·0.40 = 0.50. Kappa is (0.70 - 0.50) / (1 - 0.50) = 0.40, which the convention labels Fair. The standard error is 0.127, so the 95% interval is 0.40 ± 1.96·0.127, or 0.15 to 0.65. Raw agreement of 70% carries a 95% Wilson interval of 56% to 81%.

Kappa falls with skewed prevalence and moves with rater bias, even when observed agreement stays the same (Feinstein and Cicchetti, 1990). The prevalence index PI = (d minus a) / n and the bias index BI = (c minus b) / n, with failure as the positive class, where a counts items both raters passed, d counts items both raters failed, and b and c are the two disagreement cells. Kappa can be written through them: kappa = (PABAK minus PI squared plus BI squared) / (1 minus PI squared plus BI squared), an identity from Byrt, Bishop and Carlin (1993) that holds on every table where kappa is defined. When prevalence is extreme (PI further than 0.5 from zero, equivalently the majority class above 75% of the pool), chance agreement is high and kappa is driven down; this is the prevalence paradox. When the two raters use the labels at different rates (BI away from zero), kappa rises for the same observed agreement and the same prevalence index; this is the bias paradox.

AC1 (Gwet, 2008) replaces the chance term with one that stays stable under skewed prevalence: the chance of agreement on items that are hard to classify. For two labels the term is 2 p (1 minus p), where p is the pooled share of the positive class. Because the term does not climb with prevalence the way kappa's does, AC1 stays high when agreement is high regardless of label balance.

PABAK is twice observed agreement minus 1: observed agreement rescaled from the range 0 to 1 to the range -1 to 1. It sets prevalence and bias aside, so it measures only how often the raters match; its chance term is fixed at one half whatever the data. Report it as rescaled agreement (Byrt, Bishop and Carlin, 1993). The panel reports kappa, AC1 and PABAK beside the two indices, and the indices show which source is at work.

Agreement is one axis of judge reliability. For calibration and bias, see the LLM-as-a-judge reliability hub. Once you have a kappa, log it in the agreement tracker so this run has a series to sit inside.

Formula: κ = (po - pe) / (1 - pe)

See Sources below