Cohen's kappa (eval agreement)

Cohen’s kappa is a chance-corrected agreement coefficient for two raters who label the same items into unordered categories: observed agreement minus the agreement their own marginal rates would produce by accident, divided by the agreement that expectation left available (Cohen, 1960, peer-reviewed). In an eval the two raters are an LLM judge and a human reference, and the categories are usually pass and fail. It reaches 1 for perfect agreement, sits at 0 when the two match no more often than their labeling habits predict, and goes negative when they disagree systematically.

Agreement is one of three axes a judge clears before its scores license a decision, and the judge-reliability hub holds the other two: whether its stated confidence tracks its accuracy, and whether a distortion moves the ranking. Kappa measures agreement and it measures nothing else. Recompute it whenever the judge model, the rubric or the task mix changes, since last quarter’s coefficient says nothing about this quarter’s judge.

How to calculate Cohen’s kappa

Score the same calibration subset twice, with the judge and with a human reference, then cross-tabulate the two label sets into a 2x2 table: both pass, judge passes where the human failed, judge fails where the human passed, both fail. Observed agreement is the two matching cells over the total. Chance agreement comes from the margins: multiply the judge’s overall pass rate by the human’s, multiply their two fail rates, and add the products. Kappa is the gap between observed and chance agreement, divided by the distance chance agreement left below 1.

Report the coefficient with its interval and all four raw cell counts, since tables that disagree in very different ways return the same kappa, and the off-diagonal pair shows whether the judge runs lenient or strict. The inter-rater reliability calculator takes the four counts and returns the coefficient, its standard error and the interval. Instrument it where the judge writes its verdict: hold back a stratified subset, route those items to a human grader, and store both labels against the same item id so the table can be rebuilt later.

Cohen’s kappa vs percent agreement

Percent agreement is the share of items on which the two raters wrote the same label, counted directly and reported with no adjustment. Kappa takes that same share and subtracts the agreement two raters would have reached by labeling independently at their observed rates.

How far apart the two land depends on how the suite is balanced. On a suite where most items pass, percent agreement runs high while kappa collapses, since two graders who both say pass most of the time match often with no shared judgment involved. On a suite split evenly, chance agreement sits near 0.5 and kappa tracks a rescaled percent agreement closely, so the correction earns little. Publish the pair: percent agreement tells you how often a human would have had to overrule the judge, and kappa tells you how much of that matching was earned.

Why a high-agreement eval can return a low kappa

Feinstein and Cicchetti named this the first of two kappa paradoxes: when the marginal rates are lopsided, expected agreement climbs toward observed agreement and the coefficient falls even though the raters matched on nearly every item (Feinstein and Cicchetti, 1990, peer-reviewed). Eval suites are lopsided by construction: a suite the system fails half the time gets fixed or retired long before anyone calibrates a judge against it.

Take a schematic calibration subset of 200 items, each scored twice. The judge and the human both pass 170 items and both fail 5, the judge passes 10 the human failed, and the judge fails 15 the human passed. Raw agreement is 175 of 200 items, or 87.5%. Chance agreement, from a judge pass rate of 90% against a human pass rate of 92.5%, works out to 84%, so kappa is 0.035 over 0.16, about 0.22. Those counts are illustrative rather than measured, and the arithmetic holds anyway: a judge matching a human on seven of every eight items can be adding almost nothing beyond the base rate.

A calibration subset that mirrors production is therefore the wrong input to this coefficient. Stratify it so failures appear well above their natural rate, compute kappa there, and hold that stratification fixed across comparisons so two judge versions read against the same chance baseline.

Cohen’s kappa vs Krippendorff’s alpha

Krippendorff’s alpha is a general agreement coefficient defined for any number of raters, any measurement scale from nominal through ratio, and any pattern of missing labels, computed from observed and expected disagreement rather than from agreement. Cohen’s kappa covers exactly two raters on an unordered category set, with every item labeled by both.

The boundary between them falls straight out of those two definitions. Three graders scoring a 1-to-5 rubric, or a panel where each human labeled a different slice, falls outside kappa and inside alpha, and forcing it into kappa means discarding raters or flattening an ordinal scale to pass and fail. A complete judge-against-human pass/fail table falls inside both, and there the two differ in their chance model. Alpha builds expected disagreement from the pooled distribution of all labels, kappa builds expected agreement from each rater’s own margins, so where judge and human hold different pass rates the two won’t land on the same number.

Krippendorff’s alpha handles the ragged and multi-scale designs, this page owns the two-rater nominal one, and a bare “agreement of 0.71” does not say which chance model produced it.

What counts as a good kappa for an LLM judge

The bands most teams quote come from Landis and Koch, writing in Biometrics in 1977 about observer agreement on categorical medical data, where 0.61 to 0.80 reads as substantial and 0.81 to 1.00 as almost perfect. Those cut points are a convention in that field rather than a threshold derived from any decision cost, so importing them imports a clinical study’s tolerance for observer disagreement. Our own checklist of judge gates adopts 0.61 anyway, on the grounds that a published line beats one each team invents for itself.

Kappa is a ratio of two quantities estimated from one finite table, so uncertainty in the numerator and the denominator both propagate into it, and the denominator shrinks exactly when chance agreement is high. A calibration subset of a few hundred paired labels therefore produces a point estimate with real width behind it. Report kappa with a confidence interval, from the large-sample standard error of Fleiss, Cohen and Everitt (1969) that SAS and statsmodels both implement, or by resampling the table directly when the counts are thin. The reasoning that governs any interval around an eval metric governs this one.

State the cut point you chose, the cost that justified it, and the interval on your estimate, then treat an estimate above the line whose lower bound sits below it as not yet cleared. A kappa that fails the gate is a signal about the construct rather than the arithmetic: a judge diverging from its reference is scoring something other than the quality you specified, and no downstream bias correction repairs a measurement of the wrong thing.

Agreement says the judge resembles its reference, while sensitivity and specificity say how far the judge’s own errors bend the rate you go on to publish, and any judge-scored number needs both readings standing behind it. The route from a calibration subset to a corrected rate that carries both is walked through in bias-correcting a judge eval before you report it, and the statistics under every other number an eval reports are gathered in which eval number needs which test.