Fleiss' Kappa, Krippendorff's Alpha and ICC Calculator

The input is a matrix with one row per judged item and one column per rater. Raters can be LLM judges, human annotators, or a reference set treated as one more rater, so the same figures serve inter-rater and inter-coder reliability, whether the panel is a jury of LLM judges from different model families or a set of human coders working one rubric. Cells may be empty. Each coefficient is computed on the slice of the data its definition allows, and the coefficient table names that slice next to every figure, because a coefficient quoted without its basis cannot be compared with anyone else's.

Krippendorff's alpha, the headline. Every item carrying at least two ratings contributes its within-item pairs of values to a coincidence matrix; alpha is one minus the observed disagreement in that matrix divided by the disagreement expected if the same pool of values were shuffled across items (Krippendorff 2011). Missing cells simply contribute fewer pairs, which is why alpha is the one coefficient here that needs no rows dropped. The level of measurement sets the difference function: nominal counts any mismatch as one, ordinal uses Krippendorff's rank distance measured through the coincidence marginals, so a 4 against a 5 disagrees less than a 1 against a 5, and interval uses the squared difference of the numeric values. Ordinal and interval need numeric labels; unused points on the scale do not change the ordinal figure. Krippendorff's own reading is that 0.80 supports reliable conclusions and 0.667 supports tentative ones; the Landis and Koch bands (poor, slight, fair, moderate, substantial and almost perfect) are shown as well, being the more familiar convention, with the caveat that both are conventions rather than tests.

Fleiss' kappa. Fleiss (1971) generalizes Scott's pi to any number of raters and needs the same number of ratings on every item, so it runs on the complete-case items, those every rater scored. Its observed agreement is the mean share of agreeing rater pairs per item; its chance agreement is the sum of squared pooled label shares; kappa is their difference scaled by the agreement chance leaves available. The observed and chance rows in the coefficient table are the versions defined on every pairable item, weighting each item by its number of ratings and drawing the chance pair from the pooled labels without replacement; on complete data they equal Fleiss' figures, and on any data one minus the ratio of their complements is the nominal alpha, so those two rows explain the nominal alpha rather than the kappa. Fleiss' kappa treats every scale as unordered, which is why it lands well below the ordinal alpha on rubric scores. The per-category kappa in the label table is Fleiss' own decomposition and shows which labels the panel agrees on. The z row tests kappa against zero using the Fleiss, Nee and Landis (1979) variance under the null; it says whether agreement is detectably better than chance, which is a weak bar, so the bootstrap interval is the figure to quote.

Two raters, and the sibling tool. With exactly two raters, Fleiss' kappa is Scott's pi: it pools both raters' label rates into one marginal. Cohen's kappa keeps each rater's own rates and is the figure the two-rater kappa calculator reports for a judge against a human on pass/fail labels, with its large-sample interval. The two agree only when both raters use the labels at the same rates. The pairwise matrix on this page is Cohen's kappa for every rater pair, so a two-column nominal paste reproduces the sibling's number in the matrix cell while the headline stays the multi-rater family. Two raters on a binary label belong in the sibling tool; three or more raters, more than two labels, ordered scores, or gaps in the matrix belong here.

Intraclass correlation. When the labels are numeric and the level is ordinal or interval, a two-way ANOVA on the complete items gives the six Shrout and Fleiss (1979) forms, in their notation; McGraw and Wong (1996) write ICC(2,1) as ICC(A,1) for absolute agreement and ICC(3,1) as ICC(C,1) for consistency. ICC(1) treats each item's raters as a random draw with no rater identity. ICC(2,1) treats raters as a random sample from a larger pool and asks for absolute agreement, so a rater who scores everything one point higher is penalized; that fits a judge prompt you might swap for another. ICC(3,1) treats these exact raters as the panel and asks only for consistency, so a constant offset is forgiven; that fits a fixed ensemble whose scores you will average and rank. The k forms rate the reliability of the panel mean rather than one rater. Intervals are the F-based ones from those papers; the absolute-agreement interval uses the Satterthwaite degrees of freedom McGraw and Wong give, while the F test in the table uses the ordinary ANOVA degrees of freedom shown beside it. Fleiss' kappa is still computed on numeric data, and the gap between it and the ICC is the price of ignoring the order of the scores.

Rater diagnostics and the pairwise matrix. For each rater: how often its label matches the plurality of the other raters (items where the others tie are skipped); alpha recomputed with that rater removed and the change against the full panel, so a positive change names a rater the panel agrees better without; the mean signed gap between the rater's score and the mean of the others' scores on the same items, which reveals a lenient or harsh judge on numeric scales; and the rater's most-used label with its share, which reveals a rater who says pass to nearly everything. The pairwise matrix is Cohen's kappa for each pair on the items both rated, unweighted at the nominal level, linear-weighted on rank distance at the ordinal level, and quadratic-weighted on value distance at the interval level, with the exact-match share in parentheses. A rater whose row is uniformly low disagrees with everyone; a single low cell is a pair-specific problem.

Why kappa can look bad when agreement looks good. Chance-corrected coefficients divide by the agreement chance leaves available. When one label dominates, say 92% of all ratings are pass, chance agreement is already high and even 95% raw agreement leaves a modest kappa; when two raters disagree about how often to use a label, the coefficient moves again. Feinstein and Cicchetti (1990) named these the prevalence and bias paradoxes. Neither is a defect in the coefficient: it is reporting that most of the raw agreement was cheap. The label table shows the prevalence, the coefficient table shows raw and chance agreement side by side, and the honest report quotes all three.

Gwet's AC1. AC1 answers the same question with a different chance term. Fleiss asks how often two raters would agree by picking labels at the rates the panel used overall, which climbs toward one as a single label takes over. Gwet asks how often two raters would agree by chance on the items that are genuinely hard to call, so a lopsided label mix pushes his chance term down instead of up. In numbers, the term is the sum of p(1 − p) across the scale points, divided by one less than the number of points. AC1 runs on every item with at least two ratings, the same set Krippendorff's alpha uses, so a missing cell costs it nothing. Read on the same items, the two chance terms meet and the two coefficients are equal when every label is used equally often, and away from that point AC1 is the higher of the two. Fleiss' kappa drops the partly rated items that AC1 keeps, so on a matrix with missing cells the two are reading different item sets and AC1 can come out below. The unweighted form is what this page reports at every level. One input moves AC1 and nothing else: declaring the rubric's scale points. A five-point rubric where raters only ever used 4 and 5 has a chance term computed over two points unless you say there were five, and the coefficient table names which count it used.

Uncertainty. Kappa and alpha carry percentile bootstrap intervals from item resamples. For nominal coefficients, Zapf and colleagues (2016) found that interval to hold its coverage where the closed-form variances did not; the same item-level resampling is applied at the ordinal and interval levels, where it is the standard practical choice rather than a separately validated one. The resampling is seeded, so a rerun on the same data reproduces the same interval, and resamples on which the coefficient is undefined are dropped and counted. The interval widens as items get scarce; with fewer than about ten items it is only indicative, and with a single item none is shown. The ICC intervals are the analytic F-based ones.

Worked example. The preloaded panel is three LLM judges and one human scoring 20 items on a 1 to 5 rubric with three cells missing. At the ordinal level alpha is 0.851 (95% bootstrap interval 0.706 to 0.896), which clears Krippendorff's 0.80 bar; the interval alpha is 0.840 and the nominal alpha 0.413, the same data read as unordered labels. Fleiss' kappa on the 17 complete items is 0.365, and observed exact-match agreement is 53.2% against 20.3% by chance. ICC(2,1) is 0.841 and ICC(3,1) is 0.890; the gap is the cost of absolute agreement, and its source shows in the rater table: judge_c scores 0.61 points above the rest of the panel on average and matches the others' majority on only 39% of items, and removing judge_c would raise alpha to 0.898. Switch the level to nominal and the same panel reads as fair agreement, because every one-point difference now counts as a full miss.

Honest limits. Every figure is arithmetic on the matrix you paste. Agreement among judges is a floor on trust, not a proof of correctness: a panel of judges can agree with one another and still be wrong in the same direction, which is the failure mode worked through in testing whether an LLM judge is reliable and in the catalog of LLM judge biases. Kappa and alpha also depend on the label mix in the sample, so compare panels only on the same items and quote the coefficient with its level, its basis, and its interval, the way the judge validation report expects it.

Formula: α = 1 − D_o / D_e; κ = (P̄ − P̄_e) / (1 − P̄_e)

See Sources below