TOOL | Judge reliability
How many human labels to trust the judge?
How many outputs must people label before you trust the model judge, class by class? Enter the catch rate and pass accuracy you expect, and the page sizes a random sample and a sample drawn by the judge's verdict. Labeling 100 flagged and 100 passed outputs and reading them at face value gives a biased catch rate; the allocation check shows by how much. Per-class judge rates are worked through in our note on reporting a judge-scored pass rate.
Labels per class
Needed: 245 labels for the catch rate at plus or minus 5 points, 83 for the pass accuracy at plus or minus 5 points (Wilson interval, inverted, 95% two-sided). Normal shortcut: 246 and 73.
The pass accuracy plus or minus its margin reaches 0% or 100%, so the shortcut and any stratified total are unreliable. Use the Wilson counts.
Random sample
Needed: 2,450 labels from a random sample, driven by failures at a 10% positive-class share; 245 expected real positives among them. Normal shortcut: 2,459.
Verdict-stratified sample
Needed: 2,090 labels (207 from outputs the judge flagged, 1,883 from outputs it passed), using the delta method with a normal approximation. The judge's implied flag rate is 12.5%. Compare 12.5% with the flag rate you see in production. Among outputs the judge flags, 64.0% are real failures; among outputs it passes, 97.7% are real passes.
Stratified against random, both by the normal shortcut
The stratified design collects 2,090 labels against the random design's 2,459, both at the same precision and confidence.
Allocation check at 100 flagged and 100 passed
The catch rate gets plus or minus 20.64 points; the pass accuracy gets plus or minus 1.25 points. An unweighted reading of these counts would show a catch rate of 96.55% and a pass accuracy of 73.08%, against the 80% and 95% you entered.
Wilson interval, inverted (per-class counts). Buderer scaling (random design). Delta method, normal approximation (stratified design). Inverse-probability weighting (labeled counts).
Off by default. When on, the link also carries the counts on this page. Rows, labels and file names stay out of every link.
How this is calculated
Each class is sized on its own. The per-class count is the Wilson interval inverted: the smallest number of labels where the Wilson interval at the entered confidence level has a half-width at or under the entered margin.
The random design divides the catch-rate count by the failure share and the pass-accuracy count by the pass share, then takes the larger (Buderer 1996), because a random sample holds each class only in proportion to its share.
The stratified design finds the smallest total where both classes meet their margins, using the delta-method variance for the catch rate and pass accuracy (sensitivity and specificity with failure as the positive class) under verdict-based sampling. Estimates are weighted by inverse inclusion probability.
An unweighted reading of a verdict-stratified sample is biased because the judge's verdict set the sampling rates (Begg and Greenes 1983). The allocation check shows the expected bias at your planned counts; the labeled-counts panel shows the naive and weighted readings side by side.
The toggle swaps class names and roles. The per-class counts do not change, because each class is sized to its own margin. The random and stratified totals stay the same.
Questions
Why are the catch rate and pass accuracy sized separately?
The judge can be good at spotting failures and poor at passing real successes, or the other way around. A sample that is large enough for one rate can be too small for the other. Sizing each class to its own margin ensures that neither rate is reported at a precision wider than what you asked for.
Why does the positive-class share multiply the label count so steeply?
In a random sample, the number of real failures you label is the total times the positive-class share. When failures are 10% of production, 90% of the labels go toward the pass accuracy and only 10% toward the catch rate, so you need about ten times the per-class count to get enough real positives. The stratified design draws flagged and passed outputs separately, so it can place labels where they narrow the intervals most.
Why do 100 labels in each stratum give the wrong answer?
They fail twice at the defaults. With 100 flagged and 100 passed outputs labeled, the catch rate is known only to within 20.64 points, against the 5 asked for. And a face-value reading of those labels is biased: it is expected to show a catch rate of 96.55% and a pass accuracy of 73.08%, where the rates you entered are 80% and 95%. The judge's verdict decided which outputs were labeled, so each stratum's share of the sample differs from its share of production.
When does the stratified design need fewer labels?
At the defaults (failures at 10% of production, a catch rate of 80%, a pass accuracy of 95%, plus or minus 5 points on each), the random design needs 2,459 labels by the normal shortcut and the stratified design needs 2,090, also a normal approximation. The stratified design places each label where it narrows an interval most. Here that is mostly among outputs the judge passed, because the failures it missed sit there and set the width of the catch rate.
I already labeled a stratified sample. How do I read the results?
Enter the counts in the labeled-counts panel. The page shows the naive rates (dividing each stratum's counts as though they were a random draw) beside the weighted rates (adjusting by inverse inclusion probability). The weighted rates correct for how the sample was drawn. The naive rates show what you would have reported without the correction.
Why is failure the positive class by default?
Failure is the class a judge exists to catch, so this page reports the catch rate as the share of real failures the judge flags. Switching to pass as the positive class swaps the two names; every total stays the same.
How does this page relate to the bias correction calculator?
This page sizes the sample before labeling and shows the bias trap after labeling. The judge bias correction calculator takes a measured judge accuracy and corrects a pass rate for the judge's errors. Plan the labels here; correct the rates there. That page names its rates with pass as the positive class: its sensitivity is this page's pass accuracy, and its specificity is this page's catch rate.