TOOL | Eval statistics
Slice explorer with false-alarm control
Test each slice of one run against the rest, with Holm and Benjamini-Hochberg corrections across all tested slices, intervals per slice, a floor for small slices, and a Simpson's paradox check.
Type each slice's passes and totals, or paste one run's results. The page tests each slice against the rest, controls false alarms across all the slices with Holm (for gates) and Benjamini-Hochberg (for triage), and puts an interval on every difference. Slices with fewer than 16 cases show counts only. Each slice is compared with the other cases in one run; paired before-and-after comparisons are a different question. Intervals and tests follow our eval statistics guide.
Type each slice's passes and totals, paste slice counts, paste result rows or load a file, or load the returns-desk example.
How this is calculated
Each slice with 16 or more cases is tested against the remaining cases, independently. The 16-case floor matches the slice and class balance analyzer; slices below it show counts only and stay out of the corrections.
The per-slice interval is a Wilson interval at one minus the false-alarm level (95% at 0.05), unadjusted. The interval on slice minus rest is a Newcombe interval for two independent proportions at the same confidence, also unadjusted. The "minus overall" column is descriptive and uses a different denominator from the test: the overall includes the slice itself, while the test compares the slice to its complement, which does not.
The test for each slice uses a pooled two-proportion z-test when all four expected cells under the pooled null are at least 5, and Fisher's exact two-sided test otherwise. The p-value is two-sided, because a slice can perform better or worse.
Holm step-down controls the family-wise error rate; Benjamini-Hochberg step-up controls the false-discovery rate (Holm 1979; Benjamini and Hochberg 1995). Use Holm when one false alarm is unacceptable (a gate); use Benjamini-Hochberg when a few false leads are tolerable (triage).
Intervals stay wide until about 30 cases; read a slice just above the floor with care. A slice below the floor still counts toward the overall rate and toward every other slice's rest.
Questions
Why can a slice that looks worse stop being called worse after correction?
Each extra slice tested is one more chance for a slice to look extreme by luck alone. In the returns-desk example, two of the five tested slices fall under the 5% line before correction; Holm keeps one of them and Benjamini-Hochberg keeps both. Holm holds the chance of any false alarm at your level; Benjamini-Hochberg holds the expected share of false alarms among the slices it calls at that level.
When should I use the gate column and when the triage column?
Use Holm (the gate column) when a single false alarm has real consequences: holding a release, flagging a compliance issue, rejecting a model. Use Benjamini-Hochberg (the triage column) when you are scanning for leads and can tolerate a few that turn out to be noise: deciding which slices to investigate, where to add cases, or which topics to improve first.
Why is the floor set at 16 cases?
It matches the floor on the slice and class balance analyzer, so a slice reads the same on both pages. Below 16 cases the page shows counts only and leaves the slice out of the corrections. Intervals stay wide until about 30 cases, so read a slice just above the floor with care. Cases in a slice below the floor still count toward the overall rate and toward every other slice's rest.
Why does the "minus overall" column show a different number from the test?
"Minus overall" subtracts the overall rate, which includes the slice itself. The test compares the slice to its complement, which excludes it. When a large slice is removed from the overall, the complement's rate shifts, so the two differences diverge. The test's interval and p-value use the complement, which is the correct comparison.
What is Simpson's paradox and when can it appear here?
Simpson's paradox arises when the aggregate difference between two versions points in one direction while every within-slice difference points the other way, because the two versions have different slice compositions. The check fires when you select a second version and the aggregate and directly standardized differences disagree in sign. When it fires, read the per-slice rates, because the two versions served different mixes of slices.
What does the fairness preset do?
It sets the slice column to userGroup and the second column to language when those columns exist in the data. The per-slice tests then show whether any user group or language performs worse than the rest, with the same false-alarm control, as per-group rates and intervals. The page reports no disparity ratio and no single fairness number.
What happens when cases carry a cluster label or several repeats?
The tests assume that cases are independent. When cases in the same cluster tend to agree (because they share a prompt template, a retrieval source, or a user session), the real variance is higher than the test assumes, and the p-values are too small. The page warns when a cluster column is present. Repeats of the same case are correlated for the same reason, so select one repeat.
How does this page differ from the slice and class balance analyzer?
The slice and class balance analyzer counts how many cases each slice holds and grades nothing. This page reads one run's pass and fail results and tests each slice against the rest. Comparing the same slices across two versions is a different question, answered by paired before-and-after comparisons.