Imbalance ratio
4.50
INSTRUMENT | Eval statistics
Slice and Class Balance Analyzer: See What an Eval Set Contains
2 cited sources
Read what an eval set holds before you quote a pass rate. Count rows per slice, per class, and per control flag, put an interval on every share, and see the smallest slice named.
One green pass rate hides the slice that is three percent of the file. The small-slice line rests on our map of which eval number needs which test and how many runs catch a regression.
A slice under 16 rows cannot carry a 95% interval narrower than 20 points, and clearing 16 is not enough. Take the count to the interval calculator, or size it before the run. Shape belongs to the schema validator and its id and completeness check.
Showing your last valid result. Update the inputs above to recompute.
Rows read
200
200-row example set, ready to replace with your own file. Smallest slice: ja-JP in "locale", 21 rows, 10.5%.
Slice: should_trigger
| Value | Rows | Share | 95% interval | Floor |
|---|---|---|---|---|
| true | 150 | 75.0% | 68.6% to 80.5% | |
| false | 50 | 25.0% | 19.5% to 31.4% | |
| (missing) | 0 | 0.0% | 0.0% to 1.9% | under 16 |
An interval on a share means something only if this file is a sample of a larger stream of work. If the file is the whole set you care about, the share is already exact and the interval is noise.
Slice: difficulty
| Value | Rows | Share | 95% interval | Floor |
|---|---|---|---|---|
| easy | 90 | 45.0% | 38.3% to 51.9% | |
| medium | 70 | 35.0% | 28.7% to 41.8% | |
| hard | 40 | 20.0% | 15.0% to 26.1% | |
| (missing) | 0 | 0.0% | 0.0% to 1.9% | under 16 |
Slice: locale
| Value | Rows | Share | 95% interval | Floor |
|---|---|---|---|---|
| en-US | 120 | 60.0% | 53.1% to 66.5% | |
| de-DE | 47 | 23.5% | 18.2% to 29.8% | |
| ja-JP | 21 | 10.5% | 7.0% to 15.5% | |
| (missing) | 12 | 6.0% | 3.5% to 10.2% | under 16 |
Slice
| Value | Rows | Share | 95% interval | Floor |
|---|
Slice
| Value | Rows | Share | 95% interval | Floor |
|---|
Pick at least one slice column above and the tables fill in.
Label column: expected
| Value | Rows | Share | 95% interval | Floor |
|---|---|---|---|---|
| refund | 90 | 45.0% | 38.3% to 51.9% | |
| billing | 60 | 30.0% | 24.1% to 36.7% | |
| technical | 30 | 15.0% | 10.7% to 20.6% | |
| escalation | 20 | 10.0% | 6.6% to 14.9% | |
| (missing) | 0 | 0.0% | 0.0% to 1.9% | under 16 |
Classes observed
4
The ratio is the largest observed class over the smallest. It carries no band and no verdict: what counts as too far apart depends on what the set is for.
Control column: should_trigger
| Value | Rows | Share | 95% interval | Floor |
|---|---|---|---|---|
| Positives | 150 | 75.0% | 68.6% to 80.5% | |
| Negatives | 50 | 25.0% | 19.5% to 31.4% | |
| Unmapped | 0 | 0.0% | 0.0% to 1.9% | under 16 |
Positives over negatives: 3.00. Unmapped rows are values neither list claims, and they are shown rather than dropped.
Cross-tab: two columns at once
Pick two different columns under "Cross-tab" above and the grid fills in. It answers the question a single distribution cannot, such as whether every negative control sits in one locale.
Counts per cell
| Value |
|---|
share = rows in the value / rows in the file, then wilson(rows, total, level)How?
How this is calculated
Every cell is trimmed, and case is kept, so EN-US and en-us are two values.
A cell counts as missing when it is blank or reads null</code>, <code>na</code>, <code>n/a</code>, <code>nan, and the switch above turns that off. Missing rows are counted and shown as their own row,
never dropped, which is what keeps the per-slice counts summing to the file total.
The interval on a share is the Wilson score interval, the same one the rest of this shelf reports. The bound printed on a cell holding no rows is the exact Clopper-Pearson upper bound at a count of zero over the whole file, which is a stricter reading of "we saw none of these" than a zero is.
The 16-row floor is ours, and it is a floor on possibility. We chose a target of 20 points as the widest 95% interval worth quoting. A Wilson interval reaches that width only when every row in a slice lands the same way, and that best case needs 16 rows. No source states this number; it follows from the target we picked. Clearing it does not make a slice big enough: at 16 rows a rate near 50% still spans 44 points, which is why every slice carries its own precision line whether it is marked or not.
A column is offered as a slice when its distinct values are at most 20% of its non-blank cells and fewer than 1,000, or when it holds at most 10 distinct values. Booleans qualify too, because a boolean is a slice. A column holding a different value in every row is read as an identifier once it has 20 or more non-blank cells, and is not suggested. The first 3 suggestions arrive ticked and the rest are offered unticked, so nothing is applied that the picker does not show. The label and control pickers take any column by hand, but they pre-select only after the same kind test passes, which is what stops a column of free-text gold answers rendering as a 50-row class table.
Shares and interval bounds show one decimal on screen and six in the CSV, both rounded on the decimal value. Ratios show two decimals and counts are whole numbers. At most 50 values render per table with the rest rolled into one row, and a cross-tab axis renders at most 12. The CSV lists every value, so nothing is rolled up there.
Formula: share = rows in the value / rows in the file, then wilson(rows, total, level)
Questions
Why is there no balance score?
Because the number would have to decide, for you, how far apart two slices are allowed to be, and that depends on what the set is for. A support-triage set with ten times as many refund cases as escalations may be exactly right if that is the live traffic mix. The libraries that ship this check take the same position: they print the distribution and leave the threshold to whoever set the condition.
What the page does instead is tell you what each slice can support. A slice of nine rows cannot produce a usable rate no matter how the set is balanced, and that is a fact about arithmetic rather than a judgment about your data.
Why does a slice of 16 rows still get a warning about precision?
Because 16 rows is the point where the best imaginable case clears our target, not the point where a slice becomes trustworthy. The best case is every row in the slice landing the same way. A real slice sits somewhere in the middle, and at a rate near 50% those same 16 rows give an interval 44 points wide, which is close to knowing nothing.
So the marker and the precision line say different things. The marker says this slice is below the floor. The line says what this many rows can do at best and what they do at the hardest rate, and it is printed for every slice, marked or not.
What does the interval on a share actually assume?
That the file is a sample of a larger stream of work, and that the rows are independent draws from it. Under that reading, the interval says where the true share of that stream could sit. If the file is the whole set you care about, and you are not generalizing past it, the share is already exact and the interval is noise. The page states this beside the first table rather than hiding it in a note, because it changes how every number here should be read.
Why count empty cells instead of dropping them?
Because a slice column with a fifth of its cells blank is a finding, not a nuisance. Dropping those rows makes the remaining shares add to 100% and quietly changes what the denominator means, so a reader comparing two runs would be comparing two different sets. The empty cells get their own row, they count toward the total, and they carry the same interval and precision line every other row does.
Why does an empty cross-tab cell print a bound?
Because seeing none of something is not the same as there being none. On a file of 200 rows, a combination that never appears is still consistent with a true rate near 2%, and on a smaller file the bound is far higher. The number printed is the exact upper bound at a count of zero, which is the arithmetic behind the older rule of thumb that zero events in n gives an upper bound of about three over n.
Does my file go anywhere?
No. The file is read, parsed and counted in this tab. There is no upload, no server call, and no analytics payload that carries a file name, a size, a row count, a column name, a value or a cell. The export is built in the browser from what is already on this page.
Sources
- Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. The paper behind reporting per slice rather than as one aggregate: held-out accuracy "often overestimates the performance of NLP models".
- Hanley and Lippman-Hand, "If nothing goes wrong, is everything all right? Interpreting zero numerators", JAMA 1983, volume 249 number 13, pages 1743 to 1745. The paper behind the bound this page prints on an empty cell.