TOOL | Eval statistics
Review sample drawer: random and stratified sampling for eval review
Take a reproducible random, stratified or fixed-rate review sample from a large CSV or JSONL, with each row's inclusion probability and a manifest of the settings. Your file stays in this browser tab.
Pick a file, choose a mode, and take a sample. First read pulls a simple random sample. Labeling set samples each stratum separately. Production review selects rows by a seeded hash of their key, stable across files. Every row carries its inclusion probability, so that accuracy measured on an uneven sample can be corrected to the whole file. A manifest records the seed, file hash, and settings. Your file is processed in this browser tab and never uploaded.
These settings came from the link you opened. Your file was not in it.
CSV, TSV, JSONL or JSON array. JSON arrays are limited to 10 MB because they cannot stream.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
The figures below are computed from the example, a fictional returns desk for an online outdoor-gear shop. They update when you change any setting and redraw.
Pick a file or load the example. Your file stays in this browser tab and is never uploaded.
Sample ready
Counting each row by one over its inclusion probability turns rates measured on the sample into estimates for the rows it was drawn from, even when some strata are over-represented.
Review a first-read sample on the error analysis workbench; for a labeling set, give the labeler copy to your labelers and compare their labels with the judge's verdicts, using each row's inclusion probability.
Strata in the sample, sorted by stratum value.
| Stratum | Population | Sampled | Inclusion probability | Weight |
|---|
w = 1 / inclusion probability; first-read n/N, labeling-set min(k, N_h)/N_h, production the entered rateHow?
How this is calculated
One pass reads every row, applies the filter, and fills a reservoir per stratum (or one reservoir in first-read mode).
Equal allocation draws up to k rows from every stratum, and a stratum smaller than k is taken whole. Proportional allocation gives each stratum k times its share of the rows, rounded down, then gives the rows left over to the strata with the largest remainders, ties going to the lower stratum value. The reservoirs are concatenated in stratum-value order and shuffled with the same seeded generator, so that labelers cannot infer a row's stratum from its position.
Weights are the inverse of the inclusion probability: w = 1/pi. For a fixed-size design the weights of the sample sum to the population size. A two-stage design multiplies the new probability by the existing one.
Production mode decides each row independently by hashing its key with the seed. The same key at the same seed and rate is always in or out, whatever file it appears in and however the file is ordered. The expected sample size is the rate times the population.
The manifest records every design choice, the file hash, and every stratum's population and sampled count. Loading it back fills every setting so the draw reproduces exactly on the same file.
The file is streamed through a SHA-256 hash as it is read. If the file changes between the two passes (one for proportional allocation, one for the draw), the hashes differ and no sample is written.
Why not just take the first rows?
The first rows of a file often share a cause: the same time window, the same user, the same prompt version. A random sample avoids that bias. The seed makes the draw reproducible: anyone with the same file, seed and settings gets the same sample.
Show the formula
Simple random sample: one-pass reservoir sampling (Algorithm R). Each row is kept with probability n/N. Stratified: a reservoir per stratum; equal allocation draws min(k, N_h) (N_h, the stratum's row count) from each stratum; proportional allocation assigns floor(k N_h / N) to each, then the strata with the largest fractional remainders get one more row until k is reached. Ties go to the stratum with the lower value (code-unit order). Bernoulli: a seeded hash of the row's key produces a uniform value in [0, 1); the row is kept when that value falls below the entered rate. Two-stage: each row's probability becomes the product of the two stages; the previous design columns are renamed.
Worked example
Load the returns-desk example in first-read mode (k 10, seed 1, 40 rows). The page takes these 10 rows: tr-rd-031, tr-rd-032, tr-rd-003, tr-rd-012, tr-rd-024, tr-rd-026, tr-rd-023, tr-rd-006, tr-rd-033, tr-rd-038. Inclusion probability 0.25, weight 4. In labeling mode on the labeling example (80 rows, each with a judge verdict and a human label, strata on judge, equal, k 10, seed 1), 10 of the 17 fail rows are taken (inclusion probability 0.588235, weight 1.7) and 10 of the 63 pass rows are taken (inclusion probability 0.15873, weight 6.3). Weights sum to 80.
Formula: w = 1 / inclusion probability; first-read n/N, labeling-set min(k, N_h)/N_h, production the entered rate
Questions
Why not review the same number of flagged and unflagged rows?
Equal numbers from each judge verdict over-represent the rarer verdict, so catch and false alarm rates computed from the sample come out wrong unless each row counts as one over its inclusion probability. Begg and Greenes (1983) showed the same for diagnostic tests: when the cases checked are chosen by the test's own result, rates computed from those cases are biased unless the selection is corrected for. The labeling mode keeps the probability on every row so that the correction can be made downstream.
Same seed and file, same sample?
Yes. The reservoir's random stream comes from a single seeded generator, and the file hash is recorded in the manifest. Anyone who loads the same file bytes and the same settings gets byte-identical output apart from the timestamp.
How is this different from the slice and class balance analyzer?
The slice and class balance analyzer counts slices and classes in a file. This page takes a sample from the file with known inclusion probabilities, so that the sample can be reviewed or scored and the results weighted back to the population.
Equal or proportional allocation?
Equal allocation draws the same number from every stratum, so each stratum's estimate has roughly the same precision. Proportional allocation mirrors the file's distribution, so an unweighted overall estimate is close to representative. Use equal when the goal is accuracy within each stratum; use proportional when the goal is overall accuracy.
How does production sampling work?
Each row's key is hashed with the seed to produce a uniform number. If that number falls below the entered rate, the row is selected. The decision depends only on the key, the seed and the rate, so it is stable across files: the same key is always in or out regardless of what else is in the file, and adding rows never changes an existing row's decision.
How large a file can I load?
CSV, TSV and JSONL files are read in a stream, one batch at a time, so the page never holds the whole file; a large file takes longer to read. JSON arrays must be parsed whole and are limited to 10 MB. The sample itself is capped at 5,000 rows.
What does the manifest record?
The mode, sample size (k), seed, allocation, rate, filter, strata field, key field, the source file's SHA-256 hash and byte count, and a table of strata (value, population, sampled, inclusion probability, weight). It holds no rows from the file; the sample file carries those, each with its row number in the source. Loading the manifest back into this page fills every setting; on the same file it reproduces the sample, on a different file it applies the same rule.
What is a labeler copy?
It is the sample without the stratum field or the four sampling columns (the stratum, its size, how many rows were sampled from it, and the inclusion probability). Give it to labelers so that they cannot see which class a row came from and their judgments are not influenced by the original verdict.