TOOL | Eval statistics
Check an eval set for duplicates, leakage and personal data
Drop a test file and the page reads its schema, flags blank and malformed cells, finds exact, formatting and numbers-masked duplicates, checks for cross-split leakage, counts each class and slice, and runs five personal-data detectors. It reports per section, with a downloadable fix list. Our contamination research sets out the check ladder whose first four checks run here. These checkers go deeper on one section: the duplicate checker, the near-duplicate detector, the overlap checker and the balance analyzer.
Drop a file, paste rows, or load the example. The page reads CSV, TSV, JSON lines and JSON files up to 10 MB, and reports schema errors, duplicate and near-duplicate cases, cross-split leakage, class balance and personal-data patterns.
How this is calculated
The page reads the file with the site's shared reader, which accepts CSV (comma or semicolon), TSV, JSON lines, a JSON document, and a JSON array. YAML files are refused. Rows that do not match the header, or JSON lines that do not parse, are listed under File and left out of every other check. A row the schema check rejects stays in the other checks.
Duplicate detection runs at three rungs over the columns you choose as keys. Exact compares after trimming whitespace. Formatting also strips invisible characters, folds case and Unicode forms, unifies quotes and dashes, removes punctuation, and collapses whitespace. Numbers masked also replaces every run of digits with a single token, so "Order 12345" and "Order 67890" match. A group of two or more rows that share a numbers-masked key is one cluster, labeled by the deepest rung it needed to merge. Rows that share a formatting key but whose expected values differ (ignoring case, spacing and invisible characters) are flagged as a conflicting reference. A numbers-masked cluster whose expected values differ is shown for information only, since the changed number may be a deliberate boundary pair.
The report's transform codes are the step numbers the near-duplicate detector lists in its step table.
Label variants are class values that differ only in letter case, spacing or invisible characters while being counted as separate classes. A label variant can make the smallest class artificially small and the imbalance ratio artificially high. The imbalance ratio is the largest class's count over the smallest observed class's count.
The class floor of 16 rows is the smallest group that can carry a 95% Wilson interval narrower than 20 percentage points in the best case.
Five personal-data detectors scan every string leaf of every cell: email by address pattern, phone by North American format, payment card by digit count and a Luhn check, IBAN by length and the mod-97 check, and cloud access key by its prefix and length. The detectors flag; the matched values are never shown, exported or sent. These patterns miss names and street addresses.
Questions
When should I open the other checkers?
This page applies every check at its default settings and a fixed depth. Each checker goes deeper on its own section: the duplicate checker previews a keep rule, the near-duplicate detector offers per-step control, the overlap checker compares two files, and the balance analyzer adds confidence intervals. Use this page first to see the full picture, then open the checker for the section that needs work.
Why are digit runs masked in the numbers-masked rung?
A test set may vary a single number to test a boundary (a case delivered 30 days ago and one delivered 31 days ago, where the 30-day return window changes the answer). The numbers-masked rung treats them as near-duplicates so that you can decide whether both belong. When the expected values also differ, the cluster is flagged as informational, because the number change may be the deliberate boundary. Names are not masked; the browser cannot reliably detect them without a model.
What does a conflicting reference mean?
Two cases whose inputs match after formatting carry two different reference answers for one question. A version that answers the same way each time can pass at most one of them, so the pair caps the pass rate and leaves the right answer undecided. Choose the one reference answer the team wants and fix the source file, or remove the duplicate.
Why is every slice under 16 rows in a small set?
A slice under 16 rows cannot carry a 95% Wilson interval narrower than 20 points. In the returns-desk set with planted problems, 46 rows spread over five slices: returns 14, damaged-or-wrong 13, refunds 8, exchanges 6 and escalations 5, so every slice sits under 16. The fix is a larger set or fewer slices.
What does the personal-data check miss?
The five detectors match email addresses, phone numbers in the North American format, Luhn-valid payment cards, mod-97 IBANs, and cloud access key ids. They miss names, street addresses, phone numbers outside the North American format, and tokens that fail the Luhn or mod-97 check. A clean report means only that the five patterns found nothing.
Why must a harness file be converted first?
A harness writes its own column names and nesting. The schema check reads the input column and the published eval-set schema's other field names, which the harness file does not use. Convert it with the dataset converter first, then load the converted output.