Share of train (8 of 12 rows)
66.7%
INSTRUMENT | Eval statistics
Cross-Split Overlap Checker: Find Eval Items in Both Train and Test
5 cited sources
Compare two eval splits in your browser and see which items sit on both sides, with a separate overlap rate against the row count of each split, exact and normalized matching, and the lines to open.
An item in both your train and test splits makes a score look better than the system is. Load two files, pick the field to compare, and read the overlap against each split's own size, never a pooled one.
Repeats inside one file are the exact-duplicate check, near-duplicates the normalized one, file shape the schema validator. A sentence hidden inside a longer item needs an n-gram check. Web-scale contamination and how benchmarks get gamed are the wider reading.
Showing your last valid result. Update the inputs above to recompute.
This is the bundled example: a twelve-row train split against a ten-row test split. Load your own two files and it is replaced.
No result on screen. The message above says what to fix.
Items in both splits
7
7 items sit in both splits, 8 of 12 rows in train and 7 of 10 rows in test. Exact match, on the column "input".
Share of test (7 of 10 rows)
70.0%
7 items sit in both splits under exact matching. Normalizing finds no further items, though it does reach 1 more row.
Each rate is that side's affected rows over that side's own row count.
| Check | Items in both | Rows in train | Rows in test |
|---|---|---|---|
| Exact | 7 | 8 (66.7%) | 7 (70.0%) |
| Normalized | 7 | 8 (66.7%) | 8 (80.0%) |
| Identifier | 0 | 0 (0.0%) | 0 (0.0%) |
The content overlaps while the identifiers do not, so the item was re-keyed at split time. Matching on ids alone would have reported this set clean.
Normalized profile: canonical form (NFC), compatibility fold (NFKC), case fold, remove punctuation, collapse whitespace.
L1+L3+L4+L8+L10
Open these lines on both sides.
| Match | Item | Lines in train | Lines in test |
|---|---|---|---|
| exact | What is the capital of France? | 2, 9 | 2 |
| exact | Who wrote Pride and Prejudice? | 3 | 3 |
| exact | What is the boiling point of water in Celsius? | 4 | 4 |
| exact | How many continents are there? | 5 | 5 |
| exact | What is the largest planet in the solar system? | 6 | 6 |
| exact | Who painted the Mona Lisa? | 7 | 7 |
| exact | What year did the Berlin Wall fall? | 8 | 8 |
| normalized | Who painted the Mona Lisa? | 7 | 7, 9 |
rate = rows carrying an overlapping item / rows in that splitHow?
How this is calculated
Two rates, one per split, each against that split's own row count. A single pooled ratio is the one number this page refuses to print: it hides which side the leak lands on, and the side it lands on is what decides whether a published score is still readable. Blank rows stay in the denominators, because dropping them flatters the answer.
Every selected field is turned into text first, so a column holding 42 in a CSV matches the same column holding the number 42 in a JSONL file. Those values then become one row key through a length-prefixed, type-tagged encoder, never a joined string: a separator inside a joined string is itself part of the data, so two rows carrying a and bc would collide with two carrying ab and c. A field that is absent stays apart from a field that is present and empty, so a column your export dropped never reads as a blank answer. Rows are keyed once per side and the two key sets are intersected in one pass, which is why 200,000 rows a side costs one pass rather than forty billion comparisons.
The normalized rung runs the same arithmetic over a fixed transform: compatibility folding (NFKC), the full Unicode case-folding table, punctuation removal, and whitespace collapse. It is always on and sits beside the exact answer rather than replacing it, because an exact-only check reports clean on a set full of paraphrases: Lewis and colleagues found that 30 percent of test-set questions in open-domain question answering had a near-duplicate paraphrase in train. The transform is printed above as a short stable string, so a result is reproducible from that string alone. NFKC is our own addition to the recipe. It does not fold curly quotes, which leave at the punctuation step instead.
Article stripping is the one switch. It removes a, an and the as whole words after the case fold, which is the SQuAD answer-normalization recipe. It is off by default because it is English-only and quietly corrupts every other language, and the printed transform says which state produced the answer.
The identifier check is separate and unnormalized. Ids are regenerated when a set is re-split and content is not, so the two checks catch different defects: an item flagged by both leaked with its id attached, an id-only match means an identifier was reused, and a content-only match means the item was re-keyed on its way into the second split.
A row number is the source record, counting every record the file holds. A line number is the
file line the record starts on, which is what your editor will jump to. Percentages round to one
decimal; a nonzero share that would print as 0.0% shows as <0.1% and a share
below one that would print as 100.0% shows as >99.9%. At most 200 items render, each item's text is clipped at 80 characters, and at most 3 line numbers a side render with a count of the rest; the CSV carries every item, every row
number and every line number. When a key matches more than one row, the item column shows the
source text of the first matching row on side A.
Formula: rate = rows carrying an overlapping item / rows in that split
Questions
Why two rates instead of one overlap percentage?
Because the two splits are usually different sizes and you act on them differently. Two hundred shared items in a 200,000-row training set is a rounding error on that side and a catastrophe in a 400-row test set, and one pooled ratio reports the same number for both. The benchmark-side record makes the point: the MTEB issue that first documented this problem counts train-to-test and validation-to-test leakage separately, and its author calls 6.422 percent of one French test split biased on an additive count of the leaks found, with no union taken over the affected test rows. That is the issue author's own metric, quoted as stated, and it is not a figure this page recomputes or calibrates against.
What does an overlap do to a score I have already published?
It moves the score toward whatever the system memorized. An item the model saw in training and is then graded on is not measuring generalization, so the reported number stops standing for the thing it claims to measure, which is a construct-validity failure rather than a statistical one. Wider confidence intervals do not fix it, and neither does more data of the same kind. Cut the shared items from the training side, re-run, and quote the new number.
The identifiers match but the content does not. Is that a leak?
It is a different defect. Ids get regenerated at split time and reused by accident, so the same id on both sides usually means an identifier collision rather than a shared item. Read the two blocks together: content and identifier both flagging one row means the item leaked with its id attached; identifiers alone means an id was reused; content alone means the item was re-keyed on its way into the second split. That last case is the one a check matching only on ids reports clean.
Why is normalized matching always on?
Because switching it off would hide the finding. Exact matching reports a set clean when the same question sits in both splits with different capitalization, a different quote character, or one extra space, and that is the common shape of a real leak rather than an unusual one. The two rungs sit side by side so you can see how much of the overlap is invisible to a byte-exact check, and the transform that produced the normalized answer is printed with it.
Does either file go anywhere?
No. Both files are read, keyed, and compared in this tab. There is no upload, no server call, and no analytics payload carrying a file name, a size, a row count, a column name, a split value, a key, or a cell value. The export is built in the browser from what is already on this page.
What if my splits are in one file?
Switch to one-file mode and pick the column holding the split label. Every distinct value in that column is offered, ordered by how many rows carry it, and you choose the two to compare. Row and line numbers stay those of the original file, so the lines the findings table gives you are the lines to open. A column with one distinct value has nothing to compare and the page says so rather than reporting a clean zero.
Sources
- MTEB issue 1036: the primary record of cross-split leakage in benchmark datasets, counting train-to-test and validation-to-test leaks separately rather than pooling them
- Benchmark Data Contamination of Large Language Models: a Survey. The four contamination levels and the standing n-gram thresholds, 13-gram overlap for GPT-3 and 50-character overlap for GPT-4
- Lewis, Stenetorp and Riedel, Question and Answer Test-Train Overlap in Open-Domain QA: 60 to 70 percent of test answers appear in train, and 30 percent of test questions have a near-duplicate paraphrase there, which is why exact matching alone reports a dirty set clean
- SQuAD v1.1 official evaluation script: the answer-normalization recipe this page follows, lowercase then remove punctuation then remove the articles a, an and the, then fix whitespace
- EleutherAI lm-evaluation-harness decontamination docs: 13-grams as the convention, about four days to generate them from the Pile and about five more to sort them, and a harness that drops the contaminated eval document rather than cleaning the training set