INSTRUMENT | Eval statistics
N-Gram Overlap Checker: Find Contamination Between Two Eval Files
4 cited sources
Check a test set against a training file for shared word sequences. Pick n, read the per-row dirty-token share, see the matching text in both files, and export a CSV. Both files stay in your browser.
Matching whole rows across two splits misses a test set that leaked in pieces. This page checks two files for shared word sequences, the method the contamination literature runs on, and shows the matching text in both.
Neither file leaves your browser. Default n is 13, the GPT-3 convention. Paraphrased leakage still gets past it. The schema validator reads the same files, and what LLM evals certify covers what a benchmark score is meant to prove.
Showing your last valid result. Update the inputs above to recompute.
Rows flagged
66.7%
2 of 3 checkable rows share at least one 13-token sequence with the other file, at n = 13 and a 0 percent threshold. 1 row is shorter than 13 tokens and 0 are blank, so they sit outside every denominator. Worked example, preloaded in both slots.
test-set.json checked against training-corpus.json at n = 13 with a 0 percent threshold. Tokens split on runs of letters and digits, then normalized with L1+L3+L4.
What the run counted
| Measure | Value |
|---|---|
| Rows flagged | 2 |
| Rows checkable | 3 |
| Overall dirty-token share | 31.3% |
| Rows shorter than 13 tokens | 1 |
| Rows with no text | 0 |
Every checkable row, worst first.
| Row | Tokens | Dirty | Span | Windows | Best B row |
|---|---|---|---|---|---|
| a1 | 18 | 72.2% | 13 | 1 | b1 |
| a2 | 40 | 32.5% | 13 | 1 | b2 |
| a4 | 25 | 0.0% | 0 | 0 | none |
The two panes below mark only the spans the best-matching B row supplied.
Check this value.
test-set.json, row a1
In 1969 the Apollo 11 mission landed the first humans on the Moon and returned them safely home.
training-corpus.json, row b1
Fact: in 1969 the Apollo 11 mission landed the first humans on the Moon, a milestone.
dirtyShare(row) = covered tokens / tokens in the rowHow?
How this is calculated
Every row of both files is split into word tokens. Loose splits on runs of letters and digits and then folds each token to lower case in canonical compatibility form; strict splits on whitespace only and folds nothing, so case and punctuation still count. Normalization runs on one token at a time, never on the whole row, which is what lets every token keep the character range it came from. That range is what the highlight below the table is drawn from.
Every window of n consecutive tokens in file B is hashed once into an index. File A is then walked once against that index. There is no comparison of every A row against every B row anywhere on this page, which is why the cost grows with the number of tokens rather than with the number of rows squared. Each hash hit is verified token by token before it is accepted, so a hash collision cannot draw a span whose words differ between the two panes. At most 32 positions are kept per window; when that cap is reached the page says so above.
A row's dirty-token share is the count of its tokens covered by any verified match, divided by its token count. Overlapping windows and repeated windows are counted once, so the share can never pass 100 percent. Coverage may come from several B rows. The longest shared span and the best-matching B row are computed inside one B row on one diagonal, so the span the page reports is one that a single row of file B really holds. A row is flagged when it reaches n tokens, has at least one verified match, and its dirty share is at or above the threshold, compared on the unrounded value.
A row shorter than n tokens is counted separately and leaves every denominator: it cannot match an n-token window, so calling it clean would deflate the rate. A row whose text is empty is counted as blank and leaves every denominator too. Percentages are rounded to one decimal at print time only; counts and span lengths are whole numbers. At most 200 rows render, and the CSV carries every checkable row along with the character offset of its longest span in both files.
The defaults are the field's, not ours. The 13-token window is the convention GPT-3's report established and lm-evaluation-harness carries: "They used a range of N values between 8 and 13 depending on dataset, while we just used 13 for simplicity." The zero threshold is the same source's any-match rule, where a single shared window is enough to call a test row contaminated. The 8-token preset is the bottom of that stated range.
For a sense of what this finds in a real set: the MTEB leakage issue linked under Sources reports, for the French test split of one intent-classification dataset, 125 train-to-test leaks, 36 validation-to-test leaks, and 30 duplicated test rows, and calls 6.422 percent of that test split biased. It puts leaks in 24 percent of the English MTEB datasets and 46 percent of the French ones.
Formula: dirtyShare(row) = covered tokens / tokens in the row
Questions
Why 13 tokens by default?
Because it is the number the field publishes against. GPT-3's report used a 13-gram overlap test, lm-evaluation-harness adopted 13 for every dataset rather than varying it, and the contamination survey under Sources records both that rule and GPT-4's 50-character one. An n chosen here instead would produce a number nobody could compare to a published one.
Move it down and you catch shorter borrowings and more coincidence; a short window over ordinary prose matches on stock phrases. Move it up and only long verbatim runs survive. The control takes 5 to 25, and the page prints the n every result was computed at.
Why does one row show a span shorter than its dirty share suggests?
Because the two answer different questions. The dirty share counts every token of the row that sits inside any matched window, wherever the match came from, so two different B rows each supplying half a sentence add up. The longest span is the longest unbroken run inside one B row, because that is the thing you can go and read.
Merging runs from different B rows would report a passage that no single row of file B contains, and the pane below the table would then highlight fewer words than the number claimed.
What counts as a token?
Under loose matching, a run of letters, digits, or combining marks. Punctuation and spacing are separators and disappear, and each surviving token is folded to lower case in canonical compatibility form, so "Moon," and "moon" are the same token and a full-width letter matches its plain form. Under strict matching, a token is a run of non-whitespace and nothing is folded at all, so "Fact:" stays one token with its colon.
Neither rule is a model tokenizer. A subword tokenizer would split on a vocabulary that changes with the model, and a result computed against one model's vocabulary is not reproducible against another's.
Do my files go anywhere?
No. Both files are read, parsed, tokenized, indexed, and compared in this tab. There is no upload, no server call, and no analytics payload that carries a file name, a size, a row count, a field name, a matched span, or any cell text. The CSV is built in the browser from what is already on this page.
The page does load analytics, and it counts that a check ran and how many times you have run one in this visit. Those two numbers are all it sends.
Why is there a limit of 2 MB per file?
Because the work happens on your machine, in one tab, while the tab still has to answer you. The index holds every window of file B, so memory grows with the size of that file, and the bound is set where a laptop stays responsive rather than where the algorithm gives out. Over it the page states the bound and the size of the file you picked, and reads nothing: a truncated file would produce a rate for a set you do not have.
Sources
- Benchmark Data Contamination of Large Language Models, a survey: the source of the 13-gram and 50-character detection conventions this page defaults to and names
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples: the measured limit stated beside the result, that paraphrase and translation bypass string-matching decontamination
- lm-evaluation-harness decontamination documentation: the 8-to-13 range behind the wider preset, the choice of 13 for simplicity, and the any-match rule the zero threshold implements
- MTEB issue 1036, train-to-test leakage in the benchmark: the per-dataset leakage counts quoted in the method note, reported as the issue states them