Eval data quality calculators
Every other number on this shelf comes out of a run. These come out of the file that goes in: whether the set is well formed, whether rows repeat, and whether the test half leaked into training.
What these numbers mean
Most of what goes wrong in an eval set goes wrong before the run. The failure that costs the most is the same item sitting in the file you trained on and the file you scored on. Train test overlap is the plain name for it, test set contamination is the name it carries in papers, and it lifts a pass rate in a way no confidence interval catches. Everything below reads your own files in the browser and uploads nothing.
Work down the table. The first check sits above the rest, because every checker here reads a file through the same parser. The five in the middle run cheapest first, and each one assumes the check above it came back clean. The last asks what the set contains at all, which is a different question from whether anything repeats.
| What you have | The instrument |
|---|---|
| An eval file and no proof it parses | check it against a schema |
| One file, and no count of the rows that repeat inside it | group the byte-identical rows |
| An id column, and cells that came in empty | check the ids and the empty cells |
| Two rows that read the same, with different spacing or case | normalize the text, then regroup |
| A train file and a test file | find the items sitting in both |
| Training rows long enough to hide a test item inside them | look for shared word sequences |
| A pass rate quoted over slices nobody has counted | see what the set actually holds |
The reading behind these checks sits next door. Which leakage checks you can actually run ranks the same rungs and is straight about what a clean result never proves. How a leaderboard number gets gamed covers a different route to a score nobody earned, through the harness and its grader rather than through the data. Construct validity is the term for what a benchmark has lost by the time either one is true.
The choice people get stuck on is between the two checks that read a pair of files. Comparing whole items across the two splits is cheaper, and its finding is unambiguous: this row is on both sides. Looking for a shared window inside a row catches what whole-item matching cannot, a test item that leaked as one sentence inside a longer training item, and it costs more to run and more to defend. Run the cheaper one first. Dedupe before either, because duplicate rows sit in the denominator of every rate below them and make an overlap rate read lower than it is.
Tools in this topic
Eval Dataset Schema Validator: Golden-Set Checks and Export Profiles
Check an eval golden set against a neutral schema in your browser: findings by row and line, duplicate ids, field completeness, plus a template and exports for four eval tools.
Instrument | Eval data qualityExact Duplicate Row Checker for Eval and Golden Datasets
Find byte-identical rows in an eval or golden dataset without uploading it. Duplicate groups with row and line numbers, an identity key you choose, and the sample size left after a dedupe.
Instrument | Eval data qualityNear-Duplicate Row Detector: Find Eval Rows That Differ Only in Formatting
Find rows in an eval set that are the same test written twice. Normalize the columns you pick, group rows whose normalized text matches, and read the exact transform, the raw variants, and a CSV.
Instrument | Eval data qualityCross-Split Overlap Checker: Find Eval Items in Both Train and Test
Compare two eval splits in your browser and see which items sit on both sides, with a separate overlap rate against the row count of each split, exact and normalized matching, and the lines to open.
Instrument | Eval data qualityN-Gram Overlap Checker: Find Contamination Between Two Eval Files
Check a test set against a training file for shared word sequences. Pick n, read the per-row dirty-token share, see the matching text in both files, and export a CSV. Both files stay in your browser.
Instrument | Eval data qualitySlice and Class Balance Analyzer: See What an Eval Set Contains
Read what an eval set holds before you quote a pass rate. Count rows per slice, per class, and per control flag, put an interval on every share, and see the smallest slice named.
Analyses that use these calculators
- Eval statistics
Benchmark contamination: what you can check and what you cannot
Benchmark contamination is two problems under one name. Leakage inside your own splits is checkable in a browser in minutes. Corpus-level contamination is a research problem.
- Eval statistics
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
- Reliability testing
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
- Reliability testing
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Reliability testing
Agentic AI testing beyond a single eval run
Single-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.