Repeated ids
0
INSTRUMENT | Eval statistics
Eval Dataset Schema Validator: Golden-Set Checks and Export Profiles
5 cited sources
Check an eval golden set against a neutral schema in your browser: findings by row and line, duplicate ids, field completeness, plus a template and exports for four eval tools.
An eval score is only as good as the file behind it. What an eval measures starts with rows that carry a stable id and a real input, which is what a reproducible eval rests on.
This page checks one file against a published schema in your browser, names the ids that repeat, and counts empty fields, which is a claim about sample size first. Then: byte-identical rows, near-duplicate wording, train/test overlap, shared n-grams, slice balance. Freeze the clean file.
Showing your last valid result. Update the inputs above to recompute.
Nothing loaded yet. The schema, an example row, and the templates are below, and the limits are printed above the picker.
Schema errors
0
Load a file to check it against the published schema.
Findings, most serious first
Nothing to report. No rule fired on this file.
Duplicate ids and completeness
Two answers on the same file: which example ids repeat and on which rows, and how full each column is. The controls for both sit with the picker above.
Rows in a repeat
0
Distinctness
n/a
Uniqueness
n/a
Completeness, one row per column
| Column | Maps to | Kind | Tier | Present | Absent | Null | Empty | Fill |
|---|
Repeated ids
| Id | Rows | Data rows | File lines |
|---|
Template, schema, and exports
The template and the schema download whether or not a file is loaded. The profile exports need a set with no error findings.
Load a file to see what each export can carry.
The v1 field list
| Field | Type | Required | Rule |
|---|---|---|---|
| id | string | Yes | Not empty after trim, and unique across the set. |
| input | any | Yes | Present and not null. A string value has to hold a non-space character. |
| expected | any | Optional | The key may be absent and the value may be null. Either way it is a warning, never an error. |
| split | string | No | A free string. Train, dev and test are conventions here, not an enum. |
| slice | string[] | No | A CSV cell splits on a pipe. |
| tags | string[] | No | A CSV cell splits on a pipe. |
| metadata | object | No | Free. Unknown CSV columns collect here. The key evalset is reserved. |
One row, in the canonical form:
{
"id": "checkout-001",
"input": "Refund the last order and email the customer.",
"expected": "refund_issued",
"split": "test",
"slice": [
"billing",
"multi-step"
],
"tags": [
"regression"
],
"metadata": {
"author": "qa",
"sourceTicket": 4821
}
}findings = schema errors + duplicate ids + completenessHow?
How this is calculated
The published file at /schemas/eval-set/v1.json is the whole contract. The page runs that file rather than restating its rules, so the schema an outside validator fetches and the verdict printed here cannot drift apart. Two checks sit outside the schema because JSON Schema cannot state them: whether an id repeats, and how full each field is.
A row number is the data row with any header excluded. A line number is the file line the row starts on, which is not the same thing once a quoted CSV field spans two lines. Percentages are rounded to one decimal, half up. Counts are whole numbers.
| Rule | Severity | What fires it |
|---|---|---|
| id.required | Error | A row has no id, or its id is empty after trimming, or the id is not text after coercion. |
| id.duplicate | Error | Two or more rows share an id. Keys are compared as the JSON string form, trimmed at both ends, byte for byte. |
| id.coerced | Info | A numeric or boolean id was read as its string form. Repeats fold into one summary carrying the count and the first 20 rows. |
| input.required | Error | A row has no input, or it is null, or it is a string with nothing but spaces in it. |
| expected.missing | Warning | The row carries no reference output. One warning per row, and no matching info finding. |
| field.absent-everywhere | Warning | An optional field carries no value on any row. Split has its own rule, so it is not counted twice. |
| field.type | Error | An optional field is the wrong type: a split that is not a string, or a slice that is not a list of strings. |
| split.absent | Info | No row names a split anywhere in the set. |
| slice.single-value | Info | Every slice label in the set reads the same, so a per-slice breakdown has one bucket. |
| metadata.reserved | Error | A row sets metadata.evalset. That key is reserved for carrying these fields through a vendor export. |
| row.malformed | Error | A line could not be read: bad JSON, a value count that does not match the header, or a key the schema does not define. |
| row.unknown-key | Info | A column or key the schema does not name was folded into metadata. Repeats fold into one summary. |
| column.duplicate-header | Error | Two CSV columns share a header name. Both column numbers are printed. |
At most 200 findings render, most serious first, with a count of what is not shown. The findings CSV carries every one.
The four ways a value can be missing
The completeness table splits them and never adds them up for you, because they mean different things: absent is an exporter that never wrote the column, null is a value that was written and is empty, and an empty string is usually an authoring gap. A cell is absent when there is no such key or column, null when the value is JSON null, empty when it is an empty string, whitespace only, an empty list, or an empty object, and present for everything else, including 0 and false. Missing is the three of them together. Tick the widening above and the tokens NA, N/A, null, None, - join the empty set for that table alone; the schema verdict never sees them, so this page and a strict validator read "N/A" differently on purpose.
Three ratios, three names
Fill rate is present over rows, per column. Distinctness is how many different ids the file holds per row that carries one. Uniqueness is how many ids occur exactly once, again per row that carries one, which is this page's own definition and is printed with its denominator named. On the ids a1, a2, a1, a3, a4 that is 4 of 5 distinct, so 80.0%, and 3 of 5 singletons, so 60.0%. The Deequ paper divides singletons by distinct values rather than by rows, which would read 75.0% on the same file. Printing one of these under the other's name is the mistake the two labels exist to prevent.
Every percentage is rounded to one decimal, half up, under two guards: it never reads 100.0% while a value is missing, and never 0.0% while a value is present. A rounded percentage that contradicts the exact count printed beside it is worse than one decimal less precision.
Formula: findings = schema errors + duplicate ids + completeness
Questions
Why is the schema vendor neutral?
A format tied to one vendor inherits that vendor's calendar. A hosted eval product can go read-only and then close, and a set written in its shape has to be rewritten by hand when it does. The neutral form here is ours, it is published at a stable URL, and it converts out to promptfoo, DeepEval, LangSmith and Braintrust. Those four are named as export targets and nothing more.
One vendor format is deliberately absent: OpenAI Evals. It is named here so you know the omission was a choice.
How does versioning work, and why does v1 never change?
The version lives in the path. Version 1 is served at /schemas/eval-set/v1.json and stays served. Inside a published version only additive, optional changes are allowed: never a new required field, never a narrowed type, never a removed or renamed field, never a changed meaning. Anything else publishes v2 at a new path beside v1, not over it.
The file carries an x-publishedOn annotation, set the day it landed and never edited. That is the date printed beside the version above, so what you read on this page and what a validator fetches are the same claim.
What counts as required, and who decided?
Four layers, and each finding says which one it came from. The published schema decides the first two: the fields it lists as required are errors, and a field it leaves optional and lets you set to null is reported at warning and never failed. That second tier is why a golden set with no reference answers is reported rather than broken, which matters because most eval formats treat the reference output as optional.
The third layer is yours: tick a column beside the picker and every row missing a value in it is reported at warning, naming you rather than the schema. The fourth is everything else, which still gets a fill rate and no finding, because a sparse optional column is information, not a fault. This page never decides requiredness on its own, and it reads the list off the published file at run time rather than keeping a copy that can drift.
Why is a missing id not a duplicate?
A row with no id leaves the duplicate check entirely and is counted once, under completeness. Fold them together instead and an export with no id column at all, which is what several eval tools produce, reports one enormous duplicate group holding the whole file. The panel says how many rows were set aside and why.
Ids are compared as text, trimmed at both ends, byte for byte, after a numeric id is read as its string form, so 1 and "1" are one id and the finding says the types differed. Case and invisible characters are a separate warning class rather than part of that comparison: two ids that differ only in case, or only in a zero-width character, are almost always a mistake, but they are not the same id and this page will not silently treat them as one. Turn case sensitivity off and that fold moves into the main comparison, where you can see it.
What does this tool not check?
Everything here is deterministic. There are no embeddings, no semantic similarity, and no comparison against a web corpus, so it cannot tell you whether two rows mean the same thing in different words, and it cannot tell you whether a public benchmark leaked into a training set. Those are real problems and this is not the instrument for them.
It also does not read your file anywhere but in this tab. There is no upload, no server call, and no analytics payload that carries a file name, a size, a row count, a column name, or a cell value.
Sources
- JSON Schema: the $schema keyword declares which dialect a document is written in (2020-12 is the dialect this validator runs)
- MLCommons Croissant 1.0: dataset conformance declared by a URL, the model for pinning a schema version in its own path
- Frictionless Table Schema: a primary key is equivalent to adding required true to its fields, and the default missing-value set is the empty string alone
- JSON Schema object keywords: required constrains whether a property is present, so a property set to null is present and not absent
- Navigating Dataset Documentations in AI: all 7,433 Hugging Face dataset cards, where completion rate tracks popularity and whole sections go unfilled