INSTRUMENT | Eval statistics
Exact Duplicate Row Checker for Eval and Golden Datasets
1 cited source
Find byte-identical rows in an eval or golden dataset without uploading it. Duplicate groups with row and line numbers, an identity key you choose, and the sample size left after a dedupe.
Count a row twice and it sits in the denominator twice, so the sample size behind an agent eval is smaller than it looks and the interval around a pass rate is narrower than the evidence earns.
This page finds rows that are byte-identical once the ends are trimmed, names the fields it compared, and stops there. Case, spacing inside a value, and paraphrase are the next rung up.
Showing your last valid result. Update the inputs above to recompute.
Identity key: input, expected, split, slice, tags. Whitespace: trim, so the ends of a value are ignored.
This is a bundled ten-row example. Load your own file and it is replaced.
Rows in duplicate groups
5
5 rows in 2 duplicate groups. Duplicate rate 30.0% of 10 rows. A dedupe would leave 7 unique rows, which is the sample size behind any rate you quote over this set.
Counts for the fields in the key above.
| Measure | Rows |
|---|---|
| Rows parsed | 10 |
| Distinct keys | 7 |
| Duplicate groups | 2 |
| Rows in those groups | 5 |
| Rows removed if deduped | 3 |
| Unique rows after | 7 |
Carry 7 unique rows into the pass-rate confidence interval whenever you quote a rate over this set.
One row per duplicate group. A row number excludes the header, so in a headered file row 1 is file line 2. Where a file's rows and lines do not run in step, the file line follows its row in brackets, and that is the line to open in an editor.
| # | Size | Rows | Drops | Shared value |
|---|---|---|---|---|
| 1 | 3 | 1, 3, 6 | 3, 6 | input=Refund the last order | expected=refund_issued | split=test |
| 2 | 2 | 2, 8 | 8 | input=Cancel the renewal | expected=cancelled | split=test |
r = d - g, u = n - rHow?
How this is calculated
Every row is turned into one string key built from the fields you ticked, and the keys go into a map. A key value holding two or more rows is a duplicate group. Let n be the rows that parsed, d the rows whose key is shared, and g the number of shared key values.
| Quantity | How it is computed | Note |
|---|---|---|
| Rows in duplicate groups (d) | count of rows whose key is shared | A key is shared when two or more rows produce it. Every member counts, including the one a dedupe would keep. |
| Duplicate groups (g) | count of shared keys | One group per shared key value, numbered in the order it first appears in the file. |
| Rows removed if deduped (r) | r = d - g | One row survives each group, so a group of three costs two rows and a group of nine costs eight. |
| Unique rows after (u) | u = n - r | The same number as the distinct-key count. This is the sample size behind any rate quoted over the deduplicated set. |
| Duplicate rate | r / n | Printed to one decimal, rounded half up from the exact ratio and never from rounded parts. The counts themselves are never rounded. |
The key is built by encoding each field's value with its own type tag and its own length, then
joining the codes. A plain delimiter join would report false duplicates the moment a value
contains the delimiter, and it would read the list ["a", "b"] and the string
"a,b" as the same thing. Absent, null, and an empty string stay three different keys.
Nothing is hashed: a map is exact at this size, and hashing would add a collision case that buys
nothing.
A data row number excludes the header. A file line number is the line the row starts on, which stops matching the row number the moment a quoted field spans two lines, so both are printed and the line is the one to open in an editor.
In the group table, # is the group number and Drops is the members the removal preview would remove, so switching between keeping the first and keeping the last changes that column and nothing else.
At most 200 groups render, and a group larger than 500 members shows
its first 50. The CSV carries every group and every member row, and writes the group
id in full as row.duplicate-exact:n.
Formula: r = d - g, u = n - r
Questions
Which fields should be in the identity key?
The content ones. If the file carries an id and an input, the key starts as the input, the expected output, the split, the slice, and the tags, and it leaves the id and the metadata out. An id is a handle rather than content, and a unique id on every row gives every row its own key and returns a confident false clean. Metadata is provenance: one case added twice by two annotators is still one case.
Any other file starts with every column ticked, which is what pandas does with
subset=None. If one of those columns turns out to be an id, the note under the
selector names it and you untick it in one click. Repeated ids are a separate check, and it
runs on the schema validator's duplicate id and completeness section.
What counts as trivial whitespace, and what does not?
In the default mode a value is compared after both of its ends are stripped and a Windows line break inside it is read as a plain newline. That takes a trailing space and a trailing non-breaking space with it. Nothing else moves: upper case stays different from lower case, two spaces in the middle stay different from one, an interior non-breaking space stays different from a plain one, and two spellings of the same accented character stay different because they are different bytes.
Switch to byte for byte and even the ends count. The mode in use is printed above the result in both states, because the same file honestly gives two different counts under the two rules. Case, internal spacing, punctuation, and Unicode form belong to the near-duplicate check, which is the next rung up.
What does this tool not do?
It does not match on meaning. There is no similarity score, no threshold, no embedding, and no fuzzy match, so two rows that ask the same question in different words are two rows here. It does not hand back a cleaned file either: it reports what repeats and where, and you edit your own set.
It also does not read your file anywhere but in this tab. There is no upload, no server call, and no analytics payload carrying a file name, a size, a row count, a column name, a key, or a cell value.
How do duplicated rows change what an eval number means?
A duplicated row is counted once per copy, so the set has fewer independent items than its row count says. That inflates the denominator of any rate you quote and narrows the interval around it, which is the reproducibility problem in running the same eval twice and the construct-validity problem in claiming the set measures what you say it measures. This page reports the count and the sample size left; it reads no scores, so it cannot tell you which way a pass rate moves.
The same pair appearing in a training split and a test split is a different problem again, and a cross-split overlap check owns it rather than this one.
Why does the page list rows with a blank key separately?
Because every one of them would otherwise land in one enormous group. A row whose selected fields are all absent, null, or empty has no identity to compare, so it is listed on its own and counted as its own row in the distinct-key total. The removal arithmetic still balances: unique rows after equals rows parsed minus rows removed, whether or not blanks are present.