Code grader playground: score one expected and actual pair

Exact match compares the two texts as sequences of Unicode code points with no normalization. A single differing code point, including invisible characters such as a no-break space or a byte order mark, makes the pair fail. The explanation names the first differing position, both characters with their Unicode code, and flags a leading or trailing whitespace difference, a case-only difference, or an NFC- or NFKC-only difference.

Normalized exact match runs the selected pipeline on both sides and then applies code-point equality. The default pipeline trims leading and trailing whitespace (the 29 characters Python's str.strip() removes) and lowercases both sides. SQuAD answer normalization lowercases, removes ASCII punctuation, removes English articles (a, an, the), collapses runs of whitespace to one space, and trims.

Contains checks whether the output contains each expected reference (any one or all of them) with no normalization. The match can be required anywhere or at the start.

Regex compiles the pattern as a JavaScript regular expression with the u flag always on. Anywhere finds the first match in the output. At the start pins the match to index 0. Whole text requires the entire output to match: the pattern is wrapped so that alternatives can backtrack to cover the full string. Expected text is not used. A Python-syntax pattern is refused with a hint naming the JavaScript equivalent.

Numeric reads one number from each side. Whole text requires the entire trimmed string to be a number. First number and last number find the first or last ASCII numeral sequence that does not follow a letter, digit, underscore, period or comma, and does not run into a digit or into a period or comma plus a digit, so v2.0 scored 0.83 reads 0.83 and The refund is $42.00. reads 42. Commas count only as exact three-digit separators. A leading sign counts only at the start of the text, after whitespace or after (. 50% is not read as 0.5. The pair passes when the absolute difference is within the larger of the absolute tolerance and the relative tolerance times the size of the expected value. When the expected text is not a number, the grader returns not judged.

JSON Schema parses the output as JSON after any selected pre-parse step and validates it with the schema evaluator. A schema keyword outside the supported set makes the result not judged. A parse failure shows the browser's own error message. A fail names each violation by JSON Pointer path, keyword and message.

Token F1 applies SQuAD answer normalization by default, splits on runs of Python's 29 whitespace characters, and computes bag-of-tokens precision, recall and F1. When both sides are empty after normalization (for example the against a), F1 is 1 by the SQuAD convention; one side empty gives 0. With multiple references, the grader takes the maximum F1.

BLEU computes sentence-level BLEU-4 with uniform weights, 13a tokenization (which splits off most punctuation, including _, and keeps a hyphen between letters, so refund-status stays one token), clipped n-gram counts and effective order: an order with no output n-grams is dropped. Exponential smoothing fills orders with no matches: the first such order gets a precision of 1 divided by twice its n-gram count, the next 1 divided by four times its count, and so on. This is method 3 in Chen and Cherry, "A systematic comparison of smoothing techniques for sentence-level BLEU" (WMT 2014). When the output is shorter than the closest reference, the brevity penalty is exp(1 minus the reference length divided by the output length); a tie in closeness takes the shorter reference. The score is 0 when no n-gram matches at any order. Post, "A call for clarity in reporting BLEU scores" (WMT 2018), found that BLEU on the same test set differs by up to 1.8 points on the 0 to 100 scale across common configurations, mostly through tokenization and normalization, so this page states its settings.

ROUGE-L tokenizes by lowercasing and splitting on every run of characters that are not Unicode letters or numbers. The longest common subsequence is found by dynamic programming. Precision is LCS divided by the output token count, recall is LCS divided by the expected token count, and F1 uses beta 1 (Lin, "ROUGE: a package for automatic evaluation of summaries," 2004, defines ROUGE-L with beta set very high for recall; this page computes F1 and states it). When one side is empty the score is 0. Pairs whose token counts multiply to over 25,000,000 are not computed.

chrF removes all whitespace from both sides and counts character n-gram matches for orders 1 through 6 over Unicode code points, with clipped counts against the best reference. Precision and recall are their means over orders where both sides have n-grams. The F-score uses beta 2, which gives recall twice the weight of precision: F = (1 + 4) times precision times recall, divided by 4 times precision plus recall, where 4 is beta squared. Beta 2 was the most promising setting across several values in Popovic, "chrF deconstructed: beta parameters and n-gram weights" (WMT 2016).

A near-miss probe runs on every fail. For exact match, normalized match and contains, the probe tries each formatting step on its own, in a fixed order (trim, lowercase, NFKC, collapse whitespace, strip trailing period or exclamation mark, join words with underscore, strip quotes) and names every step that turns the fail into a pass. If no single step passes, the probe runs all seven together; a pass then reads "several formatting differences." For regex, the probe tests with the i flag added. For JSON Schema, the probe tries stripping a code fence, then trimming. For numeric, the probe tries the other read modes. For continuous graders with a threshold, the probe flags a score within 0.1 below the threshold.

These overlap graders follow the settings stated above and have not been compared against a reference library.

Formula: Each grader runs independently on the same expected and output pair.

See Sources below