LatentEval

TOOL | Evaluation design

What is your structured output failure rate?

Paste or upload a file of model outputs and see your failure rate by class. Each rate carries a Wilson interval, and the page produces the two numbers the retry optimizer needs.

Paste or upload a file of model outputs. The page classifies each response into the four failure classes from our structured-output research, counts the rate per class with a Wilson interval, and hands the detected-failure share to the retry optimizer. This page measures your rate; the research page documents why each class happens across providers.

One raw model output per record, pasted or opened as JSONL, CSV or JSON. What you paste or open is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it. Files over 10 MB or 200,000 records are refused rather than read in part.

CSV, JSONL, NDJSON, or JSON. Drag one here, or use the box below.

These outputs are a six-record sample with two planted failures. Clear them and paste your own. Up to 2,097,152 characters; use the picker for anything larger.

Check this value.

Format

The schema you asked the provider to follow. Re-validating locally catches the cases the vendor grammar let through.

JSON. Drag one here, or use the box below.

Check this value.

Calls the provider refused with a 400 before a token was generated, counted from your logs. Typing the count keeps the denominator honest; leaving it empty means the rates are over delivered responses only.

Check this value.

Confidence level for the Wilson intervals on each rate.

Check this value.

Unparseable rate

33.3%

2 of 6 | 33.3% | 95% CI [9.7%, 70.0%]

33.3% of responses were unparseable (2 of 6, 95% CI [9.7%, 70.0%]).

Counts class 3 (truncated output) together with parse failures whose cause could not be assigned to a specific class, because both produce a response your code cannot use.

At 6 records, the Wilson interval is wider than most decisions need. The sample-size calculator shows how many outputs to collect for a tighter bound.

Each class against a denominator of 6.

ClassCountRate (%)Lower (%)Upper (%)Basis
1. The schema is rejected before a token is generated not measured – – – Not measured. A rejected call produces no output row; the count is in your provider logs as 400 responses.
2. Your framework chose an enforcement method for you not measured – – – Not measured. Paste the JSON Schema you asked the provider to follow.
3. Generation stops mid-object and the guarantee stops with it 1 16.7 3.0 56.4 Estimated from an end-of-input signature test. Add the stop-reason column for a direct count. A truncated object usually clears with a larger output-token cap or a shorter output.
4. The object validates and the value is wrong not measured – – – Not measured. Class 4 requires your own pass/fail label per record, because a wrong value that validates looks exactly like a right one in the bytes.

Unclassified parse failures: 1 of 6 | 16.7% | 95% CI [3.0%, 56.4%]. This count is included in the unparseable headline but is not assigned to any of the four classes.

Why each class happens, across providers and frameworks, is worked through in the structured-output failure study.

    For the retry optimizer

    The detected-failure share and the per-attempt success rate require correctness labels. Without them, a wrong value that validates is invisible, and a default of d = 1.0 would tell you every failure is detectable by a retry.

    Export

    unparseable rate = (class 3 + unclassified parse failures) / NHow?

    How this is calculated

    The denominator. N is the count of non-blank records in your file plus any calls you entered as rejected at compile time. A rejected call (class 1) leaves no output row, so the page cannot see it in the file. Typing that count from your provider logs keeps the denominator honest. Without it, the rate is over delivered responses only, and the page states the basis.

    Per-record classification. Each record is classified into exactly one bucket. The page tries to parse the record as JSON first. If the stop-reason column is present and its value is one of the truncation codes (max_tokens, length, max_output_tokens, token_limit), that is the authoritative class-3 path. When the stop-reason column is absent, the page falls back to a structural test: if the record's bracket and quote ledger is unclosed, the record is class 3 (output truncated mid-object), and the count is labeled estimated. If the parse throws for any other reason, the record is an unclassified parse failure: something went wrong in the JSON, but the cause does not match a known class. If the parse succeeds and you supplied a JSON Schema, the page re-validates the parsed object against your schema per the structured-output research prescription for catching the under-constrained direction; a validation failure is class 2. If the record parses and passes (or no schema was given) and your correctness column marks it wrong, the record is class 4.

    Precedence. A record can trip more than one test, so the order is fixed and stated on the page: class 3, then unclassified parse failure, then class 2, then class 4, then pass. A record counted in one bucket is counted in no other. The buckets sum to the record count.

    The headline. The unparseable rate is class 3 plus unclassified parse failures, divided by N. This is the share of responses your code cannot use at all, whether because the output was cut short or because the JSON was malformed. Class 3 overlaps with the problem tool-output truncation research covers from the inbound direction: that page studies payload cut before the model reads it, while class 3 here is output cut after the model writes it.

    Intervals. Every rate carries a Wilson score interval at the confidence level you choose. The pass-rate calculator covers the method at length. Wilson stays inside [0, 1] at the near-zero rates this page commonly returns, where a normal approximation can cross below zero. A count of zero still gets its interval: the upper bound sets the ceiling on what the true rate could be given your sample, and that ceiling is the finding.

    The hand-off. When you supply a correctness column, the page computes two numbers the retry optimizer takes as inputs: per-attempt success rate (p) and detected-failure share (d). The detected share is the fraction of failures the page can attribute to a cause (class 1 from provider logs, class 2 from schema re-validation, class 3 from the stop-reason column or the signature test, and unclassified parse failures) out of all failures including class 4. Without correctness labels both numbers are suppressed, because a default of d = 1.0 would tell you every failure is detectable by a retry. Copy both numbers into the retry optimizer's form; that tool has no prefilled link from this page, so the hand-off is a link and two numbers you type.

    Formula: unparseable rate = (class 3 + unclassified parse failures) / N

    Questions

    Why does a zero rate still show an upper bound?

    The Wilson interval at count = 0 returns a non-zero upper bound that depends on N and the confidence level. That upper bound is the strongest claim this run supports at the chosen level: the true rate is below that ceiling. More responses bring the ceiling down.

    What is the difference between this page and the eval-dataset schema validator?

    The schema validator checks the rows you feed the model: field names, types, required columns in the eval set. This page checks what the model handed back: whether the output parses, whether it passes the schema you asked the provider to follow, and whether the value is correct. Different file, different direction.

    Why is class 4 always "not measured" when I leave the correctness column empty?

    A wrong value that validates looks exactly like a right one in the bytes. No parse test and no schema check can tell the difference, so the page needs your own judgment per record to count this class. The detected-failure share and the per-attempt success rate are also suppressed, because both depend on knowing the total failure count including class 4.

    How do I get the two numbers the retry optimizer needs?

    Add your correctness labels to the file. The page computes the per-attempt success rate (p) and the detected-failure share (d) and prints both at the bottom, labeled with the retry optimizer's own field names. Copy both numbers into its form.

    What does "estimated" on the class 3 row mean?

    Without the provider's stop reason, the page checks whether the JSON parse error has the signature of a string cut short: unclosed brackets, an unterminated string. The test catches most truncations, but a genuine syntax error in the model's output can produce the same signature. The estimated label and the named basis keep the count honest; adding the stop-reason column to your file makes it authoritative.

    Where do I find out why each failure class happens?

    The failure taxonomy maps each class to the provider and framework behaviors that produce it: compilation constraints, constrained-decoding gaps, max-token limits, and silent wrong values. This page counts the rate from your own outputs; that page covers the cause across providers.