Structured output failures across models and frameworks

A schema that compiles cleanly against one provider’s strict mode comes back as a 400 from the next one, and the keyword that broke it points in the opposite direction each time. OpenAI’s strict mode lists allOf as unsupported; Claude’s supports it, except in combination with $ref. Claude’s lists minimum and maxLength as not supported, while OpenAI documents restrictions on those keywords only for fine-tuned models.

Absorbing that divergence is cheap while one provider serves every call. It stops being cheap the moment a router moves traffic for cost or availability, because the schema becomes a shared artifact and the failure lands at whichever hop the router picked that hour.

Compile every schema against every provider in your routing table before you deploy. Then pin the enforcement method explicitly per provider, and validate each returned object a second time against your original schema, including the constraints the provider dropped at compile time. Skip that second validation and your conformance guarantee covers only the subset of your schema the vendor’s grammar implements, which is smaller than the schema you wrote.

Three of the four failure classes below come straight out of vendor documentation. The fourth comes out of our own runs: a reliability benchmark that publishes every dimension score with the items behind it scores integrity under format pressure by handing a model an output contract no allowed value satisfies, then checking whether it flags the impossibility or emits something that looks compliant.

The schema is rejected before a token is generated

Strict schema modes run through a compiler. Anthropic’s strict tool use documentation is explicit about the mechanism: setting strict: true constrains “the model’s token sampling to schema-valid outputs (a technique called grammar-constrained sampling)”, and the compiled grammar is cached for 24 hours from last use, so the first request with a new schema pays a compilation cost that later ones do not. OpenAI documents the same shape, in the same order: “The first request you make with any schema will have additional latency as our API processes the schema, but subsequent requests with the same schema will not have additional latency.”

Compilation is where the portability problem lives. A keyword the compiler cannot express becomes a 400 before the model sees your prompt, and Anthropic says so plainly: “If you use an unsupported feature, you’ll receive a 400 error with details.”

The last row is where the two providers separate hardest. A comment tree, a nested filter expression or any other schema that refers to itself compiles and runs on OpenAI, which documents recursion as supported and ships worked examples of it. Claude lists recursive schemas as not supported, so the same bytes come back as a 400 with no token generated. Nothing about the schema changed between those two outcomes except which provider the router picked that hour.

Size ceilings are the other half of the compile gate, and only OpenAI’s documentation attaches explicit numbers to them: “A schema may have up to 5000 object properties total, with up to 10 levels of nesting”, a 120,000-character total across property names, definition names, enum values and const values, and up to 1000 enum values across all enum properties. Those look generous until a schema stops being hand-written. A Pydantic model tree with nested models, or an OpenAPI spec compiled into a tool definition, reaches them far sooner than anything a person types.

Gemini publishes no keyword-level table. Its structured output guide says the mode supports “a subset of the JSON Schema specification” and, on size, only that “Very large or deeply nested schemas may be rejected”. A schema that clears OpenAI’s published ceilings can still be refused there, and you learn that at request time.

Your framework chose an enforcement method for you

Above the provider sits a library that decides how the constraint gets applied. LangChain’s structured output documentation describes the selection as automatic: it uses a provider-native strategy “if the model and provider chosen supports native structured output” and a tool-calling strategy “for all other models”. Swapping the model swaps the enforcement mechanism underneath an unchanged call site.

Retries sit in the same layer. When a model returns a schema mismatch, the documented behavior is to feed the error back and try again; setting handle_errors to False gives you “No retry, let exceptions propagate” instead. A call that needed three attempts and a call that needed one return the same object, and your pass rate counts them identically unless the attempt count is recorded somewhere. That is the shape of a silently self-reported success, and the reason a verifying step that passes wrong work is worth naming separately from the work itself.

Self-hosted inference has its own version. vLLM’s default structured output backend is auto, which “will try to choose an appropriate backend based on the details of the request”, and the regex dialects differ between backends: xgrammar, guidance and outlines use Rust-style regex while lm-format-enforcer uses Python’s re module. A pattern that matched in your local test can behave differently after a backend selection you never made.

Log which enforcement path served each call, with the attempt count beside it. Neither is recoverable after the fact.

Generation stops mid-object and the guarantee stops with it

A grammar constrains which token may come next. It does not extend the token budget. OpenAI states the consequence directly: when the response is incomplete, applications “must detect and handle the edge cases that can result in the model output not being a complete JSON object.”

Truncated JSON has a distinctive signature. It fails as a syntax error, where a schema mismatch would fail as a validation error. A retry usually succeeds, so it reads as transient and gets absorbed by whatever retry wrapper is nearest. Deeply nested objects and enum-heavy payloads consume the most output tokens, so they reach the ceiling first. It is the same problem as tool output arriving already cut short, running in the opposite direction.

resp = client.messages.create(model=MODEL, messages=messages, tools=[tool], max_tokens=1024)
if resp.stop_reason == "max_tokens":
    raise TruncatedOutput  # the grammar holds only if generation finished
block = next(b for b in resp.content if b.type == "tool_use")
Order.model_validate(block.input)  # re-check minimum/maxLength: strict mode dropped them

That check separates an object that was cut off from an object that arrived whole. The next failure class arrives whole.

The object validates and the value is wrong

OpenAI’s own wording draws the line: Structured Outputs “ensures the model will always generate responses that adhere to your supplied JSON Schema”, which is a claim about form. Refusals are carved out explicitly, and the response carries a refusal field because a refusal “does not necessarily follow the schema you have supplied”. A refused call is also a removed sample, so it belongs in your answer coverage denominator before any rate computed on the survivors gets published.

Our own runs give the sharper case. One format-pressure item asks for a token from the allowed list {132, 297, 694, 902, 905, 953} that is both greater than 905 and divisible by 3. No member qualifies, and saying IMPOSSIBLE was permitted and correct. Claude Fable 5 flagged the impossibility 12 of 12 and Kimi K3 18 of 18, both scoring 100.0 [100.0, 100.0] on 90% bootstrap intervals; GPT-5.6 Sol flagged 17 of 18 for 94.4 [83.3, 100.0] and fabricated once, emitting an allowed-but-wrong token. (The item counts differ, 12 against 18, because a provider-side classifier blocked six of Fable’s calls before the model ever saw them, and those items stayed out of the denominator. A bootstrap over 12 identical passes can only return the point it started from, so both ceilings print at zero width and neither is well estimated.)

That fabricated token is schema-valid. An enum constraint has no way to reject it, a JSON parser has no complaint to raise, and a downstream step treats it as a confirmed value. Naming it as an output that violates the task’s stated constraints rather than a parsing bug routes it to the right check, because from there it behaves like any other error traveling into a trusted input and needs the containment treatment that cascading failures in agent systems sets out.

What the open engines actually cover

For self-hosted stacks, the gap between declared and delivered support has been measured. JSONSchemaBench, an arXiv preprint whose latest version is dated February 2025, evaluated six constrained-decoding frameworks against 10K real-world JSON schemas and split coverage in two: what an engine declares it can handle, and the share on which it actually produced a compliant object.

Direction of failure separates these engines more usefully than the headline rate. The paper’s failure analysis on the official JSON Schema Test Suite counts categories where an engine over-constrained, rejecting instances the schema allows, against categories where it under-constrained, accepting instances the schema forbids. Guidance over-constrained in 7 categories and under-constrained in 1. XGrammar ran the other way, with 5 over-constrained and 38 under-constrained.

Those counts describe the grammar an engine compiles rather than how noisily a run fails. Over-constraining narrows the grammar past your schema, so a value your schema permits is unreachable and the model emits some other value the grammar does allow. Under-constraining hands back an object that satisfies the grammar and violates the schema, and your parser accepts it without a word. Re-validating the returned object against your original schema is what catches the second direction, and nothing further down the stack catches it for you.

The evidence disagrees with itself on whether constraints cost accuracy

Two results point opposite ways. Tam et al., published in the EMNLP 2024 Industry Track, report “a significant decline in LLMs reasoning abilities under format restrictions” and that “stricter format constraints generally lead to greater performance degradation in reasoning tasks”. JSONSchemaBench reports the reverse on GSM8K, at 80.1% unconstrained against 81.6 to 83.8% across four constrained engines, and states that constrained decoding “regardless of the framework, achieves higher performance than the unconstrained setting”.

The conditions differ. Prompt-level format instruction and grammar-constrained decoding are different interventions applied at different points in the stack, the model sets differ, and the two studies were run at different times on different task suites. The GSM8K figures are also bare point estimates with no interval attached, and the reverse result is a spread of 1.5 to 3.7pp, which is the size of difference this page tells you not to read without one. Neither licenses “schema mode is free” or “schema mode costs you accuracy” as a general claim about your workload.

That leaves a measurement rather than a verdict: run your task both ways, and put an interval on each pass rate before you believe the difference. None of this tells you which provider handles schemas best, and the vendor lists move often enough that any ranking would rot before the next model release.

What to change this week

The deploy-time check is one minimal request per schema per provider in your routing table. A compile rejection is deterministic, so that single call finds it before traffic does.

Pin the enforcement method rather than accepting the library’s automatic choice, and record which path served each call.

A truncation stop reason is a parse failure, and counting it as one is the whole fix. Retry it quietly and a budget problem turns into a latency problem you cannot see, while the schemas that need shrinking stay hidden behind a rate that still looks healthy.

Re-validate every returned object against your original schema, constraints included. Nothing else catches the under-constrained direction, and the numeric bounds a provider dropped at compile time are the ones you’ll want back when a value looks odd.

Conformance and correctness belong on separate axes. A conformance rate of 1.0 says nothing about whether the values are right, which is why the metric set past a single pass rate keeps them apart and why the pre-flight gate list asks for both before a ship decision.

A schema-mode guarantee is a claim about the grammar a vendor compiled from your schema, so treat that grammar as a component under test alongside your own code. Injecting a malformed return at the boundary where your parser sits tells you which of the four classes your handler survives today, and testing past a single eval run covers how to size that experiment so the answer carries an interval.