Silent failures: when agents report success and are wrong

An agent ends its run, writes status=success into its final message, and your harness scores a pass. Whether the work behind that pass exists is a separate question, and in 2026 it acquired measurements. On AppWorld, a benchmark that grades coding agents against the database state of its simulated apps and ignores what the agent says about its own work, false success accounted for 75.8% of failures among architectures that produce explicit completion claims. The agent recorded success, the programmatic evaluation recorded failure, and nothing in the transcript said so.

We have measured the answer-level version of this ourselves, on runs where there is no status field to inspect because the answer is the completion claim. Our three-model reliability study handed each model a document carrying one corrupted, load-bearing value and asked for routine work over it. Two of the three adopted the planted figure on every one of their 36 items, a corrupted-data catch rate of 0.0 [0.0, 0.0] on 90% bootstrap intervals, and what came back was usually the poisoned total with tax applied and nothing else. The third caught it 5 times in the 29 items that reached it, 17.2 [6.9, 27.6]; a classifier had blocked seven of its calls before the model saw them. Those five catches all landed on one of the two document variants, and on the other every model failed every item.

The work came back formatted and confident, and nothing else in the run was ever going to contradict it.

Treat an agent’s completion claim as a hypothesis, and score it against environment state or a programmatic check rather than against the transcript. Four readers take their evidence from the run’s own account or from code the run never exercised: the status field, the final answer, an LLM judge reading the trace, and a green test suite. Each has now been measured, each came out weaker than the pass rates built on top of it assume, and the state check is the one reader on this page that held.

That claim rests on four separate papers, and the first job is pinning down what each one means, because they share the two words and little else. Their evidence weights differ too: the green-test leg rests on a single self-published production report, and the table below says so in its own column.

Four papers, four meanings for the same two words

A silent failure is a run that fails while every signal the system emits reads as success: no exception surfaces, no status flips, no monitor fires, and the failure is discovered later from its consequences or never discovered at all. False success is the sharpest subtype, the case where the agent explicitly asserts completion and the environment state contradicts it. Loud failures are the easy case, since a malformed tool response or a broken schema at least leaves a parse error behind. A silent failure leaves a well-formed artifact.

Four papers published in 2026 measure something under this name. They measure four different things, on four different denominators.

TABLEShow full table (4 rows)Showing full table (4 rows)

Paper	What “silent failure” means there	Denominator behind its numbers	Evidence weight
From Confident Closing to Silent Failure (Advani, 2026)	The agent asserts completion and the environment state disagrees	Failures within the trajectories that emit an explicit status field	Workshop paper (FAGEN at ICML 2026); 11,755 trajectories across two benchmarks; labels independent of the agent’s own text. The strongest evidence in the cluster
Silent Failures in Multimodal Agentic Search (Wu, Gao, and Yang, 2026)	The answer looks right while the search trajectory under it is defective	Committed trajectories only, on one benchmark: 200 tasks by 4 models	Workshop paper (SIGIR 2026 SynthIR); labels come from LLM judges that partly disagree with each other
REFLECT (Lin et al., 2026)	A trace completes normally, the answer is wrong, and no step signals an error	Individual trace steps: which one broke	Workshop paper (FAGEN at ICML 2026); an attribution method, so it measures no prevalence at all
When Errors Become Narratives (Wei Wu, 2026)	An error signal exists somewhere yet never reaches a human in actionable form	Production incidents on one personal runtime over eight weeks	Self-published draft (v0.3) by the system’s own operator; no venue, no review, no independent annotation

A sentence of the shape “75.8% of agent runs fail silently” merges at least two rows of that table. The 75.8% is a share of failures, computed only over trajectories that carried an explicit status claim. The production report’s shares are shares of incidents on one system. The multimodal gaps are percentage points of committed trajectories. The four papers do share a target: each one measures a layer that is supposed to notice failure, and each one finds it weaker than its users assume.

Start with the paper that measures prevalence, because it is the strongest of the four and the source of the number this page opened on.

Three quarters of AppWorld’s status-reporting failures closed with a success claim

Across 9,876 tau2-bench trajectories from 8 model families, including Claude Opus 4.5, GPT-5.2, and Gemini 3 Pro, false success accounted for 45% of failures in the airline domain and 48% in retail. In telecom it accounted for 3%, and the study attributes that gap to how the domain is built: telecom is dual-control, so the user simulator can verify task state on its own. The AppWorld arm covers 1,879 trajectories from 4 model families, and there false success reached 75.8% of failures among the self-assessing coding-agent trajectories that emit an explicit status field: three quarters of those failures ended with the agent writing success over work the evaluation scored as failed.

The study can measure this because both of its benchmarks score outcomes without reading the agent’s prose. tau2-bench, a conversational tool-use suite with airline, retail, and telecom domains, checks task state programmatically. AppWorld grades coding agents against the database state of nine simulated apps, so a trajectory’s label shares no vocabulary with its closing message. An agent can’t talk its way to a pass.

Inside tau2-bench the per-model share runs from 13% for GPT-5.2 up to 79% for Alibaba’s Qwen3-Max-Thinking-Preview, a 66pp spread across the eight model families on one benchmark; the author describes the traces at the top of that range as rationalizing completion instead of verifying it. The behavioral signature is specific enough to build detectors on: false successes read state without modifying it. App-level rates line up with that mechanism. Read-heavy Spotify tasks show a 94.5% false-success share of failures while Amazon tasks, which require concrete write operations, sit lowest at 65.2%; where the work has to leave a mark, failure is harder to hide. Stopping early and declaring the job finished is one common route in, and from there the wrong result travels on as another component’s trusted input.

The caveats travel with the number. The AppWorld half runs on 2024-era models (GPT-4o, GPT-4-Turbo, LLaMA-3, DeepSeekCoder) over AppWorld’s publicly released outputs, and on 2 of the benchmark’s 4 agent architectures, because the other two write success unconditionally and make the field uninformative. The author states plainly that the 75.8% applies to a subset of trajectories rather than the full benchmark. The quantity is a share of failures, so it rises mechanically as an agent gets better at avoiding loud failures without getting better at the task. And it is a single-author workshop paper; no independent replication exists yet.

Prevalence is half the study. The other half asks whether a second model reading the trajectory can catch the lie, which is the cheapest check to reach for because it needs neither ground truth nor access to the environment.

The judge you’d hire to catch it barely beats chance

Across 5 judge models and 5 prompt strategies, including a baseline handed the full ground-truth task specification, no configuration exceeded AUROC 0.65 at separating false successes from real ones on tau2-bench, and the same judges managed 0.54 on AppWorld API-call traces. Chance is 0.5. The judges were GPT-4o, Claude Sonnet 4.5, Llama 3.3-70B-Instruct, DeepSeek-R1, and o3-mini, and the reasoning judges did no better: DeepSeek-R1 peaked at 0.573, o3-mini at 0.554.

The diagnosis is more useful than the score. Judges reward surface proxies: confident closing language on tau2-bench, sheer action-sequence volume on AppWorld, verified state changes on neither. A judge reading a transcript is grading the same artifact the failing agent produced, and that artifact is optimized to look finished. This defect is separate from the familiar judge biases of order, length, and self-preference, and separate from whether a judge’s confidence tracks its accuracy; both still apply on top of it.

The multimodal study corroborates this from a different angle. It ran cross-judge validation on its own silent-failure taxonomy and found that judges from different model families agree almost perfectly on whether a final answer is correct, at κ = 0.817, while their cross-family agreement on the silent-failure categories themselves runs from κ = 0.031 on modality shortcuts to κ = 0.467 on cross-modal contradiction. The instruments disagree about what they are measuring.

The counter-result is a TF-IDF classifier trained on labeled trajectories and evaluated on disjoint tasks: it reaches AUROC 0.83 on tau2-bench and 0.95 on AppWorld, recovers 4 to 8 times more false successes than the best judge at the same flag rate, and runs at 3,300 times lower latency. The features are boring on purpose: closing-language patterns on one benchmark, read-versus-write API shape on the other. The detectors do not travel between domains. Trained on airline and applied to retail, they fall to AUROC 0.66 to 0.69 depending on the classifier, and getting back to 0.79 to 0.83 costs 50 to 100 labeled examples per class in the new domain. So the paper’s recommendation follows the cost as much as the score: run the cheap detector over every trajectory, and stop asking an LLM judge to make this particular call.

That is one study, two benchmarks and one instrument family. A second group measured the same structural claim in a different domain, with different models and a different method, and found it again.

A second domain measures the same gap, and finds it small

Silent Failures in Multimodal Agentic Search, a SIGIR 2026 workshop paper, defines a true correctness rate: a trajectory counts only if the final answer is right and no silent-failure flag fires anywhere along the way. On 200 MMSearch-Plus tasks per model, committed-run accuracy overstates that rate for all four models tested: by 1.4pp for Claude Sonnet 4.6, 3.0pp for Gemini 2.5 Pro, 7.3pp for Gemini 3.1 Pro Preview, and 0.6pp for GPT-4o. Claude’s pair shows the shape of it: committed accuracy 36.0% [28.4, 44.4] against a true correctness rate of 34.6% [27.1, 42.9], on 95% Wilson intervals. Every one of those pairs overlaps. The direction is consistent and the sizes are small, so what this paper documents is an overstatement you can measure rather than a collapse.

The denominators deserve more attention than the gaps. Headline metrics are computed on committed trajectories only, and the committed share runs from 62% to 88% depending on the model. Gemini 3.1 posts the best committed accuracy while abandoning 38% of its runs to tool-call exhaustion, so its headline number is computed over the runs that reached a commit. That is coverage conditioning operating inside a single results table, and it is the same mechanism that lets an answered-only pass rate flatter a system that declines the hard requests. Capability, on this evidence, moves failure around rather than removing it: the strongest model here fails by running out of budget where the others fail by committing wrong answers.

Benchmarks watch agents for a few hundred trajectories. One 2026 document watches a single production system for eight weeks, and it carries both the weakest evidence on this page and the best vocabulary.

In production, the failure writes its own cover story

When Errors Become Narratives is a field report from one personal-assistant runtime: roughly 40 scheduled jobs, 8 LLM providers, one operator pair, eight weeks, 22 incidents written up as postmortems. It is a self-published draft, version 0.3 by its own title page, with no venue, no review, and no independent annotation, and its shares and latencies are descriptive statistics of one system. None of its numbers belongs in the same sentence as a benchmark result. It earns its place here for two observations and a term.

The runtime’s 4,286 unit tests and 827 governance checks stayed green through most of the 22 incidents, and a retrospective audit of 15 incidents found the governance layer had prevented none of them in advance while blocking 87% from recurring; audits, in the author’s phrasing, are regression engines rather than prediction engines. Most of the silent failures were caught by a human reading the system’s output, which the author is careful to offer as an existence proof from one attentively watched system, with no claim that it is a rate anyone should expect. The gap between watching a system and measuring its output is argued in full elsewhere on this site; this report shows what the gap looks like from the inside.

The term is fail-plausible: a failure whose error signal the LLM transforms into fluent, plausible narrative delivered to the user. Gray failure, the distributed-systems notion the author builds on, describes an observer that is blind. Fail-plausible describes an observer that is being convincingly lied to by the failure itself. One postmortem mechanism makes it concrete: a summary function counted only results with status “fail”, so invariants whose checks raised exceptions vanished from the count, and the governance audit printed “all invariants hold” over a pile of dead checks. The green dashboard was the failure artifact.

Everything to this point is detection: noticing that a run lied. The moment a detector fires, the question becomes which step broke, and that is a different problem with its own method paper.

After detection comes attribution: which step broke

REFLECT treats the silent regime as a localization problem: given a completed trace with a wrong answer and no step-level error signal, find the earliest decisive broken step. Its mechanism is interventional. Diagnose a candidate step, replay the trace with a patch targeted at that specific diagnosis, and treat a flipped outcome as contrastive evidence that the diagnosis was right. On four localization benchmarks it posts the highest exact-match accuracy among methods using the same auditor model, and that hedge has to travel with the claim, because a stronger auditor beats it outright on one of the four. Two limitations bound its production use: the strongest results assume oracle access to the expected answer, and replay requires re-executing the agent. You can’t re-run the trace that sent the money.

The general discipline of reconstructing a run to isolate its earliest corrupted node has its own glossary entry, and for coding agents specifically, our explainer on where CLI-agent failures originate in the trajectory covers the study that traces breakage back through the steps that compounded it.

Four papers, one set of instructions

The four papers disagree on definitions and agree on instructions. Two of the four below assume the outcome can be checked by a program. If your agent produces open-ended prose that only a person can grade, you can still hand-label runs and measure a judge against those labels, but the automatic state check in the first item is not available to you.

• Score completion against state. An agent’s status field is a claim by an interested party, and the 75.8% above exists only because someone checked those claims against a database instead of reading them. Where a programmatic check exists, count only what it confirms. Treat a checking step that passes wrong work as a failure mode in its own right, with its own measurable false-pass rate.
• Measure any judge before trusting it as a false-success monitor. Every configuration tested sat near chance, so assume yours does too until your own labeled runs say otherwise; measured sensitivity and specificity feed a corrected pass rate directly.
• Label 50 to 100 examples per class and train something cheap, since the classifiers that beat every judge here were TF-IDF.
• Publish the denominator next to every rate. Filtered trajectories, committed-only subsets, and answered-only calls each condition a number on the behavior under study, and an interval on the rate costs two minutes to add.

The second item is the one nobody can run on your behalf, because the share a checking layer waves through depends on your layer and your tasks. The flagship profiler is designed to make that share a standing measurement: push outcomes past a check that should have stopped them, and report how many got through with a bootstrap confidence interval. That instrument is still unshipped, so no false-pass rate of ours stands behind this section. What we have measured on this failure is the model’s own catch rate rather than a checking layer’s, and it is published with its counts and its intervals in the three-model reliability study. What a state-checked pass rate should become once you have it, pass^k across repeated runs with an interval on each figure, is laid out in how to measure agent reliability past a single pass rate, and the testing method that produces those numbers lives in the reliability testing pillar.