Task verification failure (agents)

Task verification failure happens when the step responsible for checking an agent’s work returns a pass on output that does not satisfy the task. A check ran, it produced a verdict, and the verdict was wrong in the expensive direction: the work travels downstream carrying a confirmation, and every agent after it treats the result as already checked. The MAST failure taxonomy files this as incorrect verification, FM-3.3, and holds it apart from the case where no verification was attempted at all.

The quantity is conditional, which changes how it has to be reported. It counts wrong results among verifier passes, so the denominator is the passes rather than the runs, and it moves with how much wrong work the verifier was handed in the first place. The same verifier posts a flattering figure on an easy task suite and an ugly one on a hard suite without changing its behavior at all. Report the pass count it was computed over, an interval around the rate, and the underlying error rate of the work being checked; drop the third and the number describes your task set.

Verification failure is the mechanism sitting under a great deal of what gets logged as a run that announced success and was wrong. Our own cross-vendor reliability run put a figure on the checking instinct a verifier inherits from its base model. Asked to do routine work over a document carrying one corrupted, load-bearing value, two of three frontier models flagged the planted number on 0 of 36 items each, and the third caught it on 5 of the 29 items it saw, 17.2 [6.9, 27.6]. Those brackets are 90% bootstrap percentile intervals, so on an all-zero sample they collapse to 0.0 [0.0, 0.0]: that is what resampling 36 identical misses returns, and it is not a reading that the true catch rate is pinned at zero. Forward from a bad pass, the error becomes another agent’s trusted input; backward from the symptom, reconstructing the run to isolate its earliest corrupted node is how you land on the verifier as the culprit. So treat a verifier as an instrument with its own error rate, and measure it before you spend anything on its verdicts.

How to calculate a task verification failure rate

Take every case where the verifier returned a pass, score those same cases against ground truth, and divide the passes that were wrong by the total passes. The result is the verifier’s false-pass rate, and because it is a proportion it needs a confidence interval rather than a bare percentage; the Wilson and Clopper-Pearson interval calculator handles the arithmetic. One rate on its own still leaves the instrument half-described. Push known-bad outputs past the verifier to get its sensitivity, and known-good outputs to get its specificity. That pair characterizes the verifier independently of the task mix it happened to see. The judge bias correction calculator takes exactly those two inputs, adjusts a raw pass rate, and carries both sources of uncertainty into the interval it reports.

The flagship profiler is designed to seed known-bad outputs past a verification step and report the share it waves through with a bootstrap confidence interval. No such instrument has shipped, so the rate for your verifier comes off your own runs, and the two calculators above carry it the whole way: counts in, corrected rate and interval out. Instrument where the verdict is written rather than where the run ends, since a run can finish clean for reasons unrelated to the check.

Task verification failure vs no verification at all

No or incomplete verification, FM-3.2 in the same taxonomy, is the case where the checking step is absent, skipped, or shallow enough to miss whole classes of defect. Nothing looked, or something looked at the wrong thing.

Task verification failure is the case where something did look and said yes.

A missing check and a wrong check differ. A system with no verification can still ship correct work, because nothing about a missing check makes the underlying output wrong, and its defect is invisibility rather than error. A system with thorough verification coverage can land in a worse place than one with none, because a confident wrong verdict licenses the next agent to skip the caution it would have applied to an unchecked input. The third mode in that MAST category, stopping short and declaring the job finished, compounds with both. Coverage and the false-pass rate answer different questions: coverage tells you what fraction of the work gets checked, and the false-pass rate tells you what a check is worth.

Task verification failure vs judge bias

Judge bias is a systematic direction in an evaluator’s verdicts, the tendency to favor the response shown first, the longer answer, or output from its own model family, independent of whether that output is correct. Task verification failure is the outcome measure: the share of passes that were wrong, whatever produced them. An unbiased judge can post a high false-pass rate purely from weak discrimination, and a biased judge can post a low one when its bias happens to run along the grain of the distribution it is scoring.

Measured on its own, the discrimination part goes badly. Across a corpus of 9,876 tau2-bench trajectories, no judge configuration among five judge models and five prompt strategies exceeded 0.65 AUROC at telling a false success from a real one, and on AppWorld the best configuration reached 0.537 against a chance line of 0.5. That is a 2026 workshop paper rather than a replicated result, and the figures belong to those two benchmarks rather than to judges in general. Bias and discrimination are separate repairs, so pair the false-pass rate with a bias audit: the mechanisms are laid out in our account of how judges go wrong, and the question of whether a judge’s confidence tracks its accuracy belongs to judge calibration.

Verification failure reads the opposite way in software testing

Software and systems engineering has used verification for decades to mean checking a build against its specification, held apart from validation, which checks it against the need it was supposed to serve. In a test log, a verification failure is an assertion catching a mismatch, which is the check working. The agent sense inverts the polarity: the check ran, raised nothing, and was wrong. Each sense is correct in its own field, neither is going anywhere, and a page that quietly picks one leaves half its readers misreading the number.

One premise travels across with the testing sense and should be left behind: that verification can be summarized as a single pass count, the way a test suite reports a tally of green. A test suite’s oracle is exact and nearly free, since the assertion already knows the expected value. An agent verifier’s oracle is a model reading prose: it isn’t exact about anything, and it costs a model call every time it runs, so a count of passes says little until you know how the counter behaves. What replaces the tally is a report on the verifier as a measuring instrument: false-pass rate with an interval, sensitivity and specificity on seeded cases, and the pass count all three were computed over. That discipline is also what keeps output violating the task’s stated constraints from being scored as a pass, and it is what schema and structured-output checks already apply to the narrow slice a parser can see.

A pass that should have been a fail is a boundary that did not hold, and containment rate is the quantity that reads it that way: a verifier is one of the hops where a fault either stops or carries on, and its false-pass rate is that hop’s leak. Where verification sits among the fourteen modes MAST identified, and which containment lever bounds each of them, is worked through in the multi-agent failure taxonomy, with the term-by-term route across the rest of that vocabulary in the failure vocabulary index. An unmeasured verifier is an instrument whose readings nobody has ever checked.