Specification violation (agent tasks)

Specification violation is the agent failure in which returned work breaks a constraint the task stated in writing: an output schema, a word ceiling, a tool the brief ruled out, a scope line the instruction drew. The content underneath can be entirely correct and the run still scores as a violation, because the judgement is taken against the instruction rather than against the truth of the answer.

MAST, the empirical taxonomy behind our account of how multi-agent systems fail and how those failures spread, annotates this as FM-1.1, disobey task specification, and defines it as “Failure to adhere to the specified constraints or requirements of a given task, leading to suboptimal or incorrect outcomes” (Cemri et al., arXiv:2503.13657, NeurIPS 2025 spotlight poster; arXiv v3, October 2025). The paper separates two annotated modes here: FM-1.1 for a breach of the stated task, and FM-1.2 for a breach of the assigned role. This page scores the task side, and the role side is slower and carries an entry of its own. Figure 1 of that version puts disobey task specification at 11.8% of failures observed across 1,642 execution traces, published as a point estimate with no interval attached, so read it as a prevalence signal rather than a rate your own system should expect to reproduce.

A violation is only visible against a constraint someone wrote down.

That makes it a boundary event rather than an end-of-run verdict. The violating output joins error propagation the moment a downstream agent accepts it as valid input, and from there the ordinary questions apply: how far it reached, and which node produced it, which is the work of failure attribution. Score specifications where an output crosses a boundary. That is the last point at which a violation is still one agent’s problem.

Scoring it means turning a brief written for a reader into something a checker can decide.

Scoring a specification violation rate clause by clause

A specification violation rate is only as meaningful as the constraint list written down before the run. Decompose the task brief into separately decidable clauses, one per requirement, so that a single output can fail three of them and pass the rest. Score every returned output against every clause, then report two numbers: the per-clause violation rate, which tells you which requirement the model keeps dropping, and the per-output rate, the share of outputs breaking at least one clause. Both are proportions over the outputs you actually scored, so both need an interval instead of a lone percentage, on the same reasoning behind any eval confidence interval; the Wilson and Clopper-Pearson interval calculator turns k of n into a bound. Keep the denominator honest by counting refusals, timeouts and truncated returns separately (an output that never arrived can’t violate anything, and quietly dropping it inflates compliance).

The cheapest clauses to score are the ones a program decides without a judge. IFEval formalized that construction for single-turn instruction following, assembling around 500 prompts from 25 types of verifiable instruction, such as a minimum word count or a required keyword, each of them checkable in code (Zhou et al., arXiv:2311.07911). Agent briefs carry plenty of clauses that resist that treatment, and those are where a judge enters the loop with its own error rate. Instrument the check at the validator or supervisor step where an output crosses to the next agent, and version the constraint list alongside the eval, because a rate scored against list v2 won’t compare to one scored against v1. A JSON payload rejected by schema validation is the most legible case of all, and the per-provider, per-framework picture of when structured output breaks belongs to structured output failures across models and frameworks.

A violation caught at the next gate is a contained violation. What happens when the gate looks straight at one and waves it through is a separate failure with its own name.

Specification violation vs task verification failure

Task verification failure is a failure of the checking step: a validator, a critic agent or a test suite examines work that is wrong and marks it acceptable. Specification violation is a property of the output itself, present whether or not anything ever looked at it. An agent can break a stated constraint and have the breach caught at the very next gate, which is a working system behaving as designed, and the violation still belongs in your rate because that rate measures how often the generator drifts off brief. A verifier can also pass an output that respects every stated clause and is factually wrong, which is a verification failure with clean specification compliance underneath it. Neither rate stands in for its neighbor. Score them apart from each other: the violation rate tells you how often the brief fails to bind, and a verifier that passes wrong work tells you how often the safety net has a hole in the same place.

Both of those are events you can point at in a single turn. The role-side failure rarely gives you a turn to point at.

Specification violation vs role drift

Role drift is the gradual abandonment of an assigned remit across a long run: a worker told to summarize starts recommending, a researcher starts editing, and every individual turn still reads as reasonable against the turn before it. Specification violation is a single scoreable event, decidable on one output against one written clause. Drift is a slope, and it surfaces only when a late output is compared against the original brief rather than against the preceding message. That difference changes which instrument you need pointed at the run. Per-output checks at the boundary catch violations, while catching drift means re-scoring against the opening specification at intervals through the run. Keep the two checks on separate schedules, and expect an agent sliding out of its assigned role to produce runs whose per-output violation rate stayed flat while the work walked steadily away from what you asked for.

One more source of confusion sits outside this field entirely, and it is a good deal older.

Specification violation in formal methods

Formal methods used the phrase first and used it precisely. A specification there is a formal property over a modeled system, usually written in a temporal logic, and a model checker either proves the property holds across the reachable state space or returns a counterexample: a concrete execution trace that violates it. The violation is decided, and it arrives with a witness you can replay. Agent task evaluation operates with neither the decision procedure nor the closed state space that makes that possible. A task brief is natural language, incomplete at the edges, and ambiguous in precisely the places a model finds, while the checker is frequently another language model carrying its own false-positive rate. Borrowing the word wholesale would import a guarantee this setting cannot deliver. A narrower version of the borrowing does survive: treat the constraint list as the specification, keep every clause independently decidable, retain a replayable trace of the run that broke it, and state the violation rate with its interval and the version of the list it was scored against. That gives you a counterexample and an accounting, without claiming the coverage a model checker earns.

Two instruments decide this between them: the constraint list settles what counts as a violation, and the verification step settles whether anything catches one. Which topologies let a violating output travel furthest before anything stops it is argued in the orchestration patterns and the failures they amplify. For a term-by-term route through the rest of this family, start at the multi-agent failure vocabulary.