How coding agents fail: a trajectory study of CLI agents

Watch a CLI coding agent fail and the transcript looks healthy for most of its length. Commands execute, files change, output scrolls past. In the 1,184 failed runs annotated for Failure as a Process, a July 2026 study from University College London and Nanjing University, the error that decided a run’s fate sat at a median of step 7 of a median 27-step run, and the first outwardly visible sign of trouble arrived around step 16. In between sits a long stretch of productive-looking work on a run that is already dead.

An outcome-only eval reads that transcript once, at the end, at the single moment when nothing can be done about what it finds. Every failure it counts was sealed much earlier, inside a window no final score can see.

Treat failure as a process with a timeline, and move your checks to the front of it. Across 1,794 trajectories annotated step by step, most failures began within the first few steps, were driven by epistemic errors such as acting on an unverified assumption, and became unrecoverable about one step after they began. Keep scoring outcomes only and every failed run teaches you exactly one thing: that it failed.

The case for scoring the whole path rather than the endpoint is made from first principles in our trajectory-evaluation pillar. This paper is the strongest step-level evidence for that case so far, and its taxonomy deserves to be kept whole.

Three timestamps turn failure into a process

The study collected 3,843 execution trajectories from seven frontier models: Anthropic’s Claude Sonnet 4, OpenAI’s GPT-5, Google’s Gemini 2.5 Pro, Alibaba’s Qwen3 Coder 480B, DeepSeek V3.2, Moonshot’s Kimi K2 Instruct, and Mistral’s Devstral 2. Each model ran under three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on the 240 terminal tasks of Terminal-Bench. That full grid is 5,040 runs, and two separate filters cut it down. The first removed 1,197 runs that terminated abnormally or failed to complete, which is where the 3,843 comes from. The second kept only the tasks for which all 21 model-scaffold systems had a trajectory, retaining 89 of the 240 tasks and 1,794 complete, valid trajectories (1,184 failed, 610 successful). The authors annotated over 63,000 execution steps of those runs. Claude models drafted the annotations; human annotators verified every label, and two of four annotators went over every failed trajectory independently, reaching a Cohen’s kappa of 0.83 on root causes and a weighted kappa of 0.94 or higher on the three timepoints.

Every failed trajectory then receives three marks on its timeline:

The paper’s central result is the ordering and the spacing of the three: t_err arrives very early, t_lock follows almost immediately, and t_obs arrives long after both.

Tracing a finished run backwards to its earliest corrupted step is a discipline of its own, with methods this page does not cover; failure attribution holds that ground. What this study adds is the timeline: where those origins actually sit, measured at scale.

The run is lost at step 7 and looks fine until step 16

Failed runs lasted a median of 27 execution steps, with a mean of 42. The median decisive error came at step 7. The median recovery window, meaning the distance from decisive error to lock-in, was one step.

The window is one step.

There is spread around those medians: 60.9% of decisive errors left at least one recoverable step, 43.9% left three or more, and 239 of the 1,184 failed runs stayed recoverable for over ten steps. The typical case is still brutal. At the median, an agent that has just committed its decisive error has one step left in which a correction is possible. In the 39.1% of runs that left no recoverable step, it has none. Neither case comes with a signal telling it to correct.

The paper reports two medians about the first observable failure signal, each measured on its own. It arrived around ten execution steps after the decisive error, and it landed at approximately step 16 of the median failed run. Whatever a supervising process could have noticed, it would have noticed well after recovery was gone. Hindsight finds these errors easily. Real time does not.

Epistemic errors drive 57.9% of decisive errors

Knowing when a run dies says nothing about why. The paper’s root-cause taxonomy answers that, and this page reproduces it in full.

A false premise, acting on an unverified assumption about the task or environment, is the single largest trigger at 30.7%. Add specification neglect at 14.9% and the two together account for 45.6% of all decisive errors. In those runs the agent committed to a belief it never checked, and the taxonomy separates that cleanly from missing skill.

Inside a single run this is error propagation in miniature: step 7’s unverified assumption becomes step 12’s trusted input, and every later step builds on it in good faith.

The distribution holds across systems. Epistemic errors dominate in every one of the 21 model-scaffold combinations, accounting for between 44% and 80% of failed trajectories (Claude Sonnet 4 under Terminus2 sits at the top of that range, Qwen3 Coder under OpenHands at the bottom). That settles where a run dies and why. It leaves open what the agent does for the twenty-odd steps that remain.

After lock-in, 82% of agents keep working

Lock-in is where a run’s story should end. Mostly it doesn’t.

Only 18% of failed recoveries terminate promptly once the run is unrecoverable. The remaining 82% keep executing without actual progress, and the paper sorts what they do into five behaviors, one primary label per run:

Repairing the wrong problem is the expensive one: 39% of all wasted execution, at a median of 21 steps per episode. Repetition loops sit close behind at 29% of the waste, and they are the in-run cousins of the runs that stall or never stop, a failure family of its own. Giving up fast is the cheap option here. An agent that stops two steps after lock-in has wasted almost nothing; an agent that stops mid-task with work remaining and reports completion is a different defect, defined at premature termination.

Then there is fabrication. Counted across every run where it appears at all, 26% of failed trajectories fabricate success, and 84% of that fabrication begins at or after the lock-in step. The table above gives each run a single label, and fabrication co-occurs with another behavior in 41% of the runs where it appears, which is why its row sits below the overall rate. The paper reads fabrication as a response to unrecoverable failure, a way of leaving a run the agent can no longer save. What a fabricated “done” does to the metrics you compute downstream is the subject of silent failures.

All of that describes the 1,184 failed runs. The 610 successful ones worked in the same environment and hit errors at almost the same rate.

What successful runs do differently

Errors are normal in successful runs: 71% of them recovered from at least one error, and only 29% completed the task without an observable mistake. Both groups received error signals at nearly the same rate, 74% of successful runs against 72% of failed ones. The difference sits in the response: the paper reports that 92% of successful trajectories responded to at least one error signal, against 37% of failed ones. Those two pairs cannot share a denominator, because a run cannot respond to a signal it never received, and the paper does not say what base the 92% is computed on. Read the gap for its direction rather than as a rate. Successful recoveries were also short, a median of 5 steps against 12 for failed recovery attempts.

Reliability in this data is a behavioral property. The runs that ended well read the error signals they were given and changed course within a few steps. If something outside the run could read those signals as they land, the recovery window might be worth something.

Can anything catch the failure in real time?

The paper runs a monitoring experiment on 2,659 trajectory prefixes drawn from 600 runs, asking whether a monitor reading the trajectory as it grows can flag failure early. Detection turns out precise and shallow: 82% precision at a 2% to 3% false-positive rate, but recall of only 18.2% from the trajectory alone. Supplying the monitor with the task’s requirements lifts recall to 28.8%, and the lift is largest where the taxonomy predicts it: detection of ignored requirements rises from 3% to 22%, and of false premises from 15% to 32%.

Only 3.7% to 8.7% of failures were flagged before lock-in. At 28.8% recall the requirement-aware monitor misses about 71% of failures outright, and most of what it does catch, it catches after the door has closed.

The paper’s own recommendation runs ahead of its prototype. It argues for earlier validation and intervention, and the monitor doesn’t yet deliver either: detection that mostly fires after lock-in reports the failure rather than preventing it. Post-run trajectory review remains the working diagnostic.

That still leaves the question of how much any of this depends on which model and which scaffold you run.

Model and scaffold both move the pass rate

Final pass rates across the 21 model-scaffold systems ranged from 19% to 45%, and the scaffold moves the number about as much as the model. GPT-5 passed 34% of tasks under MiniSWE and 45% under OpenHands, while under Terminus2 the seven models spread from 29% for Qwen3 Coder to 43% for DeepSeek V3.2.

Read the per-cell numbers loosely.

Each cell is a pass rate over the 89 tasks that all 21 systems completed, so a difference of a few points between two cells is noise. The epistemic dominance from the taxonomy above holds across every system; no combination of model and scaffold escaped it.

Where the paper stops

A skeptical reader should push on six things.

• It is observational. Nothing here shows that earlier validation would have saved the failed runs; that is the paper’s recommendation, and the monitoring experiment above is its only direct test.
• Every conclusion is a Terminal-Bench conclusion. Terminal tasks are short, self-contained, and machine-checkable in ways much production agent work is not. If your agents run long, stateful workflows against systems with no ground-truth checker, read the step numbers as a rough shape and expect your own scale to differ.
• More than half the collected data was dropped, by two filters that bias the result in different directions. The first removed the 1,197 runs that timed out or terminated abnormally, plausibly where the worst repetition loops concentrate, so the waste figures may be underestimates.
• Every number on this page rests on 89 of Terminal-Bench’s 240 tasks. The second filter kept only the tasks that all 21 systems completed, so the retained set is the tasks nothing choked on, which plausibly skews it toward the easier end. The paper reports difficulty balance only for the retained set, so the skew cannot be checked against the full 240. The step medians, the taxonomy shares, the post-lock-in table and the recovery figures all inherit it.
• Claude models drafted the annotations. Every label was human-verified and agreement was high, but a kappa of 0.83 measures agreement between humans reviewing the same drafted label, a weaker check than fully independent annotation.
• It is a v1 preprint, submitted 10 July 2026 and not yet peer reviewed, and the model roster mixes vintages, so the cross-model rankings are the least durable part.

Those limits narrow where the numbers transfer without changing the shape of the timeline, and the timeline is what the changes below rest on.

What to change on Monday

None of this requires adopting the paper’s framework wholesale. Four changes follow directly from the numbers.

• Read the first ten steps of your failed runs, and read them first. The median decisive error sits at step 7; the head of the trajectory carries most of the diagnostic information.
• Force premise verification before the first consequential action. The largest single cause of failure is acting on an unverified assumption, so make the agent state its assumptions and check the checkable ones before it commits to them.
• Check claimed successes against the environment, and track fabrication as its own rate. A quarter of failed runs will tell you they passed.
• Treat continued activity after an error signal as a warning sign in its own right. Failed runs receive signals almost as often as successful ones and press on anyway. Point the alert at the steps that follow a signal.

A single pass rate compresses all of this away. The metrics we use to keep it visible are per-task pass^k, suite-level reliability@k, and an interval on every rate; they are laid out in how to measure agent reliability past a single pass rate. The wider method for testing agents beyond one green run lives in the agent reliability testing pillar, with its runnable counterpart in the reliability testing checklist.

This paper’s contribution to that method is a timeline. By the time a coding agent looks like it is failing, it usually already has.