AdaMAST: an adaptive taxonomy for multi-agent failures

MAST gave multi-agent debugging its shared vocabulary. Published in March 2025, it sorted the ways multi-agent LLM systems fail into fourteen modes, hand-annotated on a 150-trace seed and then scaled to 1,642 traces by an LLM judge, and it is the taxonomy that the propagation mapping in our pillar on multi-agent failure modes is built on. In July 2026 the same first author, now sharing that role, published a successor paper arguing that a fixed, hand-authored failure taxonomy is the wrong artifact to maintain, MAST included.

That argument, if it holds, reaches every team that adopted MAST codes for triage, ours among them. Read the paper closely, though: its own tables put most of the measured gain somewhere other than its title.

AdaMAST induces a failure taxonomy from a target system’s own execution traces: named codes on three fixed axes, with no human in the annotation loop. Feedback expressed in those codes beat free-text reflection wherever the paper ran the two against each other. Against MAST’s fixed vocabulary the adaptive codes added roughly one to five points, and what supplies the rest of each climb changes with the harness: a structured vocabulary on SWE-agent, an even split on Claude Code, the selection machinery itself on Terminal-Bench.

A successor built by the group that wrote the original

The paper is Fantastic Adaptive Taxonomies and How to Use Them (arXiv 2607.16387, preprint, first posted 2026-07-17), revised on 2026-07-29. Mert Cemri, joint first author here with Andrei Cojocaru, is also the first author of MAST (Why Do Multi-Agent LLM Systems Fail?, arXiv 2503.13657, NeurIPS 2025 spotlight poster). The overlap runs past one name: Pan, Ramchandran, Gonzalez, Zaharia and Stoica appear on both author lists, and eleven of the twelve authors sit at UC Berkeley. Read that as provenance. The group that built the field’s reference taxonomy now argues the taxonomy should be induced per system instead of written once for everyone.

The preprint went up in July 2026, carries no publication venue, and showed no citations on Semantic Scholar when we checked it on 2 August 2026. Nothing in it has been replicated. The results may well hold; until somebody outside Berkeley runs them, they are one lab’s unreviewed measurements, and we treat them that way below.

What the lab built is a pipeline, and its output, one taxonomy per target system, is what every result below consumes.

Named codes on three fixed axes, none of them hand-written

AdaMAST reads a target system’s execution traces and emits a failure taxonomy for that system: named failure codes, each with a definition and an evidence pattern, every one induced from the traces. No code is hand-authored and no trace is human-annotated. The codes slot into three axes that never change:

Each axis names the place a fix would land: the harness, the role wiring, or the task knowledge. When AdaMAST processed TheoremQA, which the authors run with “a flat solver-verifier architecture” and no differentiated roles, the induced taxonomy contained no B codes at all: 13 codes, four system-level and nine domain-specific. Role failures presuppose roles, and a multi-agent system wired as a flat solver-verifier pair has none to discover. An empty axis is a reading of your topology.

An induced taxonomy also has to pass a gate before anything consumes it. Four independent LLM annotators label stratified held-out traces, over up to five rounds of five traces each. The taxonomy is accepted once mean pairwise agreement reaches a kappa of at least 0.75 with a coverage floor of 0.70. A failed round triggers merge, add, or relabel edits and a re-run. Kappa is the same agreement statistic our inter-rater reliability calculator computes for judge-versus-human labels. The gate certifies that independent annotators apply the codes consistently, and consistency and correctness are different properties; the pushback section below picks that thread up.

The vocabulary also moves. As the system evolves, online refinement merges, adds, or relabels codes “so the vocabulary tracks the current system, rather than a snapshot of its past.” In the paper’s OlympiadBench run, refinement fired at iterations 29, 78 and 92 of a 97-iteration run. The taxonomy started at 36 codes and the final snapshot holds 16, because a refinement event retires more codes than it adds.

The paper’s own examples show what the pipeline produces, and how little two of its taxonomies have in common.

Every system grows a different vocabulary

For TheoremQA’s 30 graduate-level math problems, the dominant induced code was premature reasoning truncation, which the paper’s own code list writes as Premature_Reasoning_Truncation: the solver halts a multi-step derivation partway through and emits an intermediate quantity as the final answer. It fired at 50%, with token exhaustion second at 14%. For Frontier-CS, a competitive-programming system spanning solver, reviewer and classifier roles, the taxonomy ran to 25 codes across all three axes, led by algorithm mismatch at 18.3% and budget exhaustion at 15.7%. All four are firing frequencies, and the paper publishes no failure counts under them, so on a 30-problem benchmark 50% and 14% may rest on single-digit numbers. The Terminal-Bench 2.0 codes turn into verification criteria with a sysadmin’s texture: a training run that stalls with no error message, only a timeout; systemctl failing inside a Docker container while the agent carries on unaware.

Across six domains the induced vocabularies barely overlap, at a mean pairwise Jaccard similarity of 0.14. Jaccard counts shared codes against the two taxonomies’ combined vocabulary, so 0.14 puts roughly one code in seven of that combined set in both. The rest belong to one domain only. Each vocabulary is also compact. On 223 traces the paper compresses the failure-relevant content about 18-fold with the codebook amortized across the corpus, and about 9-fold per trace on its own, while 89% of traces keep a unique code signature. On the TRAIL benchmark the induced vocabulary matched expert failure annotations more closely than a hand-crafted reference vocabulary did, at a kappa of 0.682 against 0.516.

A code names a recurring pattern. Working out where one specific run broke is a separate job, failure attribution, and coded traces make that reconstruction cheaper by handing the cascade tree its labels in advance.

Naming failures is the taxonomy half of the paper. The other half is what the names are for.

The headline numbers only hold with their baselines attached

The paper tests induced codes as a feedback medium in three procedures, and the coded feedback wins in all three.

In agent-system search, taxonomy-coded diagnoses of failed candidate systems beat the search loop’s default free-form reflection on all five benchmarks tested. The margin runs from 3.5 points on DROP to 7.5 on MMLU-Pro, where accuracy climbs from 35.0% to 42.5%; on Frontier-CS it goes from 26.0% to 32.7%.

In runtime monitoring the abstract compresses the ladder in a way that will mislead a fast reader. On SWE-bench Verified Mini:

The abstract’s “from 60% with free-text reflection to 70%” is accurate and easy to misread: the 60% is the Reflexion baseline, and the unaided base is 50%. Quote it as ten points over free-text reflection, or as twenty points over no reflection at all, and say which comparison you mean. Say which model, too: the Claude Code row runs Haiku 4.5, so the 70.7% describes a Haiku 4.5 configuration and says nothing about a frontier Claude Code setup.

In trajectory selection, AdaMAST-Judge, an LLM verifier built on the induced codes, picks among five sampled trajectories on Terminal-Bench 2.0. It improves on Pass@1 by 8.1 to 14.9 points depending on harness: terminus-2 goes from 61.8% to 73.0%, claude-code from 57.5% to 72.4%, ForgeCode from 81.8% to 89.9%.

Pass@1 is the weakest arm in that table. Substituting the MAST checklist into the same verifier pipeline already reaches 68.5%, 69.0% and 88.8% on the three harnesses, and on terminus-2 a generic LLM-as-a-verifier baseline does better still, at 71.2%. Measured against the strongest arm below it on each harness, AdaMAST-Judge is ahead by 1.8, 3.4 and 1.1 points.

Those three results do not decompose the same way as each other. Splitting each one is where the paper’s framing and its own numbers come apart.

Where the rest of the gain sits changes from one experiment to the next

One thing holds wherever the paper ran the comparison: coded feedback beat free text, and the margin the adaptive codes add over MAST is small. What sits underneath that margin is different in each experiment.

On SWE-agent, moving from free-text reflection to MAST codes is worth eight points (60% to 68%), and moving on to the adaptive codes adds two more. On Claude Code, MAST codes add 3.3 points over the unaided base and the adaptive codes another 3.4, so the two steps split that climb almost evenly.

The selection table works differently. On Terminal-Bench the adaptive margin over the MAST rows is 4.5, 3.4 and 1.1 points, and the selection machinery carries the rest. On terminus-2 a generic verifier with no failure vocabulary at all takes 9.4 of the 11.2 points between Pass@1 and AdaMAST-Judge, leaving 1.8 for everything taxonomic, and MAST at 68.5% sits below that verifier. The OlympiadBench search run reorders things again: a MAST-guided arm reached 89.5% against free-form reflection’s 87.9% and AdaMAST’s 91.9%, so there the fixed checklist captured under half the margin.

The repeated result is that coded feedback beats free text; the measured case for adaptivity is one to five points on top of a fixed taxonomy. The paper is titled and framed around the adaptive part, and the Jaccard figure gives adaptivity a real basis, since vocabularies induced for different domains genuinely differ. Its cash value stayed inside that one-to-five-point band on every harness the paper ran. It is also bought with an induction and validation pass per target system, run once and then amortized, that a fixed taxonomy never charges at all.

A result this new needs its caveats stated by someone, and the paper leaves most of that work to the reader.

What the paper does not say about itself

Search the full text of the 39-page revised PDF and the word “limitation” never appears, in a heading or anywhere else. For a paper making three separate efficacy claims, a missing limitations section is a gap a summary should surface rather than smooth over. The constraints do exist. You have to assemble them from the margins.

The selection numbers are entangled with the selection machinery. The paper says so itself: the Terminal-Bench gains reflect “both the induced vocabulary and the selection pipeline,” so the 8-to-15-point range measures the two together and neither one cleanly.

An n of 1 carries no interval, and the two largest evolution gains rest on one run each. Frontier-CS and OlympiadBench are single evolutionary runs; the multi-seed replications go to the smaller benchmarks, TheoremQA at four seeds and DROP at five.

The loop rarely touches a human. Codes are induced by LLMs, the acceptance gate is four LLM annotators, and AdaMAST-Judge is an LLM verifier; the one human-grounded check is the TRAIL comparison. A vocabulary generated by models and validated by model agreement inherits the documented biases of LLM judges, and the kappa gate certifies consistency, which four annotators sharing one blind spot would pass comfortably. That concern is general rather than partisan, since MAST scaled its own 150-trace human seed to 1,642 traces with an LLM judge.

The head-to-head is also self-scored. MAST, the baseline AdaMAST most needs to beat, comes from the same group. That comparison is fair to run, and it’s still one where the same lab wrote both contestants.

Sized that way, the paper still leaves a practitioner with something concrete to do.

Adopt the vocabulary before the pipeline

This lands first on teams running multi-agent systems in production with a trace archive, on teams doing automated agent-system search or prompt optimization, and on teams already paying for best-of-n sampling without a principled selector. If you run a single agent, or you keep no trace archive to induce from, there’s nothing here to adopt yet, because the pipeline has no input.

The transferable move needs no pipeline. If your post-incident feedback is free-text reflection, adopt a named failure vocabulary and route feedback through it; on this paper’s SWE-agent ladder that single step was worth eight of the ten points between free text and the induced codes. A hand-maintained vocabulary in the MAST style works today, and the term-by-term failure vocabulary this site maintains, with entries like task verification failure and specification violation, is built for exactly that use.

If you already label traces with fixed codes, the upgrade case is the one-to-five-point margin above, paid for with the induction cost and a validation gate you have to actually run. The released AdaMAST code is where to price that out against your own traces.

Read the empty axes, too. An induced taxonomy with no role codes is telling you your topology has no real roles, and one dominated by system-level codes points at the harness rather than the agents. Which orchestration pattern you chose decides which of those failures you amplify, so the shape of the induced vocabulary doubles as a cheap topology review.

Where a coded failure travels after it fires belongs to error propagation, and the pillar mapping each failure family to how it spreads and what contains it is the natural next page. For the neighboring result on when adding more agent discussion makes answers worse, see the companion explainer on the limits of multi-agent debate.

Adopt structured failure codes now, in whatever form your team can maintain, and move to automatic induction once a fixed vocabulary is visibly failing to name what your traces contain.