Limitations of multi-agent debate: what the paper found

A debate stage reads like a free reliability upgrade. Three models answer, argue for two rounds, and return a consensus that should beat any one of them alone. In the experiments below, that consensus was often worth less than a plain majority vote taken before the models ever saw each other’s reasoning.

Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate (Wynn, Satija and Hadfield, arXiv 2509.05396; posted September 2025, revised October 2025, out of Johns Hopkins and the Vector Institute) measures what the exchange of reasons does to accuracy in mixed-capability groups. The direction is down. It is a workshop paper, accepted at the ICML 2025 Multi-Agent Systems workshop rather than a main conference or a journal, and nothing here has been independently replicated, though the work has been picked up: Semantic Scholar’s record listed 54 citations on 6 August 2026. The cost side matters if you are weighing whether debate rounds belong in your pipeline, because debate multiplies inference spend by agents times rounds, so a stage that subtracts accuracy is charging you more for a worse answer.

Before you ship a debate stage, score it against the same agents’ first-answer majority vote on your own task. Two rounds of debate lowered that baseline in 21 of the 30 task-by-composition cells this paper reports, and in 15 of the 21 cells where the group mixed capability levels. An explicit reward for correctness made the harmful flips more common.

The convergence failure at the center of this, where debate rounds ratify a shared error instead of catching it, is defined at multi-agent debate failure. The wider catalog of ways agent teams break lives in the multi-agent failure-mode taxonomy. What follows is the paper’s evidence and the point where it stops.

The baseline is the same agents, voting in silence

Everything below is a delta against a control arm that adds no rounds to the bill.

The setup uses three models spanning a capability range: GPT-4o-mini, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2. Groups of three answer CommonSenseQA, MMLU, and GSM8K questions, then debate for two rounds, with 100 random samples per task, five random seeds, default temperature, and top_p at 0.9. The debate prompt follows the protocol of Du et al.’s multiagent debate paper (ICML 2024), the work that proposed debate as a way to improve factuality and reasoning by letting model instances critique one another across rounds.

The “without debate” figure is a majority vote over the agents’ initial, independent answers. The “after debate” figure is the same vote taken after two rounds in which each agent reads its peers’ reasoning and may revise. With agents, items, and seeds held constant, the delta prices exactly one thing: what reading and responding to peer reasoning adds or removes. Consensus and voting reliability covers the static half of that comparison. A vote locks every ballot before any agent sees another; debate opens a channel between the ballots, and whatever travels through that channel, correction or persuasion, is what the paper is measuring.

The alternative debate had to beat came from the same three models, one answer each, with no second or third round to pay for.

Accuracy fell with the stronger models in the majority

The single-agent baselines set the capability spread. On CommonSenseQA, GPT-4o-mini scores 74.8 ± 1.9%, Llama-3.1-8B-Instruct 57.0 ± 1.5%, and Mistral-7B-Instruct-v0.2 41.6 ± 2.3% (accuracy, mean ± standard error over five seeds). Debate was supposed to pull groups toward the top of that range. Seven of the thirty cells in the paper’s Table 1: two compositions across all three tasks, plus the grid’s largest single drop.

Accuracy in %, mean ± standard error over five seeds of 100 samples each. “Vote before debate” is the majority vote on initial answers; “after debate” is the same vote after two debate rounds.

The 2× Llama-3.1 plus 1× Mistral group put the stronger model in the majority and let every member read the best available reasoning in full. It lost ground on all three tasks. The abstract states the same result in general form: accuracy decreased “even in settings where stronger (i.e., more capable) models outnumber their weaker counterparts.”

We counted the sign of the after-minus-before delta in every cell of the paper’s Table 1, across its ten group compositions and three tasks: 21 of the 30 decline. Three of those ten compositions run three copies of one model, and their nine cells carry six of the declines, so the seven groups that actually mix capability levels account for 15 declines across 21 cells. The count takes every move at face value, and several of the smaller ones sit inside their own standard errors. CommonSenseQA declines in all ten compositions, MMLU in seven of ten, and GSM8K in four of ten. The nine cells that improved are all on GSM8K or MMLU. Nothing gained on CommonSenseQA. GSM8K is also the only one of the three tasks where a wrong step can be caught by redoing the arithmetic. Debate held up best on that checkable reasoning task, the one its advocates usually propose it for, and worst on multiple-choice knowledge recall, where a persuasive wrong answer has nothing to push back against it.

Models revise toward agreement

Underneath the averages, the paper tracked what each agent’s answer did from one round to the next.

The authors report that “models frequently shift from correct to incorrect answers in response to peer reasoning, favoring agreement over challenging flawed reasoning.” Among the agents that changed their answer at all, the correct-to-incorrect direction was the larger share. The paper presents that split as a figure rather than a table of counts, so the direction is what it establishes, and no ratio can be quoted from it. A panel that talks itself out of its own right answers still produces a transcript full of confident convergence. That convergence is error propagation running over a persuasion channel: one agent’s confident wrong answer becomes input the next agent trusts, and the trust repeats each round.

The Discussion names three of the contributing factors it studied: sequential revision, social conditioning, and sycophancy. It declines to crown any of them, concluding that “multiple factors exist that likely contribute to failure modes in multi-agent debate.” The paper is an empirical failure study, so it publishes no numbered taxonomy, and it should be cited as evidence rather than as a classification scheme.

Flip probability tracks how many peers disagree with an agent, and the authors conclude that models’ “internal mechanisms for balancing correctness vs consensus may be quite fragile in the face of disagreement with peers.” The effect also splits sharply by model: on GSM8K, the authors report, “the strongest and weakest models (GPT and Mistral) are far less likely to make undesirable answer-flips than the third model (LLaMA).” Which model sits in your panel changes how much conformity you buy.

Two boundaries keep this finding distinct from its neighbors in the vocabulary. The damage here travels through context that arrived intact: in information withholding a downstream agent underperforms because context never reached it, while in debate degradation the peer reasoning arrives in full, gets read, and does the harm. And in the taxonomy’s terms the pattern sits closest to inter-agent misalignment: each agent’s revision toward its peers is locally cooperative, and the joint outcome lands below what the group already had before anyone spoke.

Paying the models to be right made it worse

The obvious mitigation is to tell the models that being right pays, and the paper ran that experiment.

The authors add a correctness payoff to the debate prompt, modeled on political-science survey experiments from 2015 that cut partisan bias by paying respondents for correct answers: the model is told it will be rewarded for a correct final answer. The incentive did not help. The paper reports that “adding a payoff for correctness in the model prompt does not significantly reduce the likelihood that LLM agents flip their answers from correct to incorrect,” and that in many cases the number of correct-to-incorrect transitions actually increases under the incentive.

The paper reports that increase and moves on. No mechanism for it is established, and we have not seen a replication, so treat the direction as a caution and check it on your own stack. Either way the practical point holds: if your mitigation plan for debate drift is a system-prompt line about accuracy, this experiment already ran it, and the flips it was meant to prevent became more common.

Where the result stops

The finding is real and the conditions around it are narrow. The debaters are small 2024-era open models plus GPT-4o-mini. Alone, Mistral-7B-Instruct-v0.2 scores 41.6 ± 2.3% on CommonSenseQA and 34.0 ± 1.9% on MMLU, close to the level where the task is barely being attempted. Three of those agents voting together on MMLU reach 33.6 ± 1.8% before any debate opens, which is the single-agent score again inside the same error bars. A group built that way is a weak test of capability diversity. Whether frontier models in 2026 conform the same way under debate is a question this paper cannot answer.

Each cell rests on 100 samples and five seeds, and the drops in the table above run from 5.0 to 12.0pp against standard errors between 1.2 and 3.9pp. The paper publishes no interval on the differences themselves, so that comparison against the per-arm errors is a rough one, and several individual cells sit near their own noise floor. The direction repeats across compositions regardless, above all in the clean sweep on CommonSenseQA.

The debates are also short. With T = 2, any claim about accuracy decaying “as the debate progresses” covers a very short progression, and the paper doesn’t establish that longer debates keep falling. Two of its three benchmarks are multiple-choice knowledge recall, and the one where debate did best, GSM8K, is the one whose answer an agent can re-derive from scratch.

One structural note for anyone quoting the paper: it carries no section titled “Limitations.” The authors’ own hedging lives in the Discussion, so the list above is our skeptical reading, and the model-vintage and two-round points are the ones a defender of debate would press first.

What to run before you ship a debate stage

None of this says debate can never pay. Nine of the thirty cells improved, and if your task looks like GSM8K, where an agent can recheck a peer’s arithmetic instead of taking its word for it, debate has a real case to make. The burden of proof still sits on the debate stage, and the paper hands you the protocol for testing it.

1. Score the no-debate baseline in every debate eval: a majority vote over the same agents’ first answers, on the same items. A debate stage that can’t beat that vote is subtracting accuracy at added cost.
2. Log per-round answers and count transitions by direction. Aggregate accuracy can hold steady while correct-to-incorrect flips climb, and flip direction is the failure signal this paper validated.
3. Treat debate-on versus debate-off as a paired comparison on the same items, which calls for the McNemar test rather than two independent pass rates. Size the run first with the sample size and power calculator, because at the paper’s own scale of 100 items per task, a 5pp drop can sit inside the interval.
4. Skip the correctness-incentive prompt as a fix. It has been tried, and in many of the paper’s settings the flips it was meant to prevent became more common.

The authors’ own prescription points the same way: future debate frameworks should promote “critical evaluation over consensus,” weight contributions by confidence or credibility, and reward independent verification of claims. Their code is public in the paper’s repository if you want to rerun the grid with your own models.

Debate is one of five coordination topologies, and each amplifies its own failure family; the orchestration patterns page maps the other four and the propagation shape each one buys. For the names these failures answer to, start from the multi-agent failure vocabulary, and for the classification work that keeps extending as new failure modes surface, see AdaMAST, the adaptive successor to MAST. Whichever topology you land on, the baseline to beat stays the same one: your own agents, answering once, voting in silence.