Multiagent LLM Debate: Real Accuracy Gains, Uncontrolled Compute, and Collective Delusion

I've been thinking about multi-agent verification for Beancount write-back safety — specifically, whether a checker agent can meaningfully debate a writer agent before a ledger commit lands. That question led me back to the foundational paper on multiagent debate, which ran as an ICML 2024 paper and has since attracted a useful body of critical follow-up work.

The paper

"Improving Factuality and Reasoning in Language Models through Multiagent Debate" by Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch proposes what they call a "society of minds" approach: multiple LLM instances each generate an initial response, then read the full set of peer responses and update their answer over multiple rounds. The key design choice is that the approach requires only black-box access to model outputs — no gradients, no fine-tuning, no architecture changes. They test it across six benchmarks: arithmetic, GSM8K, chess move optimality, biographical factuality, MMLU, and chess move validity.

The setup they report most results on is 3 agents debating for 2 rounds. The conceptual bet is that disagreement forces agents to articulate their reasoning, while convergence signals genuine confidence rather than lucky consistency.

Key ideas

• On arithmetic, debate reached 81.8% accuracy vs 67.0% for a single agent and 72.1% for single-agent reflection — a 14.8-point gain over the baseline.
• On GSM8K (grade school math), 85.0% vs 77.0% single agent and 75.0% with reflection.
• On MMLU (100 questions spread across subject areas), 71.1% vs 63.9% single agent and 57.7% with reflection.
• On biographical factuality, 73.8% vs 66.0% single agent.
• Cross-model debate (ChatGPT + Bard on 20 GSM8K problems) solved 17/20 vs 11–14 for either model individually — the most striking result in the paper because it shows heterogeneous agents catching each other's errors.
• Performance scaled with both agent count and round count through 4 rounds, with diminishing returns beyond that. "Long" prompts explicitly encouraging agents to slow down before consensus consistently outperformed short prompts.

What holds up — and what doesn't

The gains are real, and the benchmark coverage is wider than most prompting papers. I believe the directional finding: having multiple agents critique each other catches more errors than a single agent reflecting on its own output.

The problem is what isn't controlled. Three agents debating for two rounds means roughly 6× the inference compute of a single call, before you account for longer context. The paper never presents an equal-budget baseline. Self-consistency — majority voting over many independent single-agent samples — is a natural comparison that the paper addresses only briefly. A 2025 paper (arXiv:2604.02460) runs exactly this control on multi-hop reasoning benchmarks across Qwen3, DeepSeek-R1, and Gemini 2.5 with matched reasoning-token budgets, and finds that "single-agent systems can match or outperform MAS" once compute is equalized. That's a direct challenge to the main claim.

The other failure mode the paper acknowledges but underweights is what M3MAD-Bench (arXiv:2601.02854) calls "Collective Delusion": across a manual analysis of 100 debate failures, 65% involved agents mutually reinforcing wrong answers rather than correcting them. The paper's own text notes that agents sometimes "confidently affirm that their answer is correct" even when converging on an incorrect answer. When all agents share the same training distribution — the homogeneous case — they're likely to share the same blind spots. The debate then amplifies the error rather than catching it.

A related finding from the same paper: "Incorrect Conformity" accounts for a non-trivial share of failures — a correct agent abandons sound reasoning after reading peer responses that are wrong. This is the opposite of what the debate framework is supposed to do. It's a reminder that the persuasion dynamics in these multi-agent loops can run in either direction.

Why this matters for finance AI

The architecture is genuinely appealing for Beancount write-back safety: writer proposes a ledger entry, checker debates it, consensus triggers commit. The risk analysis changes depending on what you're writing. For a routine grocery expense, the cost of a debate round isn't worth it. For a tax-year-end journal entry or an intercompany transfer, having a second agent scrutinize the account codes and amounts before commit is defensible.

But Collective Delusion is particularly dangerous for accounting. If both a writer and checker agent share the same wrong belief about how a specific deduction is categorized under a given jurisdiction's rules, the debate confirms the error rather than flagging it. The paper's own cross-model result hints at the fix: heterogeneous agents — different models, different system prompts, or one agent grounded on external documentation — are more likely to surface genuine disagreement. M3MAD-Bench confirms that "collaborative heterogeneous debate" substantially outperforms homogeneous setups.

The compute multiplication also matters at production scale. Ten ledger edits per session × 3 agents × 2 rounds = 60 LLM calls. That's sustainable for high-stakes writes but not for routine transaction import. The right design is probably a tiered approach: fast single-agent path for well-structured entries, debate invoked only when the writer expresses uncertainty or when the entry affects a high-sensitivity account class (tax liabilities, retained earnings, intercompany).

What to read next

• arXiv:2604.02460 — "Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets": the cleanest published challenge to debate's claimed compute advantages.
• arXiv:2601.02854 — M3MAD-Bench: large-scale evaluation of debate across 9 models and 13 datasets, with the Collective Delusion failure taxonomy.
• arXiv:2406.09187 — GuardAgent: a guard agent that translates safety policies into executable code; a more direct approach to write-back safety than debate-based consensus.