LOG-009 covered PAL, which offloads arithmetic to a Python interpreter so the model never has to compute itself. Self-consistency attacks an orthogonal problem: what if the model reasons correctly most of the time, but not always? The answer turns out to be statistical rather than architectural — and surprisingly effective.
The paper
"Self-Consistency Improves Chain of Thought Reasoning in Language Models" by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou (ICLR 2023, arXiv:2203.11171) introduces a decoding strategy that replaces a single greedy chain-of-thought path with a majority vote over many sampled paths. The intuition is compact: a hard reasoning problem typically has one correct answer but many valid routes to it; a wrong answer is more likely to come from idiosyncratic errors that do not all converge on the same mistake.
The method is plug-and-play. Take whatever chain-of-thought (CoT) prompt you already have, sample N completions at non-zero temperature, extract the final answer from each, and return the majority answer. No fine-tuning, no extra models, no additional human labels.
Key ideas
- Sample size and temperature: The paper uses 40 reasoning paths per problem at temperature 0.7. This is not a magic number from hyperparameter search — ablations show gains plateau around 20–30 samples, so 40 is a conservative choice.
- Main gains over standard CoT: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4%, ARC-challenge +3.9% — all absolute accuracy improvements with the same model and prompt.
- GSM8K results by model: On text-davinci-002 (GPT-3), self-consistency lifts accuracy from 78.7% to 86.5%. On Codex, from 74.5% to 82.3%. Gains are consistent across model families.
- No training cost: Everything happens at inference time. The approach works with any black-box API where you can sample at temperature > 0.
- Majority voting for extractable answers: Aggregation is clean when answers are discrete (a number, a letter choice). For open-ended generation the paper is less specific about defining "most consistent" — a limitation the authors acknowledge.
What holds up — and what doesn't
The empirical gains are real, widely replicated, and the method is genuinely useful. But several structural weaknesses deserve attention.
First, cost scales linearly with the number of samples. Sampling 40 paths at inference costs 40× the token budget of a single path. For tasks where latency and API cost matter — an agent processing hundreds of transactions a night — that is not free. Follow-up work (Early-Stopping Self-Consistency, ICLR 2024) addresses this: by stopping once a vote reaches a confidence threshold, you can cut samples by 80% on GSM8K with no measurable accuracy loss. The base paper does not discuss cost at all, which is an odd omission.
Second, the majority-vote assumption collapses when the model is systematically wrong. If the model consistently misreads a currency conversion or misapplies a tax rule across all 40 paths, the wrong answer wins the vote. Self-consistency amplifies the most common error, not the correct one. That is the central epistemological gap: the method raises precision inside the model's belief distribution, but does nothing for calibration when that distribution is centered on the wrong answer.
Third, Wang & Wang (2025, arXiv:2503.16974) study LLM consistency directly on financial and accounting tasks across 50 independent runs. They find binary classification and sentiment analysis are already nearly perfectly reproducible with a single sample, while complex tasks (forecasting, generation) show real variability. Their practical finding: aggregating just 3–5 runs dramatically improves consistency on complex tasks — a much cheaper version of the same self-consistency idea.
Why this matters for finance AI
Beancount ledger operations that involve multi-step arithmetic — tax calculations, FX-adjusted cost basis, amortization schedules, invoice matching — are exactly the tasks where a single greedy decode is unreliable, yet the correct answer is unique and verifiable. Self-consistency is a cheap intervention that should be standard for any finance-agent task where the output can be checked (is the balance still zero?).
The more interesting implication is architectural. Self-consistency turns inference into a voting ensemble. For write-back safety — an agent posting journal entries into a ledger — I would gate on majority confidence: post only if 35 of 40 paths agree. Disagreement is a signal that the agent should escalate to a human instead of writing. That is a concrete, implementable safety filter that costs inference budget, not engineering complexity.
The systematic-bias failure mode matters especially for tax and regulatory rules, where models are known to hallucinate jurisdiction-specific details. In those cases PAL (LOG-009) is the right fix: offload the computation entirely. Self-consistency and PAL complement each other — PAL handles arithmetic correctness; self-consistency handles ambiguity and reasoning reliability.
What to read next
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et al., 2023, arXiv:2305.10601) — extends self-consistency from voting over paths to searching over paths, which matters when the reasoning space branches rather than runs in parallel.
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning (Lei et al., ICLR 2024) — the fix for the cost problem; cuts sampling by over 80% on GSM8K while preserving accuracy.
- Universal Self-Consistency for Large Language Models (Chen et al., arXiv:2311.17311) — extends majority voting to open-ended generation with an LLM judge, closing the aggregation gap the original paper leaves open.
