DIN-SQL: GPT-4 jumps 67.4%→85.3% EX on Spider
DIN-SQL lifts GPT-4 from 67.4% to 85.3% Spider execution accuracy via schema-link and self-correct stages—same decomposition fits Beancount BQL.
#plain-text-accounting
Plain-Text Accounting
Research grounded in plain-text accounting formats and workflows
BIRD: GPT-4 hits only 54.89% EX on real databases
On BIRD, GPT-4 reaches 54.89% execution accuracy with domain hints and 34.88% without—a 20-point gap any Beancount NL→BQL interface must close.
GraphRAG: 72–83% wins vanish after LLM-judge bias fix
Microsoft's GraphRAG posts 72–83% sensemaking wins over vector RAG; a 2025 audit collapses those after correcting judge bias—caution for multi-doc ledger QA.
StructRAG (ICLR 2025): Picking the Right Document Structure Beats GraphRAG by 28 Points
StructRAG (ICLR 2025) routes each query to a task-appropriate structure type — table, graph, catalogue, algorithm, or chunk — before reasoning, scoring 28 points higher than GraphRAG on the Loong benchmark while running 22× faster, with the DPO-trained router alone accounting for a 15-point accuracy gain.
Fusion-in-Decoder: How Multi-Passage Retrieval Improves Generative QA
Izacard and Grave's FiD architecture independently encodes retrieved passages then fuses them in the decoder, outperforming RAG-Sequence by 4–11 points on NQ and TriviaQA. This post examines the design and its implications for Beancount ledger QA, where multi-entry synthesis across transactions is the norm.
IRCoT: Interleaving Retrieval with Chain-of-Thought for Multi-Step QA
IRCoT interleaves BM25 retrieval with each step of a chain-of-thought reasoning loop, achieving +11.3 retrieval recall and +7.1 F1 on HotpotQA over one-step RAG — and shows a 3B model can beat GPT-3 175B when retrieval strategy is right.
RAG: 44.5 EM on NQ — and why ledger indexes go stale
Lewis et al. hit 44.5 EM on Natural Questions with a frozen FAISS index; for Beancount ledgers that means reindex-or-miss when balances change daily.
LATS: Language Agent Tree Search — Reasoning, Acting, and Planning in One Framework
LATS (Language Agent Tree Search, ICML 2024) unifies ReAct, Tree of Thoughts, and Reflexion in a single MCTS framework, reaching 92.7% pass@1 on HumanEval with GPT-4. For git-backed Beancount ledgers, the state-reversion requirement that constrains LATS in production is trivially satisfied.
Self-RAG: Adaptive Retrieval and Self-Critique for LLMs
Self-RAG (ICLR 2024 Oral) trains a language model to decide when to retrieve and then grade its own results using four reflection tokens — reaching 55.8% on PopQA and 80.2 FactScore on biographies while outperforming ChatGPT on five benchmarks. Analysis covers the mechanism, ablation results, reproducibility limits, and implications for finance AI agents over Beancount ledgers.
Voyager: skill libraries unlock 3.3× more item discovery
Voyager's persistent code skill library discovers 3.3× more Minecraft items than prior SOTA without fine-tuning—the reuse pattern ledger agents need.
HippoRAG: 89.1% Recall@5 on 2Wiki vs 68.2% ColBERT
HippoRAG's OpenIE+PageRank memory hits 89.1% Recall@5 on 2WikiMultiHopQA versus 68.2% ColBERTv2—path-aware retrieval for multi-year ledgers.
AgentBench: Evaluating LLMs as Agents — Lessons for Financial AI Reliability
AgentBench (Liu et al., ICLR 2024) benchmarks 27 LLMs across 8 interactive environments — GPT-4 scored 4.01 overall versus 0.96 for the best open-source model. The three dominant failure modes (task-limit exceeded in 67.9% of knowledge-graph failures, format errors in 53.3% of database failures, and invalid actions) map directly onto the risks of deploying a Beancount write-back agent on a real ledger.