DIN-SQL: GPT-4 jumps 67.4%→85.3% EX on Spider
DIN-SQL lifts GPT-4 from 67.4% to 85.3% Spider execution accuracy via schema-link and self-correct stages—same decomposition fits Beancount BQL.
#plain-text-accounting
Plain-Text Accounting
Research grounded in plain-text accounting formats and workflows
BIRD: GPT-4 hits only 54.89% EX on real databases
On BIRD, GPT-4 reaches 54.89% execution accuracy with domain hints and 34.88% without—a 20-point gap any Beancount NL→BQL interface must close.
GraphRAG: 72–83% wins vanish after LLM-judge bias fix
Microsoft's GraphRAG posts 72–83% sensemaking wins over vector RAG; a 2025 audit collapses those after correcting judge bias—caution for multi-doc ledger QA.
StructRAG (ICLR 2025): Picking the Right Document Structure Beats GraphRAG by 28 Points
StructRAG routes each query to a table, graph, catalogue, algorithm, or chunk structure, beating GraphRAG by 28 points and running 22× faster.
Fusion-in-Decoder: How Multi-Passage Retrieval Improves Generative QA
FiD encodes retrieved passages separately and fuses them in the decoder, beating RAG-Sequence by 4–11 points on NQ and TriviaQA, the synthesis ledger QA needs.
IRCoT: Interleaving Retrieval with Chain-of-Thought for Multi-Step QA
IRCoT adds +11.3 recall and +7.1 F1 on HotpotQA over one-step RAG by querying retrieval at every reasoning step. A 3B model can then beat GPT-3 175B.
RAG: 44.5 EM on NQ — and why ledger indexes go stale
Lewis et al. hit 44.5 EM on Natural Questions with a frozen FAISS index; for Beancount ledgers that means reindex-or-miss when balances change daily.
LATS: Language Agent Tree Search — Reasoning, Acting, and Planning in One Framework
LATS unifies ReAct, Tree of Thoughts, and Reflexion in one MCTS framework, hitting 92.7% pass@1 on HumanEval with GPT-4.
Self-RAG: Adaptive Retrieval and Self-Critique for LLMs
Self-RAG trains an LLM to decide when to retrieve and self-grade results, hitting 55.8% on PopQA and 80.2 FactScore — beating ChatGPT on five benchmarks.
Voyager: skill libraries unlock 3.3× more item discovery
Voyager's persistent code skill library discovers 3.3× more Minecraft items than prior SOTA without fine-tuning—the reuse pattern ledger agents need.
HippoRAG: 89.1% Recall@5 on 2Wiki vs 68.2% ColBERT
HippoRAG's OpenIE+PageRank memory hits 89.1% Recall@5 on 2WikiMultiHopQA versus 68.2% ColBERTv2—path-aware retrieval for multi-year ledgers.
AgentBench: Evaluating LLMs as Agents — Lessons for Financial AI Reliability
AgentBench scored GPT-4 4.01 versus 0.96 for the best open-source LLM. Those failures are exactly what breaks a Beancount write-back agent on a live ledger.