HippoRAG: 89.1% Recall@5 on 2Wiki vs 68.2% ColBERT
HippoRAG's OpenIE+PageRank memory hits 89.1% Recall@5 on 2WikiMultiHopQA versus 68.2% ColBERTv2—path-aware retrieval for multi-year ledgers.
#beancount
Beancount
Beancount ledger format, tooling, and ecosystem research
AgentBench: Evaluating LLMs as Agents — Lessons for Financial AI Reliability
AgentBench (Liu et al., ICLR 2024) benchmarks 27 LLMs across 8 interactive environments — GPT-4 scored 4.01 overall versus 0.96 for the best open-source model. The three dominant failure modes (task-limit exceeded in 67.9% of knowledge-graph failures, format errors in 53.3% of database failures, and invalid actions) map directly onto the risks of deploying a Beancount write-back agent on a real ledger.
BloombergGPT and the Limits of Domain-Specific LLMs in Finance
Bloomberg trained a 50B-parameter LLM on 569B tokens of financial data and beat general models on sentiment and table-reasoning benchmarks — then GPT-4 matched it without any finance-specific pretraining. What the $10M experiment reveals about domain pretraining trade-offs, tokenization of numbers, and why tool-use is more reliable than model internals for accounting agents.
AutoGen: Multi-Agent Conversation Frameworks for Finance AI
AutoGen (Wu et al., 2023) introduces a multi-agent conversation framework where LLM-backed agents pass messages to complete tasks; a two-agent setup lifts MATH benchmark accuracy from 55% to 69%, and a dedicated SafeGuard agent improves unsafe-code detection by up to 35 F1 points — findings directly applicable to building safe, modular Beancount automation pipelines.
Gorilla: How Retrieval-Aware Training Reduces LLM API Hallucinations from 78% to 11%
Gorilla (Patil et al., NeurIPS 2024) fine-tunes a 7B LLaMA model with Retriever-Aware Training on retrieved API documentation, cutting hallucination rates from 78% to 11% versus GPT-4 zero-shot — with direct implications for finance AI write-back agents where wrong account names or inverted signs are correctness failures, not annoyances.
MemGPT: 92.5% multi-session recall vs 32.1% baseline
MemGPT's OS-style memory tiers push GPT-4 multi-session chat accuracy to 92.5% versus 32.1% fixed-context—needed for multi-year ledger agents.
SWE-agent: How Interface Design Unlocks Automated Software Engineering
SWE-agent (NeurIPS 2024) introduces Agent-Computer Interfaces (ACIs) — purpose-built layers between LLMs and software environments — showing a 10.7-percentage-point improvement over raw shell access and 12.47% resolution on SWE-bench with GPT-4 Turbo. Interface design, not model capability, is the primary bottleneck for autonomous coding agents.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
SWE-bench evaluates language models on 2,294 real GitHub issues across 12 Python repositories using execution-based tests; at publication, Claude 2 resolved only 1.96% of issues with realistic retrieval, establishing the de facto benchmark for coding agents and revealing retrieval and patch-length failure modes directly relevant to Beancount write-back agents.
CodeAct: Why Executable Python Code Makes LLM Agents 20% More Accurate
CodeAct (ICML 2024) replaces JSON tool-calling with executable Python code, improving GPT-4 agent success rates by ~20 percentage points on multi-tool tasks and reducing interaction turns by 30% — with direct implications for building reliable Beancount reconciliation agents.
Self-Correction: GPT-4 falls 95.5%→91.5% on GSM8K
Huang et al. (ICLR 2024): intrinsic self-correction drops GPT-4 from 95.5% to 91.5% on GSM8K—ledger agents need external validators, not self-review.
Reflexion: Language Agents That Learn from Mistakes Without Retraining
Reflexion (NeurIPS 2023) lets LLM agents improve by storing verbal post-mortems in an episodic buffer — no weight updates required. It reaches 91% on HumanEval with GPT-4 but fails on WebShop, revealing a structural constraint: verbal reinforcement only works when the evaluator produces a crisp, actionable signal. Here is what that means for building a self-correcting Beancount ledger agent.
PAL: +38pp on hard math by handing arithmetic to Python
PAL gains +38.1pp over chain-of-thought on GSM-hard by running Python for arithmetic—the right split for reliable Beancount ledger calculations.