GuardAgent: Deterministic Safety Enforcement for LLM Agents via Code Execution
GuardAgent enforces LLM agent policies by running Python code, hitting 98.7% accuracy with no task failures, versus 81% and up to 71% failure for prompt rules.
#ai
AI
Artificial intelligence research and applications in finance and accounting
Multiagent LLM Debate: Real Accuracy Gains, Uncontrolled Compute, and Collective Delusion
Multiagent debate gained 14.8 points on arithmetic, but equal-budget single agents match it. That limits debate as a safe check before a ledger commit.
LLMs Are Not Useful for Time Series Forecasting: What NeurIPS 2024 Means for Finance AI
NeurIPS 2024 ablation: dropping the LLM from Time-LLM and CALF improves accuracy, with up to 1,383× faster training. Use purpose-built models for finance AI.
AuditCopilot: LLMs for Fraud Detection in Double-Entry Bookkeeping
AuditCopilot cuts journal entry fraud false positives from 942 to 12, but ablation shows the LLM mainly synthesizes Isolation Forest scores.
TAT-LLM: Fine-Tuned 7B LLaMA 2 Beats GPT-4 on Financial QA
TAT-LLM fine-tunes LLaMA 2 7B to 64.60% EM on FinQA, edging GPT-4's 63.91%: an extract-reason-execute pipeline lets a small model do table arithmetic.
Fine-Tuning vs. RAG: Why Retrieval Wins for Injecting New Knowledge into LLMs
RAG hits 0.875 accuracy on post-cutoff facts while fine-tuning plateaus at 0.504. For agents needing frequent ledger updates, retrieval beats fine-tuning.
IRCoT: Interleaving Retrieval with Chain-of-Thought for Multi-Step QA
IRCoT adds +11.3 recall and +7.1 F1 on HotpotQA over one-step RAG by querying retrieval at every reasoning step. A 3B model can then beat GPT-3 175B.
FLARE: Active Retrieval Augmented Generation
FLARE hits 51.0 EM on 2WikiMultihopQA versus 39.4 for single-retrieval RAG, but calibration failures in chat models limit it for finance agents.
RAG: 44.5 EM on NQ — and why ledger indexes go stale
Lewis et al. hit 44.5 EM on Natural Questions with a frozen FAISS index; for Beancount ledgers that means reindex-or-miss when balances change daily.
MultiHiertt: Benchmarking Numerical Reasoning Over Multi-Hierarchical Financial Tables
MultiHiertt shows models score 38% F1 against 87% for humans on 10,440 financial QA pairs, with a 15-point drop on cross-table questions.
ConvFinQA: models trail experts by 21 points (68.9%)
ConvFinQA's best model scores 68.9% execution accuracy versus 89.4% for human experts—a 21-point gap that multi-turn ledger chat still faces.
TAT-QA: Hybrid Table-Text QA Benchmark for Financial Annual Report Reasoning
TAT-QA's hybrid table-text questions showed evidence grounding, not arithmetic, is finance AI's bottleneck. Fine-tuned 7B LLMs hit 83% F1 by 2024.