Can your agent balance these books?
One agent run reconciled a three-row ledger to 2958.50 USD checking and 1958.50 USD September profit, recovering from one failure it diagnosed itself.
#llm
LLM
Large language model research with applications in financial tasks
FinRAGBench-V: Multimodal RAG with Visual Citations in the Financial Domain
FinRAGBench-V finds top models reach only 20–61% block-level citation recall on financial pages. Multimodal retrieval beats text-only by nearly 50 points.
Can LLM Agents Be CFOs? EnterpriseArena's 132-Month Simulation Reveals a Wide Gap
Only Qwen3.5-9B survives 80% of EnterpriseArena's 132-month CFO runs, while GPT-5.4 and DeepSeek-V3.1 hit 0%. Skipped ledger reconciliation causes the failures.
WildToolBench: Why No LLM Exceeds 15% Session Accuracy in Real-World Tool Use
WildToolBench finds no LLM exceeds 15% session accuracy on 1,024 real-user tasks, with hidden intent and instruction transitions the sharpest failure modes.
LLM Confidence and Calibration: A Survey of What the Research Actually Shows
Verbalized GPT-4 confidence hits only ~62.7% AUROC, barely above chance. Uncertainty-aware finance agents need better calibration than that.
JSONSchemaBench: Real-World Schema Complexity Breaks LLM Structured Output Guarantees
JSONSchemaBench finds coverage collapses from 86% on simple schemas to 3% on complex ones, so LLM structured output can silently emit non-compliant JSON.
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under MCP
FinMCP-Bench scores the best of six LLMs at just 3.08% exact match on 613 real MCP financial tasks, a 20× collapse from single-tool to multi-turn use.
FinTrace: Trajectory-Level Evaluation of LLM Tool Calling for Financial Tasks
FinTrace shows frontier LLMs pick the right financial tools (F1 ~0.9) but score just 3.23/5 on using the results, the step that breaks write-back agents.
FinToolBench: Evaluating LLM Agents on Real-World Financial Tool Use
FinToolBench finds intent mismatch above 50% for every LLM tested, so aggressive tool calling does not mean better answers on financial tasks.
OmniEval: Omnidirectional RAG Evaluation Benchmark for the Financial Domain
OmniEval scores the best RAG systems at 36% numerical accuracy across 5 financial task types, so ledger agents need validation before writing entries.
LLM Anomaly Detection Survey (NAACL 2025): Strong Taxonomy, Absent Tabular Coverage
The NAACL 2025 taxonomy holds, but tabular coverage is absent. Finance AI teams must adapt vision-model methods themselves.
Found in the Middle: Calibrating Positional Attention Bias Improves Long-Context RAG
Subtracting positional bias from LLM attention weights recovers up to 15 points of RAG accuracy when evidence sits mid-context, aiding finance agent pipelines.