FinRAGBench-V: Multimodal RAG with Visual Citations in the Financial Domain
FinRAGBench-V finds top models reach only 20–61% block-level citation recall on financial pages. Multimodal retrieval beats text-only by nearly 50 points.
#finance
Finance
Financial research, analysis, and domain knowledge for accounting AI
LLM Confidence and Calibration: A Survey of What the Research Actually Shows
Verbalized GPT-4 confidence hits only ~62.7% AUROC, barely above chance. Uncertainty-aware finance agents need better calibration than that.
FinTrace: Trajectory-Level Evaluation of LLM Tool Calling for Financial Tasks
FinTrace shows frontier LLMs pick the right financial tools (F1 ~0.9) but score just 3.23/5 on using the results, the step that breaks write-back agents.
OmniEval: Omnidirectional RAG Evaluation Benchmark for the Financial Domain
OmniEval scores the best RAG systems at 36% numerical accuracy across 5 financial task types, so ledger agents need validation before writing entries.
FinDER: Real Analyst Queries Expose a 74% Recall Gap in Financial RAG
FinDER's 5,703 real analyst queries show top RAG recalls only 25.95% of 10-K evidence. Normalize abbreviations first, before swapping embeddings.
Lost in the Middle: Position Bias in LLMs and Its Impact on Finance AI
LLMs score up to 20 points worse when the answer sits mid-context, so finance RAG pipelines should place the best passages first or last.
AnoLLM: Fine-Tuning LLMs for Tabular Anomaly Detection in Financial Data
AnoLLM beats classical baselines on mixed-type fraud data by scoring rows with LLM negative log-likelihood, but adds no edge on purely numerical tables.
DocFinQA: Long-Context Financial Reasoning on Full SEC Filings
DocFinQA swaps FinQA's 700-word passages for full SEC filings, a 175× longer context that nearly halves GPT-4 accuracy on long documents.
TheAgentCompany: Benchmarking LLM Agents on Real-World Enterprise Tasks
TheAgentCompany benchmarks 175 enterprise tasks in a simulated intranet, and the best model, Gemini-2.5-Pro, finishes only 30% at $4 each.
InvestorBench: Qwen2.5-72B tops stocks at 46.15% CR
InvestorBench: Qwen2.5-72B leads stock trading at 46.15% CR; finance-tuned Palmyra-Fin backfires on equities—size beats domain fine-tuning.
Single-Agent: equals MAS on multi-hop at equal tokens
Under equal thinking-token budgets, single-agent LLMs match or beat multi-agent systems on multi-hop reasoning—favor simpler finance agent designs.
M3MAD-Bench: Are Multi-Agent Debates Really Effective Across Domains and Modalities?
M3MAD-Bench finds Collective Delusion drives 65% of multi-agent debate failures, and adversarial debate cuts accuracy by up to 12.8%.