TheAgentCompany: Benchmarking LLM Agents on Real-World Enterprise Tasks
TheAgentCompany benchmarks 175 enterprise tasks in a simulated intranet, and the best model, Gemini-2.5-Pro, finishes only 30% at $4 each.
#automation
Automation
Automation techniques and tools for financial data processing workflows
τ²-bench: Measuring the Cost of Dual-Control in Conversational AI Agents
τ²-bench finds active users with their own tools cut conversational agent success by 18–25 points. Beancount agents sharing write access lose the same way.
WorkArena++: The 93% Gap Between Human and AI Agent Performance on Compositional Enterprise Tasks
GPT-4o solves 2.1% of WorkArena++'s 682 compositional enterprise tasks; humans solve 93.9%, showing why agents stall on implicit-goal accounting work.
GAIA Benchmark: Measuring What Frontier AI Agents Can Actually Do
GAIA's 466 tasks show frontier AI agents at 74.55% versus 92% for humans, with Level 3 coordination still the hardest gap to close.
OSWorld: Desktop AI Agents Succeed on 12% of Tasks Where Humans Succeed on 72%
OSWorld finds desktop AI agents succeed on 12.24% of real tasks versus 72.36% for humans, with 75% of failures from visuomotor grounding, not reasoning.
WebArena: The 812-Task Benchmark That Measures What Web Agents Actually Can and Cannot Do
GPT-4 finishes just 14.41% of WebArena's 812 web tasks versus 78.24% for humans; agents mostly fail by falsely declaring a task impossible.
WorkArena: How LLM Web Agents Perform on Real Enterprise Knowledge Work
WorkArena's 33 ServiceNow tasks show GPT-4o at 42.7% overall but 0% on list filters, a wall structured UI interaction must clear.
τ-bench: Measuring AI Agent Reliability in Real-World Tool-Use Domains
τ-bench finds top LLMs fall from pass@1 0.692 to pass@4 0.462 on retail tool-use tasks. Write-back agents on a ledger face the same consistency cliff.
TAPAS: Weakly Supervised Table QA Without SQL, and What It Means for Beancount
TAPAS answers table questions by selecting cells, never generating SQL. It fits small Beancount ledger queries but breaks down at scale.
MAC-SQL: Multi-Agent Collaborative Text-to-SQL
MAC-SQL's three-agent design hits 59.59% execution accuracy on BIRD, with the Refiner adding +4.63 points — a template for generating Beancount ledger queries.
Verifiably Safe Tool Use for LLM Agents: STPA Meets MCP
STPA plus capability-enhanced MCP yields formal safety specs for LLM tool use, with Alloy proving no unsafe flows in a calendar case study.
Single-Agent: equals MAS on multi-hop at equal tokens
Under equal thinking-token budgets, single-agent LLMs match or beat multi-agent systems on multi-hop reasoning—favor simpler finance agent designs.