Saw the Mercor writeup yesterday. They gave 12 licensed CPAs four simplified month-end close scenarios from APEX-Accounting: dig through the company files, find the right numbers, do the math, hand back a table. The humans averaged about 37% of the rubric. Opus 5 got full marks on all 20 attempts, each one under ten minutes.
I do books for nine small clients, all in beancount now. First reaction was a little queasy!
But I don’t know what to do with this. Is anyone actually changing their workflow because of numbers like these, or is it just another benchmark?
Opus 5 got full marks on all 20 attempts
on the simplified versions. Mercor themselves say no model fully solves ~60% of the real APEX tasks. Nothing to change yet.
Full benchmark paper if anyone wants the task breakdown: [2607.27189] APEX-Accounting
Best model scores about 56% on the full set.
@helpful_veteran I’d push back on “nothing to change.” The simplified tasks are exactly the boring part of my close. Find the number in the right file, do the math, put it in the schedule. That’s what eats my second week every month.
Last month I tried it on our own books. Gave a model the ledger plus the bank and vendor PDFs and asked it to draft the accrual and prepaid amortization entries. I reviewed every one, fixed a couple of account names, didn’t find a bad amount. Review took about 40 minutes vs. most of a day writing them myself.
What made me comfortable wasn’t trusting the model. It’s that the ledger is text, so the review is a git diff. I’m reading exactly what changed, line by line, then bean-check. If that same output was sitting in some app’s “AI suggested” queue I wouldn’t have approved any of it.
Small thing nobody’s mentioned. Mercor sells expert training data to the AI labs, and the benchmark partner, Ramp, sells accounting software. I’m not saying the numbers are wrong, only that everyone who produced them has a reason to like this headline.
@finance_fred wait how do you check the amounts tho. recompute each one by hand?
@newbie_accountant nah I don’t recompute every line. I tie the totals back to the PDFs, hand-check only the big or weird ones, then rely on git diff plus bean-check for the rest. If totals tie and the diff looks clean, I sign off.