For eight years, dozens of small businesses in South Florida did everything right. They hired a payroll company, paid their invoices on time, and trusted that the taxes withheld from every employee's paycheck were being sent to the IRS. They had, in the language of the tax code, "outsourced" their payroll.
What they hadn't outsourced was the liability.
Matthew Brown, owner of the payroll services company Elite Payroll, was sentenced to more than four years in federal prison after admitting he collected the full amount his clients owed in payroll taxes — federal income tax withholding, Social Security, and Medicare — and then simply didn't send most of it to the IRS. Between 2014 and 2022, Brown filed false employment tax returns that understated what his clients actually owed, pocketed the difference, and used the money to buy a multimillion-dollar home, a yacht, a private jet, and a garage full of Ferraris, Porsches, and Rolls-Royces. By the time investigators caught up with him, the scheme had cost the IRS more than $22 million.
The part that should worry every small business owner isn't the yacht. It's what happens next to Elite Payroll's clients — because outsourcing your payroll to a third party does not outsource your responsibility for making sure that money actually reaches the IRS.
Why "I Paid My Payroll Company" Isn't a Legal Defense
Every paycheck you issue includes money that was never really yours to begin with. The income tax, Social Security, and Medicare amounts withheld from an employee's wages are, by law, held "in trust" for the government from the moment they're withheld until they're deposited. That's why the IRS calls unpaid withholding a trust fund issue, not an ordinary debt.
When a business fails to remit that money — whether because the owner spent it on payroll gaps, or because a payroll provider stole it — the IRS has a specific tool for collecting it: the Trust Fund Recovery Penalty (TFRP), under Internal Revenue Code Section 6672.
The TFRP is not a fine on the business. It's a penalty assessed directly against individuals — anyone the IRS determines was a "responsible person" who acted "willfully" in failing to ensure the taxes were paid. A few things make it uniquely dangerous for small business owners:
- It equals 100% of the unpaid trust fund taxes. There's no partial liability; the IRS can pursue the full amount.
- It pierces the corporate veil. Your LLC or S-corp structure, which normally shields your personal assets from business debts, offers no protection here. The TFRP attaches to you personally.
- It's joint and several. If more than one person at your company had authority over payroll decisions, the IRS can collect the entire penalty from any one of them — it doesn't have to split it evenly, or at all.
- It survives bankruptcy. Unlike most business debts, the TFRP generally cannot be discharged.
- "Willful" has a low bar. You don't need to have intended fraud. Simply knowing taxes were unpaid and choosing to pay other creditors first — including, in some interpretations, continuing to pay a payroll vendor whose remittances you never verified — can be enough.
In other words: if your payroll company keeps the withholding instead of remitting it, the IRS's first move is often to figure out who at your company qualifies as a "responsible person" — and Elite Payroll's clients are not an isolated case. Similar prosecutions hit business owners in Texas, Maryland, and elsewhere in the same stretch of years, each involving withheld taxes that were collected from employees' paychecks and never made it to the Treasury.
How the Scheme Actually Worked
What made Brown's fraud persist for eight years wasn't a single dramatic theft — it was a quiet, repeatable gap between what clients were told and what the IRS was told.
Elite Payroll billed clients for their full, accurate tax liability. Nothing about the invoice looked wrong. But when Brown filed the quarterly Form 941 employment tax returns with the IRS, he reported a smaller liability than what he'd actually collected — then kept the spread. Because the false returns were internally consistent (understated liability matched understated payments), there was no obvious mismatch for a client glancing at their own books to catch, unless they compared what they'd been billed against what the IRS actually received.
That's the structural weakness in outsourcing payroll: you can verify that you paid your vendor. You generally cannot, without extra effort, verify that your vendor paid the IRS.
Four Ways to Verify Your Payroll Provider Is Actually Remitting
The IRS's own guidance on outsourcing payroll duties is blunt: "the employer is ultimately responsible" for the income tax withheld and for both the employer and employee shares of Social Security and Medicare tax, no matter who physically submits the payment. That responsibility comes with a few concrete tools to protect yourself.
1. Enroll in EFTPS yourself — don't rely on your provider's portal. The Electronic Federal Tax Payment System (EFTPS) is the IRS's own payment platform, and any employer can register for their own login, separate from whatever access a payroll company grants you. Once enrolled, you can see up to 16 months of payment history showing exactly what's been deposited on your behalf, with confirmation numbers you can independently check. If your provider has been submitting deposits, this view will match your payroll register. If it doesn't, you'll know within a quarter instead of within eight years.
2. Check whether your provider is a CPEO — and understand what that does and doesn't change. A Certified Professional Employer Organization (CPEO) is a designation the IRS grants after a rigorous application, bonding, and audit process. Critically, when a CPEO handles your payroll under a proper service agreement, the CPEO — not you — becomes solely liable for the federal employment taxes on wages it pays. The IRS publishes a public list of certified CPEOs; if your provider claims the designation, verify it against that list rather than taking their word for it. Note the tradeoff: CPEO customers can't see federal deposits through EFTPS the way EFTPS-only clients can, so ask directly what reporting and confirmation your CPEO agreement provides.
3. Reconcile your own withholding against filed returns, at least quarterly. You already know how much you withheld from every paycheck — it's on your own payroll register or ledger. Compare that total against what your provider reports on each quarter's Form 941. A provider that's skimming will show a mismatch here, exactly as Elite Payroll's clients would have if they'd compared invoiced amounts to filed liabilities rather than assuming the two matched.
4. Treat payroll-tax problems as a red flag worth escalating immediately, not a billing hiccup. Late notices from the IRS, unexplained penalty letters, or a provider that goes quiet when you ask for deposit confirmations are not administrative noise — they're often the first external sign that withheld money isn't reaching the government. Don't wait for a second notice to start asking pointed questions or moving to a new provider.
Why Your Own Records Are the Real Safety Net
The clients who eventually noticed something was wrong with Elite Payroll didn't catch it from a bank statement — they caught it by comparing what they'd been told against what actually happened. That's only possible if your own financial records are detailed and independently verifiable, not just a mirror of whatever numbers your payroll vendor hands you.
This is exactly where plain-text, version-controlled bookkeeping earns its keep. When your general ledger lives in a transparent, auditable format rather than being locked inside a vendor's dashboard, reconciling "what I was billed" against "what was actually remitted" is a straightforward comparison you control — not a favor you have to ask your payroll company for. Beancount.io gives you exactly that: plain-text accounting with a full history of every entry, so you can catch a mismatch on your own timeline instead of the IRS's. Get started for free and keep your books — and your liability — under your own control.