You type your tax question into ChatGPT, and the answer comes back instantly, confidently, and formatted like it belongs in an IRS publication. It sounds right. The problem is that "sounds right" and "is right" are not the same thing, and when NerdWallet ran a structured head-to-head test of ChatGPT, Gemini, and Perplexity against real tax scenarios in 2026, the gap between the two showed up fast.
For a small business owner deciding whether to lean on a chatbot for tax questions this filing season, that gap matters. Here's what the testing actually found, where AI holds up, where it quietly falls apart, and how to use it without letting a hallucinated answer cost you money.
The Test: Straightforward Questions vs. Personalized Advice
NerdWallet's research team built 63 chat transcripts by asking ChatGPT, Gemini, and Perplexity seven questions each. Three of the questions came straight from IRS enrolled agent practice exams — the kind of question with one objectively correct answer. The other four were open-ended scenarios built around fictional tax filers, designed to test how well each chatbot could personalize advice to a specific financial situation.
The split in results was stark. On the practice-exam questions, all three chatbots performed well, nearly acing the multiple-choice-style material. That's the kind of question AI is good at: a bounded fact with a documented answer, phrased clearly, with no ambiguity about which tax year or which filer's situation applies.
The open-ended, personalized questions were a different story. That's where the chatbots started making the kind of mistakes that would matter to an actual business owner:
- Incorrect standard deduction amounts — off by nearly $3,000 in one test case
- False claims about electric vehicle credit eligibility for a filer who didn't qualify
- Wrong state filing recommendations, including confusing a filer's college state with their work state
- Inconsistent answers — the same fictional profile got different tax software recommendations depending on which chatbot (or which run) answered
- Fabricated biographical assumptions pulled from earlier chat history that weren't actually true
None of these mistakes came with a hedge or a warning. As the researchers put it, the answers "can sound right" purely because of how confidently they're delivered — the tone doesn't change whether the underlying number is correct.
This Isn't a One-Off Finding
NerdWallet's results line up with a broader pattern researchers have been documenting all year. A Loyola University Chicago study found chatbots answered a simple tax question incorrectly two-thirds of the time, and state-of-the-art models correctly calculated less than a third of federal income tax returns even on simplified test cases. Tax professionals have separately flagged AI mistakes on K-1 and 1099 income, on state rules that diverge from federal ones, and on 2026-specific provisions tied to recent tax law changes — exactly the kind of nuance a small business owner is likely to be asking about.
The financial fallout is already showing up in practice. In a survey of UK accountants and bookkeepers, half of the firms said they'd seen a client suffer a direct financial loss — an overpayment, a missed deduction, a penalty — traceable to AI-generated tax or financial advice. That's not a hypothetical risk; it's an ongoing cost some small businesses are already absorbing.
Why AI Struggles Specifically With Tax Questions
Tax advice is a bad match for how large language models generate answers. A few reasons this keeps tripping up even frontier models:
Tax rules are jurisdiction-stacked. Federal, state, and sometimes local rules interact, and a chatbot trained mostly on federal-level content tends to default to federal answers even when the question depends on state specifics — which is exactly how the "wrong state" errors happened in testing.
The correct answer depends on facts the model doesn't reliably track. A single-filer's deduction, credit eligibility, or safe filing state depends on details spread across a conversation. The moment the model has to hold several facts about you in mind at once — instead of retrieving one clean fact from training data — accuracy drops.
Confidence isn't calibrated to correctness. These models are built to produce fluent, complete-sounding answers, not to flag uncertainty. A wrong answer about your standard deduction reads exactly as authoritative as a correct one.
Strategic advice compounds risk. A wrong answer to "what's the mileage rate" is easy to catch. A wrong answer to a multi-year tax strategy question might not surface as a problem until an audit two or three years later — by which point the fix is much more expensive than a quick correction would have been.
How to Actually Use AI for Tax Questions Without Getting Burned
None of this means AI chatbots are useless for taxes — it means they're useful for a narrower job than a lot of users assume. A reasonable, safer workflow looks like this:
-
Use AI to get oriented, not to get a final number. Ask it to explain a concept, summarize what a form is for, or draft a list of questions to bring to your accountant. Don't ask it to compute your actual liability or tell you which credits you qualify for and stop there.
-
Match the question to the tool. A factual lookup ("what's the deadline for Form 941") is low-risk. A personalized judgment call ("should I elect S-corp status this year") is exactly the category where testing shows chatbots go wrong most often — that's a question for a professional or, at minimum, a cross-check against the actual IRS guidance.
-
Know your safety net. If you're using AI-assisted output inside tax software, the software's own error-checking can catch some mistakes before filing. If you're taking AI advice directly with no second check, you have no safety net at all.
-
Check which model and version answered you. Accuracy and behavior shift between model versions, and a screenshot from a friend using a different chatbot (or an older version of the same one) isn't a guarantee your result will match.
-
Verify anything state-specific or credit-related against the source. These were the two most common failure categories in testing. When a chatbot cites a specific dollar threshold, deduction amount, or credit eligibility rule, that's the moment to open IRS.gov or your state's tax authority site and confirm it directly.
The safest framing: AI can save you the hours you'd otherwise spend hunting through IRS publications to understand a concept or prepare good questions. It shouldn't be the last step before you file.
Keep Your Financial Records Clean, Whatever Tools You Use
The tax questions that trip up AI chatbots the most — deduction eligibility, state filing nuances, entity-specific rules — get much easier to answer correctly when your underlying books are accurate and well-organized in the first place. That's true whether you're asking a chatbot, a CPA, or doing it yourself. Beancount.io offers plain-text accounting that gives you full transparency into your financial data — version-controlled, auditable, and easy to hand to whichever advisor (human or AI) you're consulting this tax season. Get started for free and keep your numbers trustworthy from the source up.