Here's a number worth sitting with: as of mid-2026, automated requests make up more than 57% of all HTML traffic on the web — outnumbering actual human visitors for the first time in internet history. A meaningful and fast-growing slice of that traffic isn't Googlebot indexing your site so it can show up in search. It's AI crawlers scraping your blog posts, product pages, and documentation to train models or answer questions in a chatbot, often without a human ever clicking through to your site at all.
If you run a website behind Cloudflare — and a huge share of the small business web does, whether you know it or not — a policy change lands on September 15, 2026 that will quietly reshape who gets to read your content for free, who has to pay, and who gets blocked outright. If you don't check your settings before then, Cloudflare will make the decision for you.
Here's what's actually happening, why it matters to your bottom line, and what to do about it — even if you've never opened a Cloudflare dashboard in your life.
What Cloudflare Actually Changed
Cloudflare sits in front of a large percentage of the internet as a content delivery network and security layer, which puts it in a unique position to see — and control — bot traffic at scale. In 2025, it launched an experimental feature called Pay Per Crawl: site owners could flip a switch and start charging AI companies a per-page fee every time their crawler fetched content, using the obscure HTTP 402 Payment Required status code to tell a bot "pay up, or you don't get in."
That experiment has since evolved into something broader called Pay Per Use, paired with a new Monetization Gateway. The shift in thinking is subtle but important: instead of charging AI companies every time a bot merely fetches a page (which is easy to game and doesn't reflect actual value), the new model is oriented toward charging when your content actually creates value — for example, when it's cited in an AI-generated answer, or when an AI agent pulls specific data to complete a task for its user. Early partners taking this approach include AI search products that pay publishers when their content surfaces in results.
Alongside that shift, Cloudflare is tightening its default behavior for how AI bots are allowed to behave on your site at all. As of September 15, 2026, Cloudflare will sort crawlers into three buckets:
- Search bots — the crawlers that power AI-assisted search results and, generally, still send human visitors to your site. These remain allowed by default because they still drive discovery and traffic.
- Agent bots — crawlers acting on behalf of an AI agent completing a task for a user (booking something, comparing prices, summarizing a page).
- Training bots — crawlers whose entire purpose is hoovering up your content to train a model, with no visit-generating benefit to you at all.
The controversial piece is what Cloudflare calls "mixed-use" crawlers — bots that blend more than one of these purposes, making it hard to tell whether a given request is going to send you a customer or just strip-mine your content for free. Starting September 15, Cloudflare will block mixed-use crawlers by default on any page that carries advertising, unless the site owner has manually changed the setting. This new default applies to brand-new Cloudflare customers, new sites created by existing customers, and — notably — all free-tier accounts, which covers a huge number of small business and personal sites.
Why This Matters Even If You've Never Touched These Settings
It's tempting to file this under "big publisher problem" and move on. It isn't. A few reasons this reaches small business owners directly:
Your defaults are about to change without you doing anything. If your site runs ads (even something as modest as an affiliate link or a display ad network) and you're on Cloudflare's free tier or you spin up a new site after September 15, the new blocking behavior applies automatically. You don't have to opt in to be affected — you have to opt out if you disagree with it.
AI-driven traffic to your site is either growing or shrinking depending on how you're categorized. Recent data on AI crawler behavior shows roughly 20% of verified bot traffic now comes from AI crawlers, plus another 6-7% from AI-search bots — and AI-related crawler requests overall have more than doubled as a share of crawler traffic in a little over a year. Some of that traffic is genuinely valuable (an AI answer engine that cites your page and sends you a reader). Some of it is pure extraction — a training bot copying your content into a dataset that will never send you a single visitor.
Blocking indiscriminately can hurt you too. It's easy to hear "AI is scraping my site for free" and want to block everything. But if you accidentally block the search-oriented bots that power AI answer engines, you can disappear from an increasingly important discovery channel — the same way blocking Googlebot would tank your organic search traffic. The point of the new bucket system is to let you be selective rather than all-or-nothing.
There's now an actual mechanism to get paid. For years, the deal was implicit and one-sided: crawlers took your content, you got nothing unless a human eventually clicked a link. Pay Per Use is the first mainstream, infrastructure-level attempt to change that math — letting a website put a real price tag on AI access to its content, the same way a paywall puts a price on human access.
What to Actually Do Before September 15
You don't need to become a bot-traffic expert overnight. For most small business sites, the right move is a short, deliberate audit rather than either extreme (ignore it entirely, or rush to charge for everything).
1. Confirm whether Cloudflare sits in front of your site
If your developer, agency, or website platform set up hosting or DNS for you, you may not even know Cloudflare is involved. Check your DNS provider or ask whoever manages your site's infrastructure. If you're not on Cloudflare, none of this applies to you directly yet — though expect other CDNs to follow a similar playbook given where the industry is headed.
2. Find your AI bot management settings
In the Cloudflare dashboard, this lives under the bot management / AI Crawl Control area of your zone settings. This is where you'll see the Search / Agent / Training categorization and the controls for how each is treated on your site.
3. Decide deliberately, category by category
- Keep search-oriented AI bots allowed. These are the ones most likely to send you a visitor via an AI-powered answer or search result. Blocking them is usually self-defeating for a business trying to be found online.
- Think hard about agent bots. If you run an e-commerce site or anything with pricing/booking data, AI agents may be comparison-shopping or completing purchases on a customer's behalf. Blocking them might mean losing a sale you'd otherwise get; allowing them might mean giving away your pricing intelligence for free. There's no universal right answer here — it depends on your business model.
- Be more comfortable blocking pure training bots, especially on any page that isn't meant to be a public reference document. They offer you close to zero business benefit, so the "block by default" behavior for ad-carrying pages that Cloudflare is rolling out is a reasonable default for most small sites, not something to reflexively override.
4. Don't rush into charging for access
The Monetization Gateway is aimed at publishers with large, valuable content libraries or proprietary datasets — think large media companies, research databases, or platforms with content nobody else has. For a typical small business site (a service business's blog, a local shop's product pages, a niche newsletter), the honest advice is: not yet. Charging aggressively for AI access can quietly cut you out of the AI-generated answers your future customers are increasingly using instead of a traditional search — the opposite of what you want if visibility is still your priority.
5. Put a reminder on your calendar, not just in your head
Because the new defaults roll out automatically for free-tier and new accounts, and because this is clearly an evolving area (Cloudflare has already moved from "Pay Per Crawl" to "Pay Per Use" inside about a year), treat this as a recurring check-in rather than a one-time fix. Revisit your bot settings every quarter, the same way you'd revisit your ad spend or your pricing.
The Bigger Pattern: AI Traffic Is Becoming a Line Item
Step back, and this is really a story about a new category of business economics showing up for the first time: content as a monetizable input to AI systems, distinct from content as a driver of human traffic. Whether or not you ever flip on Pay Per Use, the underlying shift is real — a portion of the value your website creates is now being consumed by machines instead of people, and infrastructure is finally emerging to put a price on that.
That has a direct implication for how you should be thinking about your books. If you do eventually enable AI content monetization — even a modest amount of Pay Per Use revenue — that's genuine income that needs its own line in your chart of accounts, separate from ad revenue, affiliate income, or product sales. Mixing it into a generic "other income" bucket makes it much harder to answer a simple question a year from now: is this actually worth the operational overhead, or noise? The same logic applies to any new, unfamiliar revenue stream — clean separation now saves a reconciliation headache later, especially once a payment processor like Stripe is involved and you need to match settlement reports against what actually landed in your bank account.
Keep Your Books as Clear as Your Content Strategy
Deciding which bots to let in and which to charge is a business decision — and like any business decision, it's only as good as the records behind it. Beancount.io gives you plain-text, version-controlled accounting, so a brand-new revenue stream like AI content monetization gets its own clean, auditable line from day one instead of getting lost in a spreadsheet. Check the docs to see how easy it is to track a new income source alongside everything else, or explore the Fava dashboard for a visual read on where your revenue is actually coming from. Get started for free and keep your financial records as transparent as the rest of your tech stack.