Estimated read time: 7 minutes
On September 15, Cloudflare changed a default setting on a large slice of the internet, and most of the business owners affected have no idea it happened. If your website sits behind Cloudflare on a free plan, and your pages carry ads or affiliate links, a category of AI crawler that used to reach your content is now being turned away at the door. Nobody emailed you. Nobody asked. The switch flipped.
This matters more than a normal settings change, because the crawlers in question are not only training bots. Some of them are the same bots that decide whether your business shows up when somebody asks ChatGPT or Perplexity for a recommendation. Blocking them protects your content from being harvested. It can also quietly remove you from the answer.
Table of Contents
What actually changed on September 15
Cloudflare announced the policy back on July 1 and gave the internet about ten weeks of warning, which in practice meant the publishers who read Cloudflare blog posts got warned and everybody else did not. As of September 15, the default behavior for qualifying sites is to block what Cloudflare calls mixed-use crawlers from pages that carry advertising.
The logic is straightforward once you say it out loud. If a page makes its money from ad impressions, and an AI system reads that page, extracts the answer, and hands the answer to a user who never clicks through, the page just did unpaid labor. Cloudflare decided that the default answer to that arrangement should be no rather than yes. Publishers who want the old behavior are free to have it. They just have to go turn it back on themselves.
Alongside the block, Cloudflare converted its Pay Per Crawl experiment into a broader Pay Per Use marketplace, with Ceramic.ai and You.com signed on as early participants. The pitch is that AI companies pay for access instead of taking it. Whether that ever produces meaningful revenue for a site doing a few thousand sessions a month is a separate question, and the honest answer is covered further down.
Who got flipped without asking for it
Three groups are affected by the new default. New Cloudflare customers get it. New sites added by existing customers get it. And every existing free-plan customer gets it, which is the group that contains almost every small business, solo consultant, local service company and independent publisher using Cloudflare today.
That last category is the one worth sitting with. Paid Cloudflare customers keep whatever configuration they already chose. Free customers, who by definition are the least likely to be monitoring bot policy, were changed for them. It is not malicious. Cloudflare is arguably doing free users a favor by protecting content they never thought to protect. But a default applied to people who did not opt into it is still a default those people need to go look at.
If you have never logged into a Cloudflare dashboard, and you are not certain whether your host put your site behind Cloudflare without telling you, that is extremely common. Plenty of managed WordPress hosts route traffic through Cloudflare as part of the stack. Checking takes a minute and is worth doing before you assume this does not apply to you. Our guide to small business web hosting covers which hosts bundle it in.
Why mixed-use is the whole fight
Here is the part that gets lost in the headlines. Most AI companies now run at least two separate crawlers with two separate names. One collects content to train future models. The other fetches pages in real time when a user asks a question, so the assistant can cite a current source. OpenAI runs GPTBot for training and OAI-SearchBot for search. Anthropic runs ClaudeBot and Claude-SearchBot. Google separates Google-Extended from its ordinary search crawler.
When a company splits its bots like that, you can make a clean decision: block the training bot, allow the search bot, keep your visibility while keeping your content out of the next model. That is the setup most publishers actually want.
A mixed-use crawler refuses to make that distinction. It runs one user agent that does all three jobs at once, indexing for search, fetching for agents, and collecting for training. You cannot separate the part you want from the part you do not. So Cloudflare treats the whole thing as a training crawler and blocks it under a training block. The crawler operator could fix this tomorrow by publishing separate tokens. Some have. The ones that have not are the ones getting blocked, and that is by design. The block is pressure, not an accident.
What it costs you if buyers ask AI first
For a publisher whose entire model is ad impressions, blocking is close to free. You were not getting paid by the AI traffic anyway.
For a small business whose website exists to get found and sell something, the math is different and worse. If a prospective customer types a question into an assistant and the assistant needs to fetch live pages to answer it, a blocked crawler means your page is not in the candidate set. Your competitor’s page, sitting on a host that did not flip the switch, is. You do not get an error message. You just do not get mentioned.
This is the same structural shift we covered in what AI search is doing to small business traffic. The volume of people who ask an assistant before they ask Google keeps climbing, and the sites that get cited in those answers are increasingly the ones that stay technically reachable. Being invisible to an answer engine is a different problem from ranking poorly, because there is no page two to climb out of.
So the right decision genuinely depends on what your site is for. Content site monetized by ads: the block probably helps you. Service business trying to get discovered: the block probably hurts you. Both on the same domain: you have a real tradeoff to think through rather than a default to accept.
How to check your own setting in four minutes
You do not need a developer for this. Log into Cloudflare, select your domain, and go to Security, then Bots, then AI Crawl Control. You will see a list of every AI crawler that has actually hit your site, with request counts and an individual toggle for each one.
The request counts are the useful part, and most owners find them surprising. You will likely see names you recognize, GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bytespider, CCBot, Applebot, and some you do not. Look at which ones are driving volume before you decide anything.
The configuration most small businesses land on is to block the pure training crawlers, GPTBot, CCBot, Google-Extended and Applebot-Extended, while allowing the search and retrieval crawlers, OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. That keeps you findable in AI answers without donating your archive to the next training run. Cloudflare exposes this as presets, so you can set Training to Block and Search to Allow rather than toggling fourteen individual rows.
Whatever you choose, write down the date you chose it. Defaults change, as September 15 just demonstrated, and a setting you cannot remember making is a setting you will not think to re-check.
The honest case for leaving it blocked
There is a real argument for doing nothing, and it deserves more than a sentence.
If your traffic from AI assistants is currently negligible, and your content represents years of accumulated expertise that is genuinely hard to replace, then letting a mixed-use crawler take all of it in exchange for a possible future citation is a bad trade. You are giving away the durable asset for a lottery ticket. Cloudflare is betting that enough publishers reach that conclusion to force crawler operators to separate their bots, and that bet is not obviously wrong.
There is also a middle path that costs nothing: block the mixed-use crawlers, allow the well-behaved ones that publish separate search tokens, and let the market sort out the rest. The operators who split their bots get access. The ones who refuse do not. You lose visibility only in the systems that would not give you a clean choice, which is a fairly defensible place to draw a line.
Pay Per Use and whether it applies to you
Cloudflare’s marketplace lets participating AI companies pay for crawl access rather than being blocked. It is a genuinely interesting structural idea and it is almost certainly not a revenue line for a small site in 2026.
Two early partners does not make a market. The buyers who matter most, the large model labs, are not the ones signed up. And per-crawl pricing on a site doing modest traffic produces numbers that round to nothing after you account for the time spent configuring it. Treat Pay Per Use as a thing to watch rather than a thing to plan around. If it grows into a real clearinghouse over the next year, it will still be there when you have the traffic to make it worth the setup.
The immediate action is the free one: go look at your AI Crawl Control panel and make the setting deliberate. Whether you end up blocking more or blocking less, the important change is that you decided it instead of inheriting it.
FAQ
Does this block Googlebot or hurt my normal search rankings?
No. Ordinary Googlebot is not an AI crawler in this context and is not covered by the change. Google-Extended, which governs use of your content for Gemini training, is separate and can be blocked without affecting how you rank in regular Google search.
How do I know if my site is even on Cloudflare?
Check your domain registrar or DNS settings for nameservers ending in cloudflare.com, or ask your host directly. Many managed WordPress hosts include Cloudflare in their stack without making it obvious in your dashboard.
Does robots.txt do the same thing?
Not really. Robots.txt is a request that well-behaved crawlers choose to honor. A Cloudflare block happens at the network edge and is enforced whether the crawler cooperates or not. Using both is reasonable. Relying on robots.txt alone is not.
If I unblock everything, does my content get used for training?
Yes, for any crawler whose user agent includes training among its purposes. That is exactly the bundling the mixed-use designation exists to flag. Unblocking is a decision to accept training use along with search visibility.
I run ads and also sell services. What should I do?
Split the decision by purpose rather than by domain. Block training crawlers everywhere, and allow the retrieval crawlers that publish a separate search token so your service pages stay reachable in AI answers. That combination protects the archive and keeps the sales path open.
Related Coverage
- What AI Search Is Doing to Small Business Traffic: the demand-side version of the problem this block creates.
- How Google AI Search Changed in 2026: why being cited now matters more than ranking.
- What Is Schema Markup: the other technical lever that decides whether machines can read your pages.
Faceted Media Magazine covers business, AI, and entrepreneurship for the people building what’s next.