Cloudflare has rolled out a major change to how it handles automated web traffic, and the update carries a side effect that could catch some site owners off guard. Depending on how certain settings are configured, the change could result in a site owner unintentionally blocking Googlebot along with AI bots. The company announced this change as part of its second Content Independence Day and rolled it out for all customers, including those on the free plan.
The core idea behind the update is simple. Cloudflare has replaced the option to treat “AI bots” as a single unified category for either blocking or allowing all crawlers in this category. With the new update, the company now lets customers sort crawlers based on what they actually do on the site rather than by label. That distinction matters even more once a separate set of default settings takes effect on September 15.
Sorting Crawlers Into Three Buckets
There are three categories of AI-related crawlers that the new system uses. The first is Search. These are crawlers that index a site so they can answer questions about it later. Cloudflare ties this activity to referral traffic.
Then, there are Agent crawlers. These are real-time crawlers that act on behalf of the person making the request, such as ChatGPT-User. Unlike typical crawlers, which run on their own, Agent crawlers are triggered by an actual person.
Finally, there are Training crawlers which crawl the content of the website in order to train or fine-tune an AI. This is the type of crawler activity that has worried publishers the most, since Training crawlers send no traffic back to the original site.
According to Cloudflare, it intends to persuade bot operators to operate their bots in separate modes for each of these actions. That way, websites can see exactly why a bot is visiting instead of grouping every bot under a single “AI” label.
The September 15 Deadline That Changes Things
There are two major changes which will come into force on September 15, 2023 and which require closer attention if the site uses Cloudflare services.
For new customers and new sites from existing customers, Training and Agent crawlers will be blocked by default on pages with ads, while Search crawlers remain allowed. Cloudflare also mentioned that the customers who use its free services and have not changed the settings before September 15 will get these default settings automatically.
The second change deserves even closer attention. Starting September 15, Cloudflare will treat crawlers that serve multiple purposes based on their overall behavior, applying whichever rule is strictest. In other words, if a crawler performs both Search and Training functions, and the site owner chooses to block Training, the whole crawler will be blocked, regardless of the Search function.
Cloudflare points to Googlebot, Applebot, and Bingbot as examples, since each one crawls the web for both regular search indexing and AI training. Sites that have enabled the old option “Block AI bots” will also be included in this change.
In case a site owner wishes to allow Googlebot, Applebot or Bingbot, the fix is quite easy – one may review the settings in Cloudflare dashboard and adjust them. Cloudflare also announced that it will notify its customers about this upcoming deadline.
A New Signal for How Bots Can Use Content
Besides the new categories of crawlers, Cloudflare has also started a test of a new content-use signal based on its existing Content Signals framework used in robots.txt files. There are three types of signals, the most restrictive one is “immediate” – it means that the content should not be stored at all. The next one is “reference” – the crawler is allowed to index the content and create a reference for it, and it is the new default option. Finally, there is a “full” signal which allows a crawler to summarize and reproduce the content.
Cloudflare has made it clear that these are preferences rather than enforced restrictions, so a bot could still choose to ignore the signal and store content even when the value is set to “immediate.” Cloudflare has also changed the criteria for a bot being labeled as “verified”. Now it depends on the category of a bot, so it does not guarantee the access anymore. Reproducing content in full is no longer the criterion for a bot to get verified. Additionally, Cloudflare has introduced a searchable directory for Enterprise Bot Management customers named BotBase which contains how each bot is classified together with detection ID that can be used to create security rules.
The Numbers Behind the Decision
Cloudflare mentioned the numbers behind this update as a part of a report celebrating the first anniversary of the first Content Independence Day. According to that report, AI training crawlers have become the majority of crawling requests in Cloudflare network traffic now, making a sharp increase from around 20 percent of total requests back in spring 2025. Daily requests from AI agents have increased by more than 1,700 percent during that time.
One should keep in mind that this data reflects only Cloudflare network traffic and does not include crawler activity worldwide. Nevertheless, the numbers give a clear explanation why Cloudflare decided to implement such a detailed mechanism instead of one on-off switch.
Why Site Owners Should Pay Attention
The problem with the changes to be made on September 15 is linked to the fact that now the blocking of AI training blocks will be linked to search crawling as well. In other words, by blocking the training layer of crawlers to protect the content from AI models, site owners may accidentally block Googlebot because the latter belongs to the same category of combined crawlers.
Such a measure is stronger than blocking through the robots.txt file. A robots.txt instruction is a mere request and sometimes Google crawls the pages even when robots.txt asks the crawler not to. A Cloudflare-level block is more difficult to bypass because it works on the network level. By blocking the access of Googlebot to a certain site, site owners make sure that Google will not crawl that particular site and the pages there will be less visible in the search results.
It is not the first case like that. During the last year, the number of publishers switching to default-deny configuration, which involves the blocking of both AI training and retrieval crawlers, has significantly increased. Each time this happens, the trade-off is the same, protecting content from AI models can come at the cost of search visibility.
What To Do Before September 15
All site owners who use Cloudflare need to take some measures before September 15 and check their crawler settings in the dashboard. The major question that needs to be answered is whether Search crawlers should remain allowed while restricting training and agent crawlers.
This applies especially to those who turned on the AI bot block in the past and haven’t revisited it since. The new combined-crawler rule may result in blocking Googlebot instead of AI bots. Free-tier users will simply be switched to the new defaults after the deadline. As it was stated by Cloudflare, the company expects mixed-purpose crawler operators to separate the bots by the purpose they serve over the next year. If it happens, site owners will have a chance to choose between protecting the content from AI crawlers and keeping the site visible in search results. Otherwise, they will keep being connected as now.
FAQs
Q 1: What is Cloudflare changing about AI crawlers?
Cloudflare now allows sites to classify AI-related crawlers in three groups, Search, Agent, and Training. The site owner can control each group separately.
Q 2: Why could this block Googlebot?
Beginning from September 15, Cloudflare will apply the strictest rule to crawlers which have more than one purpose. Since Googlebot crawls for both search and training purposes, the blocking of the training group may also result in blocking Googlebot.
Q 3: When does this change take effect?
The changes and the new default settings take effect beginning from September 15. Site owners can check their settings any time before that date.
Q 4: Does this affect free Cloudflare accounts?
Yes. Free-tier users whose settings were not changed before September 15 will be automatically switched to the new default rules.
Q 5: What is the difference between Search, Agent, and Training crawlers?
Search crawlers index content and refer the traffic back to a site. Agent crawlers act in real-time for the sake of one particular person. Training crawlers gather content to train AI models and do not refer the traffic back.
Q 6: What is the new content-use signal Cloudflare is testing?
It is a signal that Cloudflare adds to robots.txt with three settings – immediate, reference, and full. It tells bots how they are allowed to use a page’s content. According to Cloudflare, it is merely a preference and does not block access.
Q 7: What happened to Cloudflare’s “Verified” bot label?
Now verification does not guarantee full access everywhere. It depends on the bot’s category and now crawlers that reproduce entire pages cannot be verified.
Q 8: How much has AI crawler traffic grown?
According to Cloudflare’s report, the AI training became a major type of requests made to Cloudflare, having grown from 20% of the requests in spring 2025 and increasing the daily number of requests of AI agents 1,700% during the year.
Q 9: Should I turn off “Block AI bots” if I still want Googlebot to crawl my site?
If you want to keep the full access of Googlebot, then it is worth considering that before September 15.
Q 10: What is BotBase?
It is a searchable directory that Cloudflare created for Enterprise Bot Management customers. It shows how each tracked bot is classified and provides the detection ID to create security rules.
