Pedro Dias@pedrodias
Cloudflare just reshaped how AI crawlers get classified — and SEO teams need to pay attention before Sept 15.
On its second "Content Independence Day," Cloudflare moved away from the blunt "block all AI bots" model toward a behaviour-based taxonomy. Instead of asking "is this AI?", the question is now what is the bot doing, what does it store, and how does it reshare your content?
The three use cases every site owner can now control (yes, including the Free tier):
☛ Search: indexing your content to answer queries later; the behavior that drives referral traffic back to you. Allowed by default.
☛ Agent: real-time bots acting on a human's behalf (ChatGPT-User, Gemini/Claude driving Chrome).
☛ Training: crawling to permanently absorb your content into a model.
The change with the biggest SEO implications ☛ new defaults land September 15, 2026.
On pages that display ads, Training and Agent will be blocked by default while Search stays allowed (for new domains onboarding). More importantly: multi-purpose crawlers will now be blocked according to all their behaviors. That means if you block Training, combined crawlers like Googlebot, Applebot, and Bingbot get blocked too — because they crawl for both Search and Training.
You can opt out in Security settings before Sept 15 to protect your search visibility. Don't sleep on this one.
For AI-visibility work:
☛ A new use signal extends Content Signals in robots.txt — use=immediate (store nothing), use=reference (index, excerpt, link back — the new default), or use=full (summarize and reproduce). These are stated preferences, not hard blocks.
☛ Bots that reproduce content in full can't be Verified. Abusing signals costs a bot its Verified status.
☛ "Verified" no longer means "allowed by default" — it now just makes a bot allowable within its category.
Also worth knowing:
☛ BotBase — a new searchable directory (Enterprise Bot Management) showing every tracked bot, its classification, and a copyable detection ID for security rules.
☛ Transitive trust — an experiment using the RFC 7239 Forwarded header (e.g. Forwarded: for="openai";use="reference") so operator-level trust decisions hold even through layers of intermediaries.
The through-line: control is getting granular, and the era of "search and training are the same crawler" is ending. If you manage crawl access, audit your Training rules now so you don't accidentally deindex yourself in September.