AI agent crawlers now need permission. Here’s how to get it

AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare announced the change on July 1, and most of the coverage since then has focused on Google. The more useful part is what it asks of everyone building agents, and what it offers them in return.

Cloudflare has replaced its single block-AI-bots switch with three categories. Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents. Training covers crawlers that pull content into a model’s weights. The controls went live on July 1 for every customer, including the free tier.

From September 15, the defaults change. Training and Agent will be blocked on pages that display ads, while Search will stay allowed. The new defaults apply to domains newly onboarding t...