Your site, your rules: new AI traffic options for all customers (opens in new tab)
Cloudflare is replacing its broad “Block AI Bots” approach with finer controls based on what automated systems do: Search, Agent, or Training. The goal is to let website owners preserve discoverability and useful automation while blocking uncompensated model training and other unwanted access. These controls will be available to all Cloudflare customers, including Free-tier users. ## Why AI traffic needs more nuance - The traditional crawler exchange—content in return for referrals—has weakened as AI systems increasingly consume content without sending traffic back. - Website owners previously faced a binary choice: - Allow AI access to remain discoverable. - Block automation and protect content at the risk of losing visibility. - This tradeoff particularly harms small sites and can favor established search providers that use the same crawlers for search and training. ## A behavior-based AI taxonomy Cloudflare will classify automated traffic by its purpose rather than simply labeling bots as “AI”: - **Search** - Collects or indexes content to answer future queries. - Builds a database proactively. - Should generally provide referrals or other fair compensation. - **Agent** - Acts in real time on behalf of a person. - Includes chat-fetch bots such as ChatGPT-User and browser-use agents driven by Gemini or Claude. - Visits a site to complete a specific task for a human. - **Training** - Collects content to train or fine-tune a model. - Permanently incorporates data into the model’s underlying architecture. Bots may have multiple classifications. Cloudflare encourages operators to separate Search, Agent, and Training crawlers so site owners can understand and control their access more effectively. ## New controls for AI traffic - Cloudflare is adding separate controls for Search, Agent, and Training traffic. - These options replace the need for a single all-or-nothing AI blocking decision. - The controls will be available to all customers, including those on the Free plan. - Cloudflare will continue tracking other automated behaviors, such as ad verification, feed fetching, and agentic transactions. ## New default rules Starting September 15, 2026: - For new domains, **Training** and **Agent** crawlers will be blocked by default on pages displaying ads. - **Search** crawlers will remain allowed by default because they are more likely to send visitors back. - The policy treats ads as an indication that human attention—and therefore monetizable traffic—is the intended outcome. - Multi-purpose crawlers will be governed by all of their classifications, using the most restrictive applicable rule. - As a result, crawlers such as Googlebot, Applebot, and BingBot may be blocked when customers choose to block Training traffic. - Website owners can opt out of the new defaults through Cloudflare Security settings before September 15. Cloudflare’s recommendation is to manage AI access by behavior: allow Search when referrals matter, permit Agents when real-time user tasks are valuable, and block Training where content reuse is not adequately compensated.
