Making AI search smarter (opens in new tab)
Cloudflare argues that AI-powered search has broken the traditional bargain in which publishers exchanged crawl access for traffic and revenue. AI summaries increasingly answer users’ questions without sending them to source sites, leaving publishers to choose between reduced visibility and uncompensated content use. The company proposes rebuilding this relationship through smarter crawling and payments tied to actual content usage. ## Rebuilding the Bargain - Cloudflare’s responsible AI bot principles emphasize: - Transparency about a bot’s identity and purpose - Respect for site owners’ choices - Good-faith behavior - Blocking unwanted crawlers protects publishers but does not create a sustainable business model. - Cloudflare’s broader goal is to make AI search beneficial to users, AI companies, and content creators. ## Making AI Search Smarter - Cloudflare is launching a research program using signals from its global network, which covers more than 20% of the web. - These signals may identify: - Which pages are fresh or have genuinely changed - Which content attracts human and automated traffic - Which sources are high quality and relevant - Answer engines could use this information to surface better content and avoid repeatedly crawling unchanged pages. - More than 50% of traffic from legitimate crawlers reportedly goes toward re-fetching unchanged pages. - Reducing unnecessary crawls would lower: - AI companies’ compute costs - Publishers’ server load and bandwidth expenses - The program is intended to be neutral, limited to search, and will not share content or train foundation models. - Cloudflare plans to publish results and make the capability broadly available later in the year. ## From Pay Per Crawl to Pay Per Use - Cloudflare’s existing Pay Per Crawl model lets publishers charge AI companies for accessing their content. - Cloudflare says crawling is an imperfect measure of value: - A page may be crawled once but cited in thousands of answers. - It may also be crawled repeatedly without ever being used. - The company is therefore experimenting with Pay Per Use, where compensation reflects how often content contributes to search results or answers. - Early partners include Ceramic.ai and You.com. - Ceramic’s pay-per-query model pays publishers when their content appears in Ceramic search results. - Cloudflare’s network is intended to help AI companies scale these payment systems across millions of participating content owners. Cloudflare’s proposed model combines efficient, change-aware crawling with compensation based on actual content use. If adopted broadly, it could give publishers better control, reduce needless infrastructure costs, and create a more sustainable economic relationship between AI search services and the web.