cloudflareJul 1, 2026

Unmasking the crawls with Attribution Business Insights (opens in new tab)

Cloudflare argues that the traditional exchange between crawlers and publishers has broken down as AI bots extract content without sending meaningful referral traffic. This creates lost revenue for publishers while increasing hosting costs, making granular traffic attribution essential. Its new Attribution Business Insights dashboard aims to help site owners identify which bots provide value and make informed decisions about access, blocking, and commercial relationships. ## The Internet’s Changing Economics - Traditional search engines generally crawled content a few times for each visitor they referred. - That crawl-to-referral balance supported advertising, affiliate revenue, subscriptions, and direct audience relationships. - AI crawlers increasingly create a “zero-click” ecosystem by summarizing content without directing users to the original publisher. - Cloudflare observed AI crawl-to-referral ratios ranging from 118:1 to nearly 50,000:1. - Publishers face both reduced traffic-based revenue and higher infrastructure costs from unproductive automated access. ## Attribution Business Insights Dashboard - The dashboard is available to Cloudflare Bot Management customers. - It provides an immediate view of bot activity without requiring extensive manual analytics filtering. - It measures: - Human versus bot traffic to content pages. - Overall and operator-specific crawl-to-referral ratios. - Crawl-to-referral trends over 24 hours, seven days, or 30 days. - Top bots by traffic volume, country, bandwidth usage, and current allow/block status. - AI crawlers are classified by behavior: - **Training:** collecting data for future large language models. - **Search:** refreshing indexes used by retrieval-augmented generation. - **Agent:** supporting automated interactions that return answers to users. ## Turning Traffic Data into Business Strategy - Site owners can use high-level metrics to evaluate whether their content security policies are effective. - More detailed operator-level data helps publishers understand how individual AI companies use their content. - Comparing operators can support negotiations about: - Blocking or allowing specific crawlers. - Licensing content. - Reconsidering existing commercial agreements. - Prioritizing relationships with companies that provide meaningful compensation or referrals. - The dashboard is intended to give publishers concrete evidence—such as comparative crawl volumes and referral performance—when discussing content access with AI companies. Cloudflare’s recommendation is effectively to stop treating all crawlers alike. Publishers should use crawl-to-referral ratios, resource consumption, crawler purpose, and commercial value to decide which bots deserve access and under what conditions.

ai-crawlersllmragweb-crawling+4

cloudflareApr 2, 2026

Why we-re rethinking cache for the AI era (opens in new tab)

AI traffic is fundamentally changing how CDNs should think about caching. Unlike human visitors, AI crawlers make broad, high-volume, often sequential requests for long-tail content, creating low reuse and substantial cache churn. Cloudflare argues that traditional LRU-based caching and techniques such as prefetching are increasingly poorly suited to this traffic, forcing operators to rethink cache design if they want to support AI access without harming human performance. ## Why AI Traffic Is Different - Automated traffic accounts for 32% of Cloudflare’s network traffic, including crawlers, scrapers, and AI assistants. - AI agents often: - Send many requests in parallel. - Scan large portions of a website sequentially. - Request rarely visited or loosely related pages. - Fetch documentation, images, and articles from many sources. - AI crawlers represent approximately 80% of self-identified AI bot traffic. - Most single-purpose AI bot traffic is associated with model training, with search-related crawling a distant second. ## The Three Defining Characteristics of AI Crawlers - **High unique URL ratio:** More than 90% of pages observed in large-scale Common Crawl datasets are unique by content. - **Content diversity:** Different crawlers target different materials, including source code, technical documentation, media, and blog posts. - **Crawling inefficiency:** Many requests lead to 404 errors or redirects because of poor URL handling. - AI crawlers generally lack browser-side caching and shared session behavior, so independent crawler instances may repeatedly appear as new visitors. - They can also repeatedly revisit content while iteratively refining search results, but each iteration still tends to fetch mostly new pages. ## How AI Crawling Disrupts Traditional Caches - Conventional CDN caching keeps frequently requested content available near users and evicts less recently used objects when storage fills. - Cloudflare uses an LRU (least recently used) policy, but broad AI scans introduce large numbers of low-reuse objects. - These objects can evict content that human visitors are more likely to request. - AI-driven long-tail access increases cache misses and sends more requests back to origin servers. - Cache speculation and prefetching become less effective because crawler access patterns are difficult to predict. - Higher miss rates can cause: - Slower responses. - Increased origin-server load. - Greater egress costs. - Reduced cache hit rates for human traffic. ## Implications for Website Operators - Operators face a tradeoff between optimizing infrastructure for human visitors and accommodating AI crawlers. - Some organizations may want to encourage AI access: - Developers may want documentation represented in AI models. - E-commerce companies may want product information included in LLM search results. - Publishers may seek compensation through systems such as pay-per-crawl. - The challenge is supporting useful AI traffic without allowing it to degrade the cache performance experienced by human users. Cloudflare’s analysis, conducted with ETH Zurich researchers, suggests that CDN caching strategies need to evolve beyond traditional assumptions about popularity and reuse. Cache systems designed specifically for AI-era traffic may need to isolate crawler workloads or otherwise prevent broad, low-reuse scans from displacing content valuable to human users.

ai-crawlersragcdndata-fetching+4

cloudflareFeb 12, 2026

Introducing Markdown for Agents (opens in new tab)

AI agents increasingly need structured, efficient access to web content, making traditional HTML a costly format for machine consumption. Cloudflare’s Markdown for Agents lets enabled websites serve HTML pages as Markdown when clients request `text/markdown`, reducing token usage and parsing overhead. The post argues that websites should treat AI agents as first-class visitors alongside humans and search engines. ## Why Markdown Matters for AI - Markdown conveys document structure with far less surrounding markup than HTML. - A Markdown heading such as `## About Us` uses roughly 3 tokens, compared with 12–15 tokens for an equivalent HTML heading. - The post’s HTML uses about 16,180 tokens, while its Markdown version uses approximately 3,150—a reduction of about 80%. - Converting HTML to Markdown inside an AI pipeline adds computation, cost, and complexity, and may not preserve the publisher’s intended structure. ## How Markdown for Agents Works - Cloudflare-enabled zones can respond to content negotiation requests containing: ```http Accept: text/markdown ``` - Cloudflare fetches the original HTML from the origin, converts it to Markdown at the network edge, and returns the converted response. - Clients can request Markdown with `curl`, while Workers-based agents can use a `fetch()` request with `Accept: "text/markdown, text/html"`. - Responses use `Content-Type: text/markdown` and include `Vary: accept`. - Existing coding agents, including Claude Code and OpenCode, already send compatible `Accept` headers. ## Token Estimates and Agent Workflows - Converted responses include an `x-markdown-tokens` header. - Agents can use this estimate to: - Determine whether content fits within a context window - Plan chunking strategies - Manage processing costs and limits ## Content Signals - Markdown responses include: ```http Content-Signal: ai-train=yes, search=yes, ai-input=yes ``` - These signals indicate that the content may be used for AI training, search results, and AI input, including agentic applications. - Cloudflare says future versions will support custom Content Signal policies. ## Availability - Cloudflare enabled Markdown for Agents on its Developer Documentation and Blog. - AI crawlers and agents can test the feature by requesting those pages with `Accept: text/markdown`. Web publishers can make their content more accessible to AI systems by supporting Markdown negotiation, while agents should request `text/markdown` whenever available to reduce tokens, parsing work, and processing cost.

ai-crawlersai-agentcloudflarecloudflare-workers+4

cloudflareJan 15, 2026

Human Native is joining Cloudflare (opens in new tab)

Cloudflare has acquired Human Native, a UK-based marketplace that helps AI developers obtain structured, licensed multimedia data. The acquisition supports Cloudflare’s broader effort to create a fairer AI ecosystem in which content owners control access, receive compensation, and get credit. Together, the companies aim to replace indiscriminate crawling with structured data delivery and machine-to-machine payments. ## Human Native’s Role - Human Native transforms messy, unstructured content into data that can be searched, understood, licensed, and valued. - Its approach treats data as a transparent, governed asset rather than something freely scraped. - A UK video AI customer reportedly replaced its existing training data with higher-quality, fully licensed content sourced through Human Native. - Cloudflare expects the team to help publishers and developers make content more usable for AI systems. ## Strain on the Internet’s Existing Economic Model - The traditional web exchange has been: - Creators publish content. - Search engines and social platforms send traffic. - Creators monetize that traffic through advertising, subscriptions, or direct support. - AI crawling is disrupting this model, with tens of thousands of bot crawls sometimes occurring for every human visit. - Content owners often do not know how crawlers use their material and increasingly want control over AI access. - Cloudflare’s AI Crawl Control and Pay Per Crawl tools are intended to let owners choose whether content is broadly accessible, restricted, or available only for compensation. ## Structured Access for AI Developers - Crawling the open web is costly and unreliable, producing duplicate, spammy, illegal, or otherwise low-quality material. - Cloudflare’s AI Index is designed to provide AI companies and agents with content at scale without repeated, indiscriminate crawling. - Through a publish/subscribe model, participating websites could send structured, real-time updates whenever content changes. - This approach may improve data quality while giving publishers more opportunities to create sustainable business models. ## Machine-to-Machine Payments - Cloudflare is working with Coinbase on the x402 Foundation to support automated payments for digital resources. - Traditional web payments require human actions such as adding items to a cart and entering card details. - AI agents and other automated systems need protocols that allow them to discover, purchase, and exchange digital resources directly. - Human Native’s technology and Cloudflare’s payment infrastructure are intended to support these emerging transactions. ## Practical Outlook The acquisition positions Cloudflare to build an AI data ecosystem based on licensed content, structured distribution, owner-controlled access, and automated compensation. Its stated goal is an Internet that remains open and useful while becoming fairer and more economically sustainable in the age of AI.

ai-crawlersgen-aifoundation-modelsai-training+4