Introducing Markdown for Agents
AI agents increasingly need structured, efficient access to web content, making traditional HTML a costly format for machine consumption. Cloudflare’s Markdown for Agents lets enabled websites serve HTML pages as Markdown when clients request `text/markdown`, reducing token usage and parsing overhead. The post argues that websites should treat AI agents as first-class visitors alongside humans and search engines. ## Why Markdown Matters for AI - Markdown conveys document structure with far less surrounding markup than HTML. - A Markdown heading such as `## About Us` uses roughly 3 tokens, compared with 12–15 tokens for an equivalent HTML heading. - The post’s HTML uses about 16,180 tokens, while its Markdown version uses approximately 3,150—a reduction of about 80%. - Converting HTML to Markdown inside an AI pipeline adds computation, cost, and complexity, and may not preserve the publisher’s intended structure. ## How Markdown for Agents Works - Cloudflare-enabled zones can respond to content negotiation requests containing: ```http Accept: text/markdown ``` - Cloudflare fetches the original HTML from the origin, converts it to Markdown at the network edge, and returns the converted response. - Clients can request Markdown with `curl`, while Workers-based agents can use a `fetch()` request with `Accept: "text/markdown, text/html"`. - Responses use `Content-Type: text/markdown` and include `Vary: accept`. - Existing coding agents, including Claude Code and OpenCode, already send compatible `Accept` headers. ## Token Estimates and Agent Workflows - Converted responses include an `x-markdown-tokens` header. - Agents can use this estimate to: - Determine whether content fits within a context window - Plan chunking strategies - Manage processing costs and limits ## Content Signals - Markdown responses include: ```http Content-Signal: ai-train=yes, search=yes, ai-input=yes ``` - These signals indicate that the content may be used for AI training, search results, and AI input, including agentic applications. - Cloudflare says future versions will support custom Content Signal policies. ## Availability - Cloudflare enabled Markdown for Agents on its Developer Documentation and Blog. - AI crawlers and agents can test the feature by requesting those pages with `Accept: text/markdown`. Web publishers can make their content more accessible to AI systems by supporting Markdown negotiation, while agents should request `text/markdown` whenever available to reduce tokens, parsing work, and processing cost.
Read original(opens in new tab)