From ranking to recommended: get your site ready to thrive in the age of AI agents (opens in new tab)
AI assistants are becoming a major channel through which customers discover, compare, and purchase from businesses. As agents replace traditional search journeys, discoverability increasingly depends on whether a site is easy for machines to access, understand, trust, and recommend. Cloudflare’s Agent Readiness and Answer Engine Optimization (AEO) tools measure both technical accessibility and visibility in AI-generated recommendations. ## The Rise of the Agentic Audience - Fewer than half of HTML page requests reportedly come from humans, with the remainder including crawlers, bots, and increasingly capable agents. - Customers may ask AI assistants for: - Solutions to specific problems - Recommendations tailored to their needs - Comparisons between products or services - Actions performed on their behalf - Traditional metrics such as clicks and page views do not show whether AI systems can use a site or recommend it. - Agent-focused discoverability requires being found, read, and confidently cited by answer engines, shopping assistants, and research tools. ## Diagnostics: Measuring Agent Readiness Cloudflare’s Diagnostics feature evaluates a site from an agent’s perspective rather than simply loading its homepage. - It checks whether agents: - Are permitted to access the site - Can discover its content - Can retrieve clean, machine-readable content - Can identify callable interfaces and authentication methods - Diagnostic checks examine: - `robots.txt` - XML sitemaps - Response headers - Markdown content - Published metadata - APIs and agent-facing tools - Results are grouped into readiness levels from “Not Ready” to fully agent-native. - Each check returns: - Pass, fail, or neutral status - An explanation of its significance - Evidence showing the exact request and response ### Diagnostic Improvement Areas - **Quick wins:** Crawler-readable `robots.txt`, XML sitemaps, AI-crawler rules, and clean Markdown. - **Technical groundwork:** Content Signals, API catalogs, link headers, and agent login instructions. - **Advanced integration:** OAuth discovery, MCP, A2A agent cards, skills indexes, Web Bot Auth, and WebMCP. - **Commerce:** Emerging standards such as x402, ACP, UCP, and AP2. These are currently informational and do not affect the readiness score. - Recommended fixes either link directly to Cloudflare settings or generate a coding prompt that can be given to an agent. ## AEO: Measuring AI Recommendations Agent Readiness shows whether agents can access a site; AEO measures whether assistants actually recommend it. - Cloudflare infers a site’s industry and category, then tests likely customer prompts against assistants such as Claude and GPT. - Prompts cover recommendations, product comparisons, and general category advice without naming the customer’s brand. - AEO reports several visibility metrics: - **Citation Rate:** How often the site is cited as a source. - **Prominence:** How early and substantially the site appears in an answer. - **Mention Rate:** How often the brand is named, even without a source citation. - **Share of Voice:** The site’s share of citations compared with competitors. - Comparing mention and citation rates distinguishes brand awareness from authoritative attribution. ## Category Benchmarks and Industry Fit Cloudflare builds a benchmark for each industry and category before scoring individual sites. - AI assistants are queried with representative prompts to identify: - Which sites are cited - Where citations appear - How prominently each brand is represented - The benchmark is reused across accounts in the same category rather than regenerated for every scan. - This approach provides: - Instant result loading - Lower AI-compute costs - An **Industry Fit** score showing whether the site appears alongside its real competitors ## Multimodel Evaluation Because AI responses vary, Cloudflare queries assistants multiple times across different models using AI Gateway. - The system analyzes the actual answer text and cited sources customers would see. - Workers AI evaluates citation and mention patterns on Cloudflare’s infrastructure. - Exact text analysis is used alongside model-based judgment, rather than asking a model to grade its own response. - The process converts many variable responses into consistent, actionable metrics without requiring site owners to build their own evaluation framework. ## Operator Activity Cloudflare also reports real crawl and referral activity from AI operators. - Activity is shown by operator, including OpenAI and Google. - Site owners can see: - Which operators read their content - Which operators send visitors back - Errors encountered during crawling, such as `403` blocks and `404` links The broader goal is to connect controlled AEO benchmarks with real-world agent traffic, helping businesses understand both how AI systems perceive them and whether those systems can successfully access and use their sites. Businesses should treat agents as a core audience: first make the site technically accessible and machine-readable, then measure whether AI assistants cite and recommend it. Acting early may provide an advantage because most websites are not yet optimized for agent discovery.