techlist.io
KREN
cloudflare

From ranking to recommended: get your site ready to thrive in the age of AI agents (opens in new tab)

cloudflaremodel-context-protocoloauthai-gatewayworkers-aiwebmcpanswer-engine-optimizationagent2agent

AI assistants are becoming a major channel through which customers discover, compare, and purchase from businesses. As agents replace traditional search journeys, discoverability increasingly depends on whether a site is easy for machines to access, understand, trust, and recommend. Cloudflare’s Agent Readiness and Answer Engine Optimization (AEO) tools measure both technical accessibility and visibility in AI-generated recommendations.

The Rise of the Agentic Audience

  • Fewer than half of HTML page requests reportedly come from humans, with the remainder including crawlers, bots, and increasingly capable agents.
  • Customers may ask AI assistants for:
    • Solutions to specific problems
    • Recommendations tailored to their needs
    • Comparisons between products or services
    • Actions performed on their behalf
  • Traditional metrics such as clicks and page views do not show whether AI systems can use a site or recommend it.
  • Agent-focused discoverability requires being found, read, and confidently cited by answer engines, shopping assistants, and research tools.

Diagnostics: Measuring Agent Readiness

Cloudflare’s Diagnostics feature evaluates a site from an agent’s perspective rather than simply loading its homepage.

  • It checks whether agents:
    • Are permitted to access the site
    • Can discover its content
    • Can retrieve clean, machine-readable content
    • Can identify callable interfaces and authentication methods
  • Diagnostic checks examine:
    • robots.txt
    • XML sitemaps
    • Response headers
    • Markdown content
    • Published metadata
    • APIs and agent-facing tools
  • Results are grouped into readiness levels from “Not Ready” to fully agent-native.
  • Each check returns:
    • Pass, fail, or neutral status
    • An explanation of its significance
    • Evidence showing the exact request and response

Diagnostic Improvement Areas

  • Quick wins: Crawler-readable robots.txt, XML sitemaps, AI-crawler rules, and clean Markdown.
  • Technical groundwork: Content Signals, API catalogs, link headers, and agent login instructions.
  • Advanced integration: OAuth discovery, MCP, A2A agent cards, skills indexes, Web Bot Auth, and WebMCP.
  • Commerce: Emerging standards such as x402, ACP, UCP, and AP2. These are currently informational and do not affect the readiness score.
  • Recommended fixes either link directly to Cloudflare settings or generate a coding prompt that can be given to an agent.

AEO: Measuring AI Recommendations

Agent Readiness shows whether agents can access a site; AEO measures whether assistants actually recommend it.

  • Cloudflare infers a site’s industry and category, then tests likely customer prompts against assistants such as Claude and GPT.
  • Prompts cover recommendations, product comparisons, and general category advice without naming the customer’s brand.
  • AEO reports several visibility metrics:
    • Citation Rate: How often the site is cited as a source.
    • Prominence: How early and substantially the site appears in an answer.
    • Mention Rate: How often the brand is named, even without a source citation.
    • Share of Voice: The site’s share of citations compared with competitors.
  • Comparing mention and citation rates distinguishes brand awareness from authoritative attribution.

Category Benchmarks and Industry Fit

Cloudflare builds a benchmark for each industry and category before scoring individual sites.

  • AI assistants are queried with representative prompts to identify:
    • Which sites are cited
    • Where citations appear
    • How prominently each brand is represented
  • The benchmark is reused across accounts in the same category rather than regenerated for every scan.
  • This approach provides:
    • Instant result loading
    • Lower AI-compute costs
    • An Industry Fit score showing whether the site appears alongside its real competitors

Multimodel Evaluation

Because AI responses vary, Cloudflare queries assistants multiple times across different models using AI Gateway.

  • The system analyzes the actual answer text and cited sources customers would see.
  • Workers AI evaluates citation and mention patterns on Cloudflare’s infrastructure.
  • Exact text analysis is used alongside model-based judgment, rather than asking a model to grade its own response.
  • The process converts many variable responses into consistent, actionable metrics without requiring site owners to build their own evaluation framework.

Operator Activity

Cloudflare also reports real crawl and referral activity from AI operators.

  • Activity is shown by operator, including OpenAI and Google.
  • Site owners can see:
    • Which operators read their content
    • Which operators send visitors back
    • Errors encountered during crawling, such as 403 blocks and 404 links

The broader goal is to connect controlled AEO benchmarks with real-world agent traffic, helping businesses understand both how AI systems perceive them and whether those systems can successfully access and use their sites.

Businesses should treat agents as a core audience: first make the site technically accessible and machine-readable, then measure whether AI assistants cite and recommend it. Acting early may provide an advantage because most websites are not yet optimized for agent discovery.