cloudflareJul 1, 2026

Unmasking the crawls with Attribution Business Insights (opens in new tab)

Cloudflare argues that the traditional exchange between crawlers and publishers has broken down as AI bots extract content without sending meaningful referral traffic. This creates lost revenue for publishers while increasing hosting costs, making granular traffic attribution essential. Its new Attribution Business Insights dashboard aims to help site owners identify which bots provide value and make informed decisions about access, blocking, and commercial relationships. ## The Internet’s Changing Economics - Traditional search engines generally crawled content a few times for each visitor they referred. - That crawl-to-referral balance supported advertising, affiliate revenue, subscriptions, and direct audience relationships. - AI crawlers increasingly create a “zero-click” ecosystem by summarizing content without directing users to the original publisher. - Cloudflare observed AI crawl-to-referral ratios ranging from 118:1 to nearly 50,000:1. - Publishers face both reduced traffic-based revenue and higher infrastructure costs from unproductive automated access. ## Attribution Business Insights Dashboard - The dashboard is available to Cloudflare Bot Management customers. - It provides an immediate view of bot activity without requiring extensive manual analytics filtering. - It measures: - Human versus bot traffic to content pages. - Overall and operator-specific crawl-to-referral ratios. - Crawl-to-referral trends over 24 hours, seven days, or 30 days. - Top bots by traffic volume, country, bandwidth usage, and current allow/block status. - AI crawlers are classified by behavior: - **Training:** collecting data for future large language models. - **Search:** refreshing indexes used by retrieval-augmented generation. - **Agent:** supporting automated interactions that return answers to users. ## Turning Traffic Data into Business Strategy - Site owners can use high-level metrics to evaluate whether their content security policies are effective. - More detailed operator-level data helps publishers understand how individual AI companies use their content. - Comparing operators can support negotiations about: - Blocking or allowing specific crawlers. - Licensing content. - Reconsidering existing commercial agreements. - Prioritizing relationships with companies that provide meaningful compensation or referrals. - The dashboard is intended to give publishers concrete evidence—such as comparative crawl volumes and referral performance—when discussing content access with AI companies. Cloudflare’s recommendation is effectively to stop treating all crawlers alike. Publishers should use crawl-to-referral ratios, resource consumption, crawler purpose, and commercial value to decide which bots deserve access and under what conditions.

seollmragai-crawlers+4

stripeJun 23, 2026

Four travel and hospitality trends from HITEC 2026 (opens in new tab)

Hospitality’s AI opportunity is growing, but most operators lack the data, infrastructure, and operational systems needed to turn investment into measurable returns. AI is reshaping how travelers discover and book hotels, while fragmented data and outdated payment systems create lost revenue and guest frustration. The strongest strategy is to connect accurate data, intelligent workflows, and seamless payments so technology improves the experience without becoming visible to guests. ## AI Is Changing the Direct-Booking Battle - Hotels historically relied on SEO to compete with OTAs such as Expedia and Booking.com. - AI-generated search answers are reducing traditional website traffic: - 65% of Google searches with AI Overviews end without a click. - The figure rises to 78% on mobile. - Traditional search traffic is declining by about 25%. - AI systems prioritize accurate, structured, machine-readable information rather than keyword density and backlinks. - More than 90% of accommodation websites are reportedly undetected by AI models. - Hotels should audit whether AI tools can correctly describe: - Room categories - Amenities - Policies and cancellation terms - Local context - Real-time availability - Winning direct bookings will require both AI discoverability and a modern checkout experience supporting local currencies, payment methods, and fraud protection. ## Hospitality AI Is Held Back by Fragmented Data - Only about 25% of hospitality businesses are actively scaling AI, and fewer than 10% are considered “AI future-built.” - Property management, CRM, loyalty, food and beverage, and payment systems often operate in silos. - Incomplete data weakens: - Personalization - Guest profiles - Financial reconciliation - Operational decision-making - The main challenge is not building AI features but operationalizing them reliably in real workflows. - Successful examples connect live data to timely actions: - Delta’s AI concierge uses customer and operational data to provide context-aware support. - Wynn’s revenue managers receive predictive alerts and recommended actions. - For most operators, better data connectivity matters more than using a more advanced AI model. ## Payment Friction Directly Affects Revenue - Payments are increasingly viewed as a competitive capability rather than a back-office commodity. - Survey findings cited in the article include: - 90% of executives consider payments important to growth. - 37% say limited payment options most harm the guest experience. - 58% report that fraud tools block legitimate transactions. - 74% say fragmented systems create excessive reconciliation work. - Guests may abandon a hotel when their preferred payment method is unavailable, shifting the booking to an OTA that supports it. - Modern payment infrastructure allows smaller operators to offer international payment methods and currencies without building large in-house teams. ## Invisible Technology Creates the Best Guest Experience - Guests have little tolerance for technology failures and may simply avoid returning rather than complain. - Effective hospitality technology should anticipate needs without drawing attention to itself. - The desired experience includes details such as: - A room set to the guest’s preferred temperature - Familiar television channels - Preferred pillow firmness - Hospitality is moving from remembering information guests explicitly provided to predicting preferences based on connected guest data. Operators should prioritize clean, connected data, AI systems tied to real operational actions, and flexible payment infrastructure. The goal is not to add AI for its own sake, but to make booking and stays more seamless while quietly improving revenue, efficiency, and guest loyalty.

seoaillmai-agent+4

cloudflareJan 30, 2026

Google’s AI advantage: why crawler separation is the only path to a fair Internet (opens in new tab)

Google’s dominance in search gives it a structural advantage in generative AI: publishers must allow Googlebot to preserve search visibility, while Google can also reuse that access for AI products. The authors argue that this blurs search indexing and AI data collection, deprives publishers of traffic and compensation, and disadvantages competing AI companies. They support the CMA’s proposed UK conduct rules but say the only fair solution is to separate crawling for search from crawling for generative and agentic AI. ## CMA’s Strategic Market Status designation - The UK’s Digital Markets, Competition and Consumers Act 2024 allows the CMA to designate firms with substantial, entrenched market power as having Strategic Market Status. - In October 2025, Google received this designation for general search and search advertising, where it holds roughly 90% of the UK market. - The designation covers AI Overviews and AI Mode, allowing the CMA to impose legally enforceable conduct requirements on Google’s search ecosystem. - The authors view the CMA’s consultation as an important first step toward clearer rules for AI crawling and publisher control. ## Problems with Google’s dual-purpose crawler - Publishers cannot realistically block Googlebot because doing so could reduce their visibility in Google Search and damage advertising revenue. - Google uses the same search access not only for indexing and referrals, but also to ground AI Overviews, AI Mode, and broader generative AI services. - These AI features may reproduce publisher content while sending little or no traffic back to the original sites. - This threatens ad-supported publishing models and can put Google in direct competition with the publishers whose content it uses. - Unlike other AI companies, Google can obtain large amounts of content without negotiating payment, because publishers are effectively unable to refuse its search crawler. ## Google’s crawling advantage Cloudflare’s data indicates that Googlebot accesses substantially more unique pages than other major AI crawlers: - About 1.7 times more than ClaudeBot and GPTBot. - About 3 times more than Meta-ExternalAgent. - About 3.3 times more than Bingbot. - About 5.1 times more than Amazonbot. - Nearly 15 times more than Applebot. - Nearly 167 times more than PerplexityBot. - More than 700 times more than CCBot. - More than 1,800 times more than archive.org_bot. - Googlebot crawled roughly 8% of the sampled unique URLs during the two-month observation period. ## Limits of robots.txt and the need for separate controls - Publishers are much less likely to block Googlebot in `robots.txt` because of its importance for search referrals. - `robots.txt` expresses preferences but does not technically enforce crawler behavior; publishers must rely on bots to comply. - Web Application Firewalls can technically block unwanted crawlers, but this does not solve the core problem when search and AI access are tied to the same Googlebot identity. - The authors therefore argue that publishers need a meaningful, independent way to permit Google Search indexing while refusing the use of their content for generative AI. The proposed CMA rules should go further by requiring effective separation between search crawling and AI crawling. Publishers should be able to opt out of generative AI use without sacrificing search visibility, creating fairer conditions for content creators and competing AI developers.

seogen-aiagentic-aiweb-application-firewall+3