edge-computing

9 posts

cloudflare

Secure all your internal vibe-coded applications — in one click (opens in new tab)

AI-driven development makes it easy for employees to deploy applications, but also increases the risk of unintentionally exposing company data. Cloudflare’s new Access integration for Workers makes applications private by default at the Worker or account level, regardless of how they are reached. It also exposes authenticated user identity directly in Worker code and supports private-by-default internal deployment platforms. ## Worker-Level Access Protection - Access authentication is enforced before requests reach application code. - Protection applies across custom domains, routes, `workers.dev` subdomains, and preview URLs. - Policies can cover: - Preview deployments only - Every hostname associated with a Worker - Attaching policies to the Worker eliminates the need to update Access settings whenever a new domain is added. - Existing identity providers, email addresses, domains, groups, and service tokens can control access. ## Account-Wide Private Defaults - An account-level policy automatically protects all current and future Workers. - Organizations can protect preview traffic, production traffic, or both. - Public Workers can explicitly bypass the account-wide policy. - For individual applications, Worker policies provide targeted protection. - When multiple policies apply, precedence is: - Hostname policies - Worker policies - Account policies ## Accessing User Identity in Worker Code - Authenticated requests expose identity through `ctx.access`. - `ctx.access.getIdentity()` returns information such as: - Email address - Name - Groups - Developers no longer need to parse, validate, and extract claims from Access JWTs manually. - Applications can use this identity for personalization, authorization, and per-user logging. - Code should handle requests without Access metadata, for example by returning a `403` response. ## Local Development and Testing - `wrangler dev` can simulate authenticated users locally. - An `access.dev` block in `wrangler.jsonc` defines a test audience and identity: ```json { "access": { "dev": { "aud": "my-app", "identity": { "email": "admin@company.com" } } } } ``` - Developers can change the configured email to test different user experiences without repeatedly deploying and authenticating through Access. ## Private Internal Deployment Platforms - Workers for Platforms can host many applications inside a namespace. - Traffic is routed through a shared dispatch Worker. - Protecting the dispatch Worker with Access makes every application deployed through it private by default. - Cloudflare provides an open-source example of an internal drag-and-drop deployment platform using this model. ## Infrastructure Behind the Feature - The capability relies on FL2, Cloudflare’s Rust-based modular proxy. - Workers routing had to be separated from execution so Cloudflare could determine the destination Worker before applying Access. - This routing change would have been more difficult in the older NGINX- and Lua-based FL1 architecture. Cloudflare’s approach shifts application security from an optional developer-configured step to an organizational default. Teams deploying internal or experimental Workers should use account-level or dispatch-level Access policies, while using Worker-level policies and local identity simulation for application-specific control and testing.

google

Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction (opens in new tab)

Google introduces a way to add Multi-Token Prediction (MTP) to already-deployed, frozen Gemini Nano models, accelerating on-device generation without a separate drafting model. The approach attaches a lightweight Transformer head to the existing model, reuses its hidden states and KV cache, and preserves identical final outputs through verification. On Pixel 9 and 10 devices, it delivers faster generation, lower memory use, and reduced energy consumption for features such as Notification Summaries and Proofread. ## The Mobile Inference Bottleneck - Autoregressive models generate one token at a time, creating latency and underusing mobile hardware. - Phones face strict RAM and energy constraints that make conventional acceleration techniques difficult. - A standalone speculative-decoding drafter consumes additional memory and must independently process the prompt. ## A “Late Exit” MTP Strategy - Speculative decoding uses: - A small drafter to propose several tokens. - The large model to verify those tokens in parallel. - MTP replaces the separate drafter with a lightweight Transformer head attached near the end of the main model. - The head uses the backbone’s high-dimensional activations to predict future tokens, benefiting from semantic context already computed by the larger model. ## Retrofitting a Frozen Backbone - Google freezes the fully trained Gemini Nano v3 weights and trains only the attached MTP head. - This avoids retraining or fine-tuning the production foundation model. - Incorrect draft tokens are discarded during verification, so the final output remains bit-for-bit identical to the original model. - The method therefore improves efficiency without changing the model’s capabilities or safety alignment. ## Zero-Copy Memory Architecture - The MTP head cross-attends directly to the backbone’s existing KV cache instead of maintaining a duplicate cache. - This eliminates separate prompt-prefill work for the drafter. - It also removes redundant embedding tables, attention variants, and application-specific tuning parameters. - Compared with a standalone drafter, the design saves up to 130 MB per instance. ## Accuracy and Speed Improvements - Access to the backbone’s richer internal representations makes MTP predictions more accurate than those of similarly sized standalone drafters. - Instruction-following tasks such as summarization and rewriting show especially strong gains. - For predictable formats such as smart replies, token acceptance improved by up to 55%. - Pixel 9 experiments showed speedups of 50% or more depending on the task. ## Production Impact - The updated inference stack coordinates drafting and verification on Pixel 9 and 10 devices. - In workloads including AI Notification Summaries and Proofread, MTP predicts nearly two additional tokens per inference pass on average. - Fewer verification cycles reduce processor wake-ups, improving latency and battery efficiency. MTP provides a practical way to accelerate existing on-device models without maintaining a separate drafter for every application. Reusing the frozen backbone’s computations and memory state makes it particularly well suited to mobile deployments where RAM, energy, and backward compatibility are critical.

toss

The History of Facial Recognition and the Future of Face Pay (opens in new tab)

Face recognition has evolved from manually recorded facial measurements to deep-learning systems capable of identifying people with near-perfect accuracy. The technology’s progression depended on automated feature extraction, robust local descriptors, neural networks, and increasingly large datasets. Face payment represents its next major application, combining convenience with layered security designed to address biometric privacy risks. ## From Manual Measurements to Automated Recognition - In the 1960s, Woodrow Wilson Bledsoe pioneered face recognition through a classified government project. - Researchers manually recorded coordinates for features such as eye spacing, nose-to-mouth distance, and ear positions. - Computers performed calculations, but humans still identified the facial landmarks. - In 1973, Takeo Kanade developed a system that automatically detected the eyes, nose, and mouth. - His work demonstrated that computers could extract meaningful facial information without manually supplied coordinates. ## Eigenfaces and Statistical Representation - In 1991, Matthew Turk and Alex Pentland introduced Eigenfaces using Principal Components Analysis (PCA). - The system created an average face and represented individual faces by how they differed from that average. - These differences formed “Eigenfaces,” mathematical components that worked like an alphabet for describing faces. - Recognition became a matter of comparing the combination weights assigned to two facial images. ## Coping with Lighting and Real-World Conditions - Early systems struggled when the same face appeared under different lighting, angles, or shadows. - Researchers shifted from analyzing the entire face to examining local regions and textures. - Local Binary Patterns (LBP) compared each pixel with its neighbors, making facial descriptions less dependent on absolute brightness. - Hand-crafted features were combined with classifiers such as: - **SVM**, which finds separating boundaries between identities. - **AdaBoost**, which combines weak classifiers into a stronger recognition system. - These methods improved robustness even though they could not learn features automatically like modern deep-learning models. ## Deep Learning Surpasses Human-Level Benchmarks - Facebook’s 2014 DeepFace system used deep neural networks trained on approximately four million face images. - It achieved 97.35% accuracy on the LFW benchmark, approaching the reported human score of 97.53%. - Google FaceNet later introduced Triplet Loss and reached 99.63% on LFW. - SphereFace, CosFace, and ArcFace continued improving discriminative facial representations. - The field shifted from proving that recognition was possible to optimizing accuracy and reliability. ## The Role of Large-Scale Datasets - **FERET:** Established an early standardized benchmark with 1,199 people and 14,126 grayscale images. - **LFW:** Contained 13,233 images of 5,749 people collected from natural, uncontrolled environments. - **VGGFace:** Provided 2.7 million images of 2,600 people, reinforcing the relationship between data scale and deep-learning performance. - **MS-Celeb-1M:** Pursued celebrity recognition at massive scale but was later withdrawn over privacy concerns. - **WebFace260M:** Started with 260 million images and refined them into roughly 42 million images covering two million people. - These datasets improved recognition while also highlighting the tension between technical progress, data quality, and privacy. ## Face Recognition as a Payment Method - Face recognition had already reached smartphones, airports, and office access systems, but payments required stricter standards. - Unlike device unlocking, payment errors can directly cause financial loss. - Toss FacePay launched in South Korea in September 2025, aiming to combine strong security with a frictionless experience. - Face payment offers: - Availability without carrying a wallet or charged phone. - Hands-free checkout. - Faster payment without searching for cards or opening an app. ## How FacePay Works - A terminal camera identifies the customer among registered users. - The system must also reject unregistered users and apply additional authentication in ambiguous cases, such as similar-looking individuals or twins. - Once identified, the system connects the customer to a registered credit card, debit card, or another selected payment method. - Face payment adds customer identification and payment-method lookup to the conventional POS, VAN, and card-issuer approval flow. ## Edge and Cloud Processing - **Edge processing** offers low latency and better privacy because images need not leave the terminal. - Its limitations include restricted hardware, smaller models, lower potential accuracy, and difficult device-by-device updates. - **Cloud processing** enables stronger models, centralized logging, and simpler model updates. - It introduces transmission delays and increases the importance of network and server security. - FacePay combines both approaches: the terminal performs initial processing, while the server handles facial-feature extraction, recognition, and payment operations. ## Layered Biometric Security - Communications between terminals and servers are protected with TLS, while images receive additional AES-256 encryption. - Matrix Projection creates cancellable biometric vectors: changing the key produces a different representation of the same face. - If a biometric vector were compromised, it could theoretically be reissued with a new key, similar to changing a password. - The stored biometric representation is designed not to correspond one-to-one with the original face image or allow easy reconstruction. - Access to biometric data is restricted, logged, and limited to operational needs. - FacePay also underwent review by South Korea’s Personal Information Protection Commission and offers full compensation for fraudulent payments through its protection program. The development of face payment reflects six decades of progress in computer vision, machine learning, data infrastructure, and security engineering. Its practical success depends not only on recognition accuracy, but also on cancellable biometric protection, strict access controls, privacy oversight, and effective defenses against spoofing.

discord

How We Moved Discord Voice to the Edge (opens in new tab)

Discord migrated its voice and video infrastructure from roughly 30 hyperscaler regions to Cloudflare’s edge network, which spans more than 300 cities. The move brought measurable gains—over 80% of traffic now runs on Cloudflare, with Frankfurt seeing 34% lower ping and 42% lower packet loss—but the rollout exposed important limitations. Local server proximity alone is not enough; call placement and network peering are equally critical. ## Why Discord Moved Voice to the Edge - Traditional cloud regions left users in places such as Reykjavik, Auckland, Hawaii, and Lagos relatively far from voice servers. - Cloudflare’s 300-plus points of presence offered the possibility of hosting Discord’s software much closer to users. - Unlike typical CDN workloads, Discord needed to route live UDP packets for voice and video calls rather than cache static content. - More than 80% of Discord voice and video traffic now uses Cloudflare, and 70% of regions have improved year over year. ## Iceland: Local Servers Can Hurt Mixed Calls - Discord’s first test took place in Reykjavik, where users previously connected to servers in Rotterdam. - Iceland-only calls improved: - Ping decreased 9%. - Packet loss decreased 11%. - Mixed-region calls became worse: - Ping for non-Iceland participants increased 2.7 times. - Packet loss rose 9%. - Discord assigns one SFU to host an entire call, so all participants send traffic to that server. - A call started by one Icelandic user could therefore force German participants to route packets to Iceland and back. - Discord concluded that new edge locations help only when calls hosted there are primarily local; mixed calls require smarter host-placement logic. ## Rotterdam: Network Peering Was the Bottleneck - Discord next moved Rotterdam traffic to Cloudflare’s Amsterdam PoP. - Most regional ISPs performed normally, but Orange users experienced: - More than one second of latency during peak periods. - A 30% regression in voice freeze ratio. - The issue was not Cloudflare’s Amsterdam infrastructure itself, but the path between Orange and Cloudflare. - Traffic traveled through Telia’s transit backbone, where the Telia–Orange connection was already saturated. - Adding more traffic made the congestion worse. - Discord reverted the migration after about ten days. - Cloudflare pursued direct peering with Orange and added SFUs in Paris and London to provide shorter paths. ## A More Careful Migration Strategy - Discord changed its rollout process from capacity-based scheduling to peering-based analysis. - Before shifting meaningful traffic to a region, the team now checks whether Cloudflare has sufficient peering capacity with major local ISPs. - The migration slowed from a planned large-scale rollout to a region-by-region deployment. - The experience demonstrated that: - A geographically closer PoP does not always produce the best call experience. - SFU host placement matters for calls spanning multiple regions. - Transit routes and ISP peering can be more important than physical distance. Discord’s edge migration is succeeding, but only through incremental deployment, detailed traffic analysis, and improved call-placement decisions. The practical lesson is that low-latency infrastructure depends on the complete network path—not simply on adding servers closer to users.

aws

AWS Weekly Roundup: BYOM for Amazon RDS for SQL Server, AWS IoT Device SDK for Swift, and more (June 8, 2026) | Amazon Web Services (opens in new tab)

The AWS roundup highlights the general availability of the AWS IoT Device SDK for Swift, bringing MQTT 5, Device Shadow, Jobs, and fleet provisioning to Apple platforms and Linux. It also covers major AWS releases involving SQL Server licensing, Cognito resilience, OpenAI models on Bedrock, Kubernetes, AI agents, cost reporting, and location services. Together, the announcements show AWS expanding support for Swift edge computing, enterprise AI, multi-Region architectures, and specialized infrastructure. ## AWS IoT Device SDK for Swift - Now generally available for macOS, iOS, tvOS, and Linux. - Provides production-ready support for: - MQTT 5 connectivity - Device Shadow - IoT Jobs - Fleet provisioning - Reflects Swift’s growing use across server-side development, IoT, and edge computing. - Projects such as WendyOS are also bringing Swift to NVIDIA Jetson and Raspberry Pi hardware. ## Major AWS Headlines ### Amazon RDS for SQL Server BYOM - Amazon RDS for SQL Server now supports Bring Your Own Media. - Customers migrating from on-premises SQL Server can reuse existing licenses, including Software Assurance. - Support is provided through Microsoft’s License Mobility program. - AWS License Manager tracks license usage and compliance. ### Multi-Region Amazon Cognito - Cognito can replicate user and machine identity data to a standby Region in near real time. - Replicated data includes credentials, user pool settings, and federation configurations. - Users can continue using applications without re-authentication after a primary-Region disruption. - Available as an add-on for Essentials and Plus user pools across 16 Regions. ### OpenAI Models on Amazon Bedrock - GPT-5.5, GPT-5.4, and Codex are generally available for production use. - GPT-5.5 targets agentic coding, data analysis, and complex autonomous tasks. - Codex supports the Codex App, CLI, and integrations with VS Code, JetBrains, and Xcode. - AWS governance and security controls remain available, pricing follows OpenAI rates, and usage counts toward existing AWS commitments. ## Recent AWS Launches - **Amazon Bedrock observability:** CloudWatch metrics now cover inference counts, token usage, and client errors for OpenAI- and Anthropic-compatible APIs. - **Redesigned Bedrock console:** Adds model catalogs, side-by-side comparisons, project organization, and pre-filled code examples. - **AgentCore Identity secrets:** Credential providers can reference existing AWS Secrets Manager secret ARNs, supporting custom KMS keys, tagging, and rotation. - **Step Functions agentic reasoning:** Workflows can invoke AgentCore-powered agents sequentially or in parallel, include human approval, and trace decisions. - **Kubernetes 1.36 on EKS:** Adds User Namespaces GA, Mutating Admission Policies, in-place pod resource scaling, and resource health reporting. - **ECS Managed Instances accelerators:** Supports Trainium1, Trainium2, and Inferentia2 instances with automatic accelerator allocation. - **Amazon Quick VPC connectivity:** Enables private connections to MCP servers without exposing internal tools to the public internet. - **Cost and Usage Report 2.0:** Adds Athena and Redshift integrations with generated infrastructure templates, table definitions, and loading guidance. - **Amazon Location Service:** Routes API now supports transit and intermodal journeys across 13 Regions. AWS also directs readers to its What’s New page, Builder Center, and upcoming events for further announcements and community resources.

cloudflare

Introducing Flagship: feature flags built for the age of AI (opens in new tab)

AI-generated code is moving toward autonomous production deployment, making safety and controlled rollout essential. The post argues that feature flags provide the guardrails: agents can deploy disabled code, test it with limited cohorts, monitor results, and roll back automatically. Cloudflare’s new Flagship service is designed for this workflow, evaluating flags at the edge through Workers, KV, and Durable Objects. ## Feature Flags for Autonomous Deployment - Agents can ship code behind an off flag without affecting users. - They can enable features for themselves or small test cohorts, observe metrics, and expand or disable rollouts. - Humans define boundaries while flags limit the blast radius. - This separates not only deployment from release, but also routine shipping decisions from constant human attention. ## Problems with Feature Flags on Workers - Hardcoded flags are initially convenient because Workers deploy quickly. - Over time, flags become fragmented across teams, with no central visibility or audit trail. - Troubleshooting may require searching version history with tools such as `git blame`. - Calling an external flag service adds a network request to every user request, potentially introducing significant latency. - This undermines the advantage of running applications close to users at the edge. ## Why Local Evaluation Is Difficult on Workers - Traditional local-evaluation SDKs download rules into a long-lived process. - Worker isolates may be created and evicted between requests, requiring repeated initialization. - Serverless environments therefore need a distribution system with edge-local reads and managed synchronization. - Flagship uses Cloudflare KV to provide this distribution without persistent connections or per-request external calls. ## How Flagship Works - Flagship is built on Workers, Durable Objects, and KV, without external databases or centralized evaluation servers. - Durable Objects provide a globally unique, SQLite-backed source of truth for flag configuration and changelogs. - Changes are synchronized to KV within seconds and replicated throughout Cloudflare’s network. - Evaluations read configuration from KV at the edge and execute targeting and rollout logic inside the Worker isolate. - Both flag data and evaluation logic remain close to the request. ## Worker Binding and Typed Evaluation - Workers connect Flagship through a `wrangler.jsonc` binding containing a binding name and `app_id`. - The binding supports typed methods including: - `getBooleanValue()` - `getStringValue()` - `getNumberValue()` - `getObjectValue()` - `*Details()` methods return the value, matched variant, and selection reason. - Evaluation errors return the supplied default value. - Type mismatches throw exceptions because they indicate application bugs rather than temporary service failures. ## OpenFeature Integration - Flagship is built on OpenFeature, the CNCF standard for feature-flag evaluation. - It supports Workers as well as Node.js, Bun, Deno, and browser environments. - The service is currently available in closed beta. Flagship is positioned as an edge-native feature-flag system for safely automating deployment and rollout. For Cloudflare Workers, its direct binding avoids network round-trips while providing centralized configuration, targeting, auditability, and controlled release mechanisms.

cloudflare

The truly programmable SASE platform (opens in new tab)

Cloudflare argues that true SASE programmability goes beyond APIs, Terraform, webhooks, and alerts. It means intercepting security events, enriching them with external context, and making real-time decisions through custom logic. By running Cloudflare One and its Developer Platform on the same global edge network, Cloudflare aims to let customers apply programmable, low-latency policies without stitching together separate infrastructure. ## What “Programmability” Means - Traditional programmability supports configuration and automation, such as sending Slack alerts when policies trigger. - True programmability allows security systems to: - Inspect an event before access is granted. - Query external systems for additional context. - Make or change an access decision in real time. - Example: a request to a regulated application could be checked against a learning management system to confirm that the user’s compliance training is current. Expired or missing certification would result in denial and redirection to training. ## Cloudflare’s Programmable SASE Architecture - Cloudflare’s network spans more than 330 cities and reaches approximately 95% of Internet-connected users within 50 milliseconds. - Cloudflare One and the Developer Platform run on the same infrastructure and use shared network primitives. - This enables Workers to extend inline services such as Access without requiring separate cloud infrastructure. - Customers can: - Call external risk APIs. - Add dynamic request headers. - Validate browser attributes. - Route traffic according to custom business logic. - Running custom logic at the edge reduces latency and avoids the operational overhead of webhook-based integrations and disconnected systems. ## Custom Actions in Security Policies - Conventional gateways generally limit policy outcomes to actions such as allow, block, isolate, or quarantine. - Cloudflare is expanding policies to support managed and custom actions. - Potential uses include: - Injecting headers based on user identity claims. - Obtaining real-time verdicts from external risk engines. - Restricting access based on location or working hours. - Updating risk lists based on scheduled analysis of user activity. - Custom actions can invoke a Worker when a Gateway HTTP policy matches, giving the code access to request context and allowing decisions in milliseconds. - Managed actions will offer templates for common use cases such as IT service management, redirects, and compliance workflows. ## Automated Device Session Revocation - One customer needed periodic re-authentication for Cloudflare One Client users, similar to traditional VPN session expiration. - Cloudflare’s built-in session controls were application-specific rather than global and time-based. - The customer implemented a scheduled Worker that: - Queries the Cloudflare Devices API. - Handles cursor-based pagination to retrieve registrations. - Calculates how long each device has been inactive. - Deletes registrations exceeding a configured inactivity threshold. - Forces affected users to authenticate again through their identity provider. - The example also supports environment-based configuration and a dry-run mode for testing before revocations are applied. Cloudflare’s recommendation is to treat SASE policies as programmable decision points rather than fixed allow-or-block rules. Combining Cloudflare One with edge Workers can provide faster, more context-aware security automation while reducing integration complexity.

cloudflare

Modernizing with agile SASE: a Cloudflare One blog takeover (opens in new tab)

The post argues that changing work patterns, AI agents, and Internet-based perimeters require organizations to move beyond fragmented legacy networks toward “agile SASE.” It presents Cloudflare One as a composable platform that combines networking and security on a global connectivity cloud, using single-pass processing to avoid service-chaining bottlenecks. Cloudflare positions this architecture as a faster path to modernization, beginning with focused use cases rather than a large-scale “big bang” migration. ## The Case for Agile SASE - Hybrid work and AI-driven traffic are making traditional corporate perimeters and office-based networks obsolete. - Legacy firewalls, VPN concentrators, and hardware appliances create a “fragmentation penalty.” - Technical debt accumulates through: - Conflicting firewall rules - Manual patching - Aging hardware - Infrastructure unable to handle AI-scale traffic - First-generation SASE platforms often shifted fragmentation from physical hardware into isolated cloud and operational silos. - The resulting problem is not a shortage of security data, but difficulty enforcing consistent policies across a borderless enterprise. ## Cloudflare One’s Single-Pass Architecture - Cloudflare describes zero trust as the security model and Cloudflare One as the platform for implementing it. - The platform converges networking and security through a global connectivity cloud spanning more than 300 cities. - Security checks can run simultaneously on every server rather than processing traffic sequentially through separate services. - This avoids service chaining, which can introduce latency and operational complexity. - Cloudflare frames the result as a programmable, composable platform rather than a collection of acquired or loosely connected tools. ## Five Themes for Network Modernization The company’s planned technical series focuses on: - **A new network standard:** Building a programmable, future-ready Internet foundation. - **Identity beyond passwords:** Combining human and device verification instead of relying only on credentials. - **Signal over noise:** Using AI to convert large volumes of security data into actionable guidance. - **The autonomous edge:** Improving performance and reducing friction as part of the security strategy. - **A unified vision:** Showing how enterprises and partners can standardize on Cloudflare One at scale. ## Programmability and Developer Integration - Cloudflare One runs alongside Cloudflare Workers, allowing teams to write code that responds to security events in real time. - This extends policy enforcement beyond simple allow/block decisions. - Organizations can automate more sophisticated security and operational workflows. - Cloudflare argues that this flexibility helps technology teams support faster business growth while improving protection. ## Practical Starting Points The post recommends beginning with focused modernization projects: - **Remote access:** Replace maintenance-heavy VPNs with clientless access and faster zero trust adoption. - **Email protection:** Detect phishing, Business Email Compromise, and related multi-channel threats with AI-powered controls. - **DNS filtering:** Block malicious websites for hybrid workers using DNS protection based on the 1.1.1.1 resolver. - **AI governance:** Identify shadow AI usage and control how organizational data enters generative and agentic AI systems. - **Branch networking:** Treat offices and coffee-shop workspaces as remote sites, reducing dependence on dedicated hardware. Organizations evaluating SASE should favor platforms that provide consistent policy enforcement, programmable controls, and incremental adoption. Cloudflare’s recommended entry point is to start with a specific need—such as VPN replacement or AI governance—and expand toward a unified connectivity and security architecture.

cloudflare

Cloudflare outage on February 20, 2026 (opens in new tab)

Cloudflare suffered a 6-hour, 7-minute outage on February 20, 2026, after a software change unintentionally withdrew Internet routes for some Bring Your Own IP (BYOIP) customers. The incident was not related to a cyberattack; a buggy automated cleanup task altered customer prefix and service configurations. Cloudflare reverted the change, restored affected prefixes, and is revising its Addressing API workflows to reduce production risk. ## Customer Impact - Approximately 1,100 of Cloudflare’s 6,500 advertised prefixes were withdrawn between 17:56 and 18:46 UTC. - This affected about 25% of the 4,306 BYOIP prefixes advertised globally. - Impacted applications became unreachable from the Internet and experienced connection failures and timeouts. - Customers initially encountered BGP Path Hunting, where networks repeatedly searched for a route until connections timed out. - The `one.one.one.one` website returned HTTP 403 errors and an “Edge IP Restricted” message. - DNS resolution through the 1.1.1.1 resolver, including DNS over HTTPS, was not affected. - The incident did not affect every BYOIP customer because the configuration change was applied incrementally and was reverted before reaching everyone. ## Recovery Efforts - Engineers detected the issue through failures involving `one.one.one.one` and reverted the change. - Cloudflare published dashboard guidance at 19:19 UTC, allowing many customers to re-advertise their prefixes themselves. - Around 800 prefixes were restored by approximately 20:20 UTC. - About 300 prefixes could not be restored through the dashboard because their service configurations had been removed from edge servers. - Engineers manually restored those remaining prefixes at 23:03 UTC. - Some customers continued to experience latency and failures while addressing configuration state propagated back to the edge. ## The Addressing API - Cloudflare’s Addressing API is the authoritative dataset for IP addresses present on its network. - Changes to the API drive workflows that propagate address and routing updates across Cloudflare’s edge. - The normal process is: - Customers request advertisement or withdrawal through the Addressing API or BGP Control. - The API instructs machines to change prefix advertisements. - Routers update BGP after enough machines receive the change. - Customers bind Cloudflare products to their BYOIP ranges. - Because the API is closely connected to production systems, manual changes are risky. - Cloudflare’s “Code Orange: Fail Small” initiative aims to replace manual Addressing API operations with safer, automated, health-checked workflows. ## Root Cause: Faulty BYOIP Cleanup Automation - The failed change automated the removal of prefixes from BYOIP, a task that had previously been performed manually. - A recurring cleanup sub-task searched for BYOIP prefixes marked for deletion and removed them. - The cleanup task issued the API request: ```go /v1/prefixes?pending_delete ``` - The request contained a bug in how the API query was interpreted. - As a result, the cleanup process unintentionally withdrew customer prefixes and removed related service configurations from some edge servers. - The incident lasted much longer than the initial withdrawal because restoring both advertisements and edge configuration state required extensive automated and manual recovery. Cloudflare’s main corrective direction is to make Addressing API changes safer through incremental, health-mediated deployment, stronger safeguards around automated deletion, and elimination of risky manual production workflows.