byoip

1 posts

cloudflare

Cloudflare outage on February 20, 2026 (opens in new tab)

Cloudflare suffered a 6-hour, 7-minute outage on February 20, 2026, after a software change unintentionally withdrew Internet routes for some Bring Your Own IP (BYOIP) customers. The incident was not related to a cyberattack; a buggy automated cleanup task altered customer prefix and service configurations. Cloudflare reverted the change, restored affected prefixes, and is revising its Addressing API workflows to reduce production risk. ## Customer Impact - Approximately 1,100 of Cloudflare’s 6,500 advertised prefixes were withdrawn between 17:56 and 18:46 UTC. - This affected about 25% of the 4,306 BYOIP prefixes advertised globally. - Impacted applications became unreachable from the Internet and experienced connection failures and timeouts. - Customers initially encountered BGP Path Hunting, where networks repeatedly searched for a route until connections timed out. - The `one.one.one.one` website returned HTTP 403 errors and an “Edge IP Restricted” message. - DNS resolution through the 1.1.1.1 resolver, including DNS over HTTPS, was not affected. - The incident did not affect every BYOIP customer because the configuration change was applied incrementally and was reverted before reaching everyone. ## Recovery Efforts - Engineers detected the issue through failures involving `one.one.one.one` and reverted the change. - Cloudflare published dashboard guidance at 19:19 UTC, allowing many customers to re-advertise their prefixes themselves. - Around 800 prefixes were restored by approximately 20:20 UTC. - About 300 prefixes could not be restored through the dashboard because their service configurations had been removed from edge servers. - Engineers manually restored those remaining prefixes at 23:03 UTC. - Some customers continued to experience latency and failures while addressing configuration state propagated back to the edge. ## The Addressing API - Cloudflare’s Addressing API is the authoritative dataset for IP addresses present on its network. - Changes to the API drive workflows that propagate address and routing updates across Cloudflare’s edge. - The normal process is: - Customers request advertisement or withdrawal through the Addressing API or BGP Control. - The API instructs machines to change prefix advertisements. - Routers update BGP after enough machines receive the change. - Customers bind Cloudflare products to their BYOIP ranges. - Because the API is closely connected to production systems, manual changes are risky. - Cloudflare’s “Code Orange: Fail Small” initiative aims to replace manual Addressing API operations with safer, automated, health-checked workflows. ## Root Cause: Faulty BYOIP Cleanup Automation - The failed change automated the removal of prefixes from BYOIP, a task that had previously been performed manually. - A recurring cleanup sub-task searched for BYOIP prefixes marked for deletion and removed them. - The cleanup task issued the API request: ```go /v1/prefixes?pending_delete ``` - The request contained a bug in how the API query was interpreted. - As a result, the cleanup process unintentionally withdrew customer prefixes and removed related service configurations from some edge servers. - The incident lasted much longer than the initial withdrawal because restoring both advertisements and edge configuration state required extensive automated and manual recovery. Cloudflare’s main corrective direction is to make Addressing API changes safer through incremental, health-mediated deployment, stronger safeguards around automated deletion, and elimination of risky manual production workflows.