slackMar 31, 2026

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus (opens in new tab)

Slack needed better client-side observability while migrating edge services to HTTP/3, which uses QUIC over UDP rather than TCP. Existing SaaS tools and Prometheus Blackbox Exporter could not probe HTTP/3 endpoints, so an intern added QUIC support using Go’s `quic-go` library and open-sourced it. The result unified HTTP/1.1, HTTP/2, and HTTP/3 monitoring while making the capability available to the broader Prometheus community. ## Limitations of Legacy Monitoring - Slack used a mix of commercial monitoring services and internal tools for network measurements. - HTTP/3 introduced a major observability gap because it runs over QUIC/UDP. - Existing SaaS solutions lacked built-in HTTP/3 probing. - Prometheus Blackbox Exporter had no native QUIC support. - Without probing at scale, Slack could not reliably measure round-trip times, detect regressions to HTTP/2, or monitor hundreds of thousands of HTTP/3 endpoints. ## Adding QUIC Support to Blackbox Exporter - Intern Sebastian Feliciano selected `quic-go` because of its adoption and first-class Go HTTP client support. - The implementation used an `http3.Transport` with TLS and QUIC configuration: ```go http3Transport := &http3.Transport{ TLSClientConfig: tlsConfig, QUICConfig: &quic.Config{}, } ``` - The new transport was attached to a standard Go `http.Client`. - The implementation preserved Blackbox Exporter’s existing configuration and composability patterns. - Sebastian open-sourced the feature and eventually got it accepted upstream. ## In-House Integration and Operational Benefits - Because upstream review could take longer than the internship timeline, Slack built an internal system around the new functionality. - Grafana now provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics. - Operators can compare protocol performance and correlate it with other telemetry. - Improved visibility supports more accurate alerts and faster debugging of HTTP/3 issues. ## Future Enhancements - **SNI routing tests:** Verify that shared edge infrastructure routes hostnames to the correct backend and presents the correct TLS certificate. - **End-to-end path visualization:** Map network hops between monitoring agents and endpoints to identify latency spikes or packet loss more precisely. ## Broader Lessons - Observability should be established before a major protocol or infrastructure migration. - Filling gaps through open source can benefit both the organization and the wider engineering community. - Supporting emerging protocols such as QUIC early helps future-proof monitoring systems. Slack recommends trying the new QUIC functionality in Prometheus Blackbox Exporter and contributing to its continued development.

network-monitoringobservabilitygoprometheus+4

cloudflareJan 26, 2026

Cable cuts, storms, and DNS: a look at Internet disruptions in Q4 2025 (opens in new tab)

In Q4 2025, Internet disruptions were driven primarily by submarine cable cuts, power failures, extreme events, technical problems, and one government-directed shutdown. Tanzania experienced a prolonged election-related blackout, while damaged international cables disrupted connectivity in Haiti, Pakistan, Cameroon, and the Dominican Republic. Cloudflare’s analysis uses major deviations in network traffic and routing announcements to identify these incidents, though it is not exhaustive. ## Government-Directed Shutdown ### Tanzania - Internet traffic fell by more than 90% on October 29 during violent protests surrounding the presidential election. - The initial shutdown lasted about 26 hours, but a second near-total outage began shortly after service briefly returned. - Connectivity did not substantially recover until November 3. - Announced IPv4 and IPv6 address space declined slightly but never disappeared entirely, indicating that Tanzania was not completely disconnected from the global Internet. - Internet and social media restrictions had also occurred ahead of Tanzania’s 2020 elections. ## Submarine and Fiber Cable Cuts ### Digicel Haiti - Digicel Haiti suffered two international fiber cuts during the quarter. - On October 16, traffic fell to nearly zero; the provider reported two cuts and restored the first fiber within several hours. - On November 25, another cut along National Road 1 caused a complete outage lasting roughly six hours. - Service was restored after repairs to the international optical fiber infrastructure. ### Cybernet/StormFiber in Pakistan - Traffic dropped to about half its expected level on October 20, while announced IPv4 address space fell by more than one-third. - The cause was a cut to the PEACE submarine cable in the Red Sea near Sudan. - Pakistan has multiple international cable routes, including IMEWE and SEA-ME-WE-4, which helped enable rapid recovery. - Traffic and address announcements returned close to normal by October 21, ahead of the provider’s October 27 restoration target. ### Cameroon and the WACS Cable - Camtel, MTN Cameroon, and Orange Cameroun experienced major disruptions on October 23 because of an incident involving the West Africa Cable System (WACS). - Traffic initially fell around 05:00 local time and recovered by approximately 22:00, although it fluctuated dramatically and sometimes dropped by 90–99%. - MTN and Orange also saw reductions in announced IP address space, while Camtel’s announcements remained stable. - The volatility may have reflected attempts to reroute traffic over other submarine cables. - Connectivity in the Central African Republic and Republic of Congo was reportedly affected as well. ### Claro Dominicana - Claro Dominicana experienced two sharp traffic declines on December 9. - Traffic eventually fell 77% below the comparable level from the previous week. - The provider attributed the disruption to two severed fiber-optic cables, which caused intermittent service and slow speeds. - Technicians restored nationwide service after repairing the cables. ## Power-Related Disruption ### Dominican Republic - A transmission-line outage on November 11 caused a major national power interruption. - Internet traffic fell by nearly 50% compared with the previous week and remained depressed until December 12. - The electrical operator later reported that 96% of national demand had been restored. - A technical report traced the blackout to a manually disconnected live line at the 138 kV San Pedro de Macorís I substation. - The resulting short circuit triggered protection systems and disconnected nearby lines, separating 575 MW of generation. ## Overall Pattern - More than 180 Internet disruptions were observed globally during 2025. - Q4 included only one government-directed shutdown, but several international cable failures caused severe regional outages. - Traffic measurements and BGP address announcements helped distinguish partial connectivity loss from complete national disconnection. - The incidents demonstrate how dependent national networks remain on a limited number of submarine cables, fiber routes, and reliable electrical infrastructure. Cloudflare’s findings suggest that network operators should diversify international cable routes, improve redundancy, and prepare for power and infrastructure failures. The Cloudflare Radar Outage Center provides a broader list of verified anomalies and confirmed outages.

network-monitoringdnsbgpipv6+4

cloudflareJan 13, 2026

What we know about Iran’s Internet shutdown (opens in new tab)

Iran’s government effectively disconnected the country from the global Internet on January 8, 2026, amid escalating nationwide protests. Cloudflare observed a near-total loss of traffic after major Iranian networks withdrew most announced IPv6 address space and then lost connectivity almost entirely. Brief access windows on January 9 quickly ended, and the shutdown remained in place through January 10. ## Background and Earlier Shutdowns - Iran has previously restricted Internet access during protests: - More than five days of disruption followed fuel-price protests in November 2019. - Connectivity was disrupted across multiple providers during protests after Mahsa/Zhina Amini’s death in September 2022. - Internet traffic had already been below normal at the beginning of 2026, suggesting connectivity problems preceded the complete shutdown. ## Connectivity Collapsed on January 8 - At 11:50 UTC, Iranian networks reduced announced IPv6 address space by 98.5%, from over 48 million `/48` blocks to roughly 737,000. - This caused IPv6’s share of human-generated traffic to fall from about 12% to 2%, before IPv6 traffic nearly disappeared later that afternoon. - Between 16:30 and 17:00 UTC, overall traffic dropped by nearly 90%. - Major providers affected included: - MCCI (AS197207) - IranCell (AS44244) - TCI (AS58224) - By approximately 18:45 UTC, traffic from Iran had fallen effectively to zero, indicating a nationwide disconnection from the global Internet. ## Brief Connectivity on January 9 - Internal traffic remained below 0.01% of pre-shutdown peaks. - Access to Cloudflare’s `1.1.1.1` DNS resolver briefly returned around 10:00 UTC, producing a short-lived spike in requests. - Several universities also regained connectivity temporarily, including the University of Tehran, Sharif University of Technology, Tehran University of Medical Science, and Tarbiat Modares University. - Traffic from these networks disappeared again by roughly 15:00 UTC. ## Filtering Changes Before the Shutdown - HTTP/3 and QUIC usage declined sharply before the full outage. - On IranCell, HTTP/3 usage fell from as high as 40% to 5% by December 31 and continued declining. - On TCI, HTTP/3 dropped below 5% around January 3. - These changes may indicate increasingly severe filtering and upgraded whitelisting, according to MahsaNet. ## Ongoing Disconnection - Since January 10, Iran’s Internet traffic has shown no significant recovery. - Traffic remains at only a fraction of one percent of previous levels. - Cloudflare continues monitoring the situation through Radar’s traffic and routing data. The available measurements strongly indicate a deliberate, nationwide Internet shutdown rather than an ordinary network failure. Cloudflare Radar’s traffic and routing pages provide the most practical way to follow any restoration or further changes.

network-monitoringdnsbgpipv6+4