cloudflare·May 1, 2026·3 min readCurated summary

Code Orange: Fail Small is complete. The result is a stronger Cloudflare network

Cloudflare’s “Code Orange: Fail Small” initiative rebuilt key parts of its infrastructure to prevent repeats of the November 18 and December 5, 2025 global outages. The work focused on safer configuration rollouts, smaller failure impact, stronger emergency access, and improved incident communication. Cloudflare concludes that the network is now more resilient, though reliability remains an ongoing effort. ## Safer Configuration Changes - Configuration changes affecting customer traffic are now deployed progressively rather than instantly across the network. - Health monitoring can detect problems early and automatically roll back faulty changes. - Cloudflare introduced **Snapstone**, a unified system that: - Packages configuration changes. - Releases them gradually. - Monitors system health in real time. - Automatically rolls back unsafe deployments. - Snapstone supports different configuration types, including data files and global control flags. - New high-risk configuration pipelines have been identified and brought under the health-mediated deployment process. ## Reducing the Impact of Failure - Product teams reviewed failure modes and removed unnecessary runtime dependencies. - Systems now use the last known good configuration where possible, following a **“fail stale”** strategy. - Where stale configuration is unavailable, teams choose between: - **Fail open:** Continue serving traffic with reduced protection or functionality. - **Fail close:** Stop processing when that is safer than continuing. - The Bot Management outage scenario would now be detected during an early rollout stage, affecting only a small amount of traffic before rollback. - Services are increasingly segmented into independent systems serving different customer cohorts. - For example, the Workers runtime deploys first to less-critical segments, such as free customers, before reaching more critical traffic. - This approach limits the blast radius of faulty deployments and adjusts rollout speed based on customer criticality. - Cloudflare plans to extend cohort-based deployment to more systems. ## Revised Break-Glass and Incident Procedures - Cloudflare audited tools needed for visibility, debugging, and emergency production changes. - It created backup authorization paths for **18 key services**, along with emergency scripts and proxies. - These pathways are designed to remain usable if Cloudflare’s own Zero Trust infrastructure is affected by an outage. - More than 200 engineers participated in an organization-wide emergency drill on April 7, 2026. - Repeated exercises are intended to ensure engineers can use emergency access procedures effectively under pressure. - Cloudflare also began improving how technical incident observations are converted into clear customer communications. Cloudflare’s changes make configuration rollouts safer, reduce failure blast radius, and improve emergency response. The practical recommendation is to treat these safeguards as ongoing operational practices rather than a one-time project, continually testing them and extending them to additional systems.

Read original(opens in new tab)

Configuration ManagementCloudflareZero TrustIncident Management+4

toss·Jan 8, 2026Original article

Managing Thousands of API/ (opens in new tab)

Toss Payments manages thousands of API and batch server configurations that handle trillions of won in transactions, where a single typo in a JVM setting can lead to massive financial infrastructure failure. To solve the risks associated with manual "copy-paste" workflows and configuration duplication, the team developed a sophisticated system that treats configuration as code. By implementing layered architectures and dynamic templates, they created a testable, unified environment capable of managing complex hybrid cloud setups with minimal human error. ## Overlay Architecture for Hierarchical Control * The team implemented a layered configuration system consisting of `global`, `cluster`, `phase`, and `application` levels. * Settings are resolved by priority, where lower-level layers override higher-level defaults, allowing servers to inherit common settings while maintaining specific overrides. * This structure allows the team to control environment-specific behaviors, such as disabling canary deployments in development environments, from a single centralized directory. * The directory structure maps files 1:1 to their respective layers, ensuring that naming conventions drive the CI/CD application process. ## Solving Duplication with Template Patterns * Standard YAML overlays often fail when dealing with long strings or arrays, such as `JVM_OPTION`, because changing a single value usually requires redefining the entire block. * To prevent the proliferation of nearly identical environment variables, the team introduced a template pattern using placeholders like `{{MAX_HEAP}}`. * Developers can modify specific parameters at the application layer while the core string remains defined at the global layer, significantly reducing the risk of typos. * This approach ensures that critical settings, like G1GC parameters or heap region sizes, remain consistent across the infrastructure unless explicitly changed. ## Dynamic and Conditional Configuration Logic * The system allows for "evolutionary" configurations where Python scripts can be injected to generate dynamic values, such as random JMX ports or data fetched from remote APIs. * Advanced conditional logic was added to handle complex deployment scenarios, enabling environment variables to change their values automatically based on the target cluster name (e.g., different profiles for AWS vs. IDC). * By treating configuration as a living codebase, the team can adapt to new infrastructure requirements without abandoning their core architectural principles. ## Reliable Batch Processing through Simplicity * For batch operations handling massive settlement volumes, the team prioritized "appropriate technology" and simplicity to minimize failure points. * They chose Jenkins for its low learning curve and reliability, despite its lack of native GitOps support. * To address inconsistencies in manual UI entries and varying Java versions across machines, they standardized the batch infrastructure to ensure that high-stakes financial calculations are executed in a controlled, predictable environment. The most effective way to manage large-scale infrastructure is to transition from static, duplicated configuration files to a dynamic, code-centric system. By combining an overlay architecture for hierarchy and a template pattern for granular changes, organizations can achieve the flexibility needed for hybrid clouds while maintaining the strict safety standards required for financial systems.

Configuration ManagementKubernetesAWSInfrastructure As Code+3

datadog·Aug 11, 2016·3 min readCurated summary

Consul at Datadog

Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery. ## Consul Server Capacity and CPU Consul servers use Raft consensus to elect a leader and coordinate the cluster. - Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds. - Frequent leadership transitions usually indicate insufficient CPU capacity. - Datadog’s approximate sizing guidance: - `m3.large`: about 300 agent nodes - `c3.xlarge`: about 500 agent nodes - `c3.2xlarge`: about 800 agent nodes - If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily. - Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal. ## Auditable Configuration Changes Consul’s key-value store is useful for distributing configuration throughout a cluster. - Configuration can be retrieved through HTTP or delivered through Consul watches. - Direct edits without an audit trail make it difficult to determine who changed a value and when. - `git2consul` distributes configuration from a Git repository, providing version control and accountability. - Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day. ## Access Control with ACLs Consul ACLs prevent unauthorized processes from modifying or deleting key-value data. - Tokens should be limited to the data and operations each process requires. - Scoped permissions reduce the impact of accidental changes. - ACLs provide an important safety boundary between services and configuration areas. ## Watches Instead of Excessive Polling Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached. - Watches notify clients when key-value data changes. - This reduces unnecessary polling and distributes updates efficiently. - Watches can sometimes trigger unexpectedly or too frequently. - Tools such as `sifter` can help protect systems from excessive watch activity. ## Using dnsmasq for Service Discovery Applications using Consul’s DNS interface can reduce load by placing `dnsmasq` between clients and Consul. - Use short DNS TTLs; Datadog commonly uses 10 seconds. - Query `dnsmasq` rather than Consul directly so repeated answers can be cached locally. - At very high request volumes, cache Consul services in an additional hosts file loaded by `dnsmasq`. - This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul. - `goshe` can collect `dnsmasq` statistics for monitoring. ## Monitoring Cluster Health Monitoring is essential for operating Consul reliably. - `consul.consul.leader.reconcile.count` should remain stable and indicate that a leader exists. - `consul.serf.events.consul_new_leader` shows leadership transitions; frequent events suggest instability. - `consul.raft.leader.lastContact` measures how recently nodes contacted the leader. - `consul.consul.dns.domain_query.count` reveals how many DNS requests are reaching Consul directly. - Also monitor CPU and network usage on Consul servers. Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and `dnsmasq` absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.

Read original(opens in new tab)

Configuration ManagementMicroservicesMonitoringConsul+4