datadogOct 15, 2025

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | Datadog (opens in new tab)

Following a major 2023 incident that caused a near-total platform outage despite partial infrastructure availability, Datadog shifted its engineering philosophy from "never-fail" architectures to a model of graceful degradation. The company identified that prioritizing absolute data correctness during systemic stress created "square-wave" failures, where the entire platform appeared down if even a portion of data was missing. By moving toward a "fail better" mindset, Datadog now focuses on maintaining core functionality and data persistence even when underlying infrastructure is compromised. ## Limitations of the Never-Fail Approach * Classical root-cause analysis focused on a legacy, unsupervised global update mechanism that disconnected 50–60% of production Kubernetes nodes. * While the "precipitating event" was easily identified and disabled, the engineering team realized that fixing the trigger did not address the systemic fragility that caused a binary (up/down) failure pattern. * Prioritizing absolute accuracy meant that systems would wait for all data tags to process before displaying results; under stress, this caused the UI to show no data at all rather than "almost correct" data. * Sequential queuing, aggressive retry logic, and node-specific processing requirements exacerbated the bottleneck, preventing real-time recovery. ## Prioritizing Graceful Degradation * The incident prompted a shift away from relying solely on redundancy to prevent outages, acknowledging that some level of failure is eventually inevitable at scale. * Engineering priorities were redefined to ensure that data is never lost (even if delayed) and that real-time data is processed before stale backlogs. * The platform now aims to serve partial-but-accurate results to customers during an incident, providing visibility rather than a complete blackout. * Implementation is handled as a company-wide program where individual product teams adapt these principles to their specific architectural needs. ## Strengthening Data Persistence at Intake * Analysis revealed that data was lost during the outage because it was stored in memory or on local disks before being replicated to persistent stores. * The original design favored low-latency responses by acknowledging receipt of data before it was fully replicated, making that data unrecoverable if the node failed. * Downstream failures caused intake nodes to overflow their local buffers, leading to data loss even on nodes that remained online. * New architectural changes focus on implementing disk-based persistence at the very beginning of the processing pipeline to ensure data survives node restarts and downstream congestion. To build truly resilient systems, engineering teams must move beyond trying to prevent every possible failure trigger. Instead, focus on designing services that can survive partial infrastructure loss by prioritizing data persistence and allowing for degraded states that still provide value to the end user.

graceful-degradationk8ssystem-designincident-analysis+4

datadogOct 15, 2025

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog (opens in new tab)

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

graceful-degradationdatabase-designk8sdata-loss-prevention+4