2023-03-08 incident: A deep dive into the platform-level recovery

Datadog’s March 8, 2023 outage removed 60% of its compute capacity, forcing teams to restore infrastructure in stages while accounting for regional and cloud-provider differences. In EU1, recovery depended on rebooting affected nodes, restoring Kubernetes control planes in a strict hierarchy, and gradually bringing application capacity back online. Scaling afterward exposed infrastructure limits that had not been considered during normal operations.

EU1 Platform Recovery

• A system patch disconnected affected EU1 nodes from the network, but the nodes could be recovered through reboots.
• Recovery was initially slowed by the lack of observability and unavailable Kubernetes APIs.
• Datadog operates:
  • Parent clusters, which host the control-plane pods for other clusters.
  • Child clusters, where Datadog applications run.
• This hierarchy allows Datadog to use Kubernetes deployment, replacement, rolling-update, and autoscaling capabilities for child-cluster control planes.
• Parent-cluster control planes run on VMs and are managed with systemd.

Restoring Kubernetes Clusters

Because both parent and child environments were affected by the Ubuntu 22.04 issue, recovery had to follow a strict sequence:

• Parent control planes: Nodes running Cilium were rebooted to restore network connectivity. This finished by 08:45 UTC.
• Child control planes: All parent-cluster nodes hosting child control-plane pods were rebooted. This finished by 09:30 UTC.
• Application nodes: Thousands of instances across dozens of child clusters were restarted.
  • Recovery reached 60% by 10:20 UTC.
  • All application nodes were restored by 12:05 UTC.
  • Restarts were prioritized by workload importance and paced to avoid overwhelming Kubernetes control planes.

Scaling Capacity and Recovering Backlogs

After restoring the clusters, Datadog needed substantial additional capacity to process data buffered during the outage.

• EU1 hit a Google Cloud mesh limit of 15,500 VM instances at 14:18 UTC.
• Instance creation failures became apparent around 15:00 UTC.
• Datadog had not checked this documented limit before the incident, but Google Cloud quickly raised it after Datadog submitted a high-priority request.
• Autoscaling also exhausted the IP capacity of subnets used by three log- and trace-processing clusters.
• These clusters normally used about 35–45% of their IP capacity, but the backlog caused autoscaling to request more than twice their usual replica counts, filling the subnets.

Practical Lessons

The recovery demonstrated that restoring compute capacity is not enough: teams must also understand dependency order, control-plane architecture, cloud-provider quotas, and network-address limits. Capacity planning should account for severe backlog-driven scaling, not just normal operating utilization, and documented infrastructure limits should be validated before emergencies occur.