You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage

Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away.

The Infrastructure Background

• Discord is migrating stateful Elixir services to Kubernetes.
• Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls.
• Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely.
• The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events.
• To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods.

The Session Loss

• The resource change was deployed to the first availability zone at 12:13 PDT.
• Kubernetes terminated half of that zone’s pods because of the reduced replica count.
• A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first.
• Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff.
• The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.”

How Elixir Monitoring Amplified the Failure

• Discord relies heavily on Elixir GenServer processes, which process one mailbox message at a time.
• Process monitors notify dependent processes whenever a monitored process exits.
• The sudden loss of sessions therefore generated a large number of {:DOWN, …} notifications throughout the realtime infrastructure.
• Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect.

Reconnecting Users

• The gateway handles websocket ingress and egress, creating sessions and maintaining client connections.
• Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss.
• When a session disappears, the gateway immediately instructs the client to reconnect.
• It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb.

The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.