discord·Apr 29, 2026·3 min readCurated summary

You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage

Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away. ## The Infrastructure Background - Discord is migrating stateful Elixir services to Kubernetes. - Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls. - Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely. - The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events. - To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods. ## The Session Loss - The resource change was deployed to the first availability zone at 12:13 PDT. - Kubernetes terminated half of that zone’s pods because of the reduced replica count. - A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first. - Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff. - The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.” ## How Elixir Monitoring Amplified the Failure - Discord relies heavily on Elixir `GenServer` processes, which process one mailbox message at a time. - Process monitors notify dependent processes whenever a monitored process exits. - The sudden loss of sessions therefore generated a large number of `{:DOWN, …}` notifications throughout the realtime infrastructure. - Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect. ## Reconnecting Users - The gateway handles websocket ingress and egress, creating sessions and maintaining client connections. - Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss. - When a session disappears, the gateway immediately instructs the client to reconnect. - It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb. The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.

Read original(opens in new tab)

ElixirKubernetesDistributed SystemsWebsockets+2

discord·Mar 4, 2026·2 min readCurated summary

Tracing Discord's Elixir Systems (Without Melting Everything)

Discord runs each guild independently using Elixir’s concurrency model, helping chats and reactions feel instantaneous at scale. When a guild becomes overloaded, metrics and logs can reveal activity spikes but often fail to show the actual user experience or downstream effects. To fill this gap, Discord built distributed tracing for its Elixir services and integrated it without downtime. ## Guild-Level Isolation and Outages - Each Discord server, or “guild,” runs independently from others. - This isolation supports high concurrency and limits failures to individual guilds. - A guild may become laggy or go offline when user activity exceeds its processing capacity. - If it cannot recover automatically, on-call engineers investigate the incident. ## Limits of Metrics and Logs - Engineers inspect metrics showing: - How often each user action type is processed. - How long processing takes. - These metrics can identify bursts of activity, such as sudden waves of reactions or messages. - However, they do not clearly show how those conditions affected users. - Metrics are comparable to a car dashboard: they expose internal conditions but not necessarily the consequences. ## Guild Timings - Discord’s custom “guild timings” tool records the amount of each minute spent processing different action types. - The data is stored in memory and provides more detail than standard metrics. - Its high volume makes long-term storage impractical, so data is frequently rotated. - The tool also focuses on guild-local processing and does not capture downstream effects or complete end-to-end request experience. ## Building Distributed Tracing for Elixir - Distributed tracing shows how long each part of an operation takes across services. - Other Discord teams had already benefited from tracing and application performance monitoring. - Typical tracing systems propagate operation context through metadata such as HTTP headers. - Elixir’s built-in communication mechanisms do not provide an equivalent metadata layer. - Discord therefore built its own mechanism for propagating tracing information between services. ## Deployment Without Downtime - Although the tracing system changed how Discord services communicate, it was integrated without taking the platform offline. - The result gives engineers a more complete view of request paths, helping them understand both the source of guild problems and their impact on users. Discord’s experience suggests that detailed distributed tracing is essential when local metrics and logs cannot explain end-to-end behavior. For highly concurrent systems, investing in tracing infrastructure tailored to the platform can significantly improve incident diagnosis without requiring disruptive deployment changes.

Read original(opens in new tab)

ElixirDatabase DesignApplication Performance MonitoringDistributed Tracing+3