How Discord Indexes Trillions of Messages

Discord’s original Elasticsearch-based search system worked well for billions of messages but became fragile as message volume and cluster size grew. Redis queues could drop messages, bulk operations failed too broadly, large clusters were difficult to operate, and individual indices could hit Lucene’s roughly two-billion-document limit. Discord’s response was to modernize the platform with Kubernetes, the Elastic Kubernetes Operator, and a multi-cluster “cell” architecture built from smaller clusters.

The Original Search Architecture

• Messages were stored in Elasticsearch indices distributed across two clusters.
• Data was sharded by Discord server (guild) or direct message, keeping each guild’s messages together for efficient queries.
• Messages were indexed lazily because not every message is searched.
• Redis-backed queues supplied workers with message batches for Elasticsearch bulk indexing.

Problems with the Existing System

Redis Queue Message Loss

• The realtime indexing queue relied on Redis.
• When Elasticsearch failures caused the queue to back up, Redis CPU usage could reach its limit.
• Once overloaded, Redis began dropping messages, making the indexing pipeline unreliable.

Fault-Intolerant Bulk Indexing

• A batch could contain messages belonging to many different Elasticsearch indices and nodes.
• A batch of 50 messages might fan out to dozens of nodes.
• If one message failed because its target node was unavailable, Elasticsearch treated the entire bulk request as failed.
• All messages were then re-enqueued, increasing queue pressure.
• In a 100-node cluster with batches of 50 messages, a single failed node gave each batch roughly a 40% chance of encountering a failure.

Large-Cluster Overhead

• Adding nodes and indices enabled horizontal scaling but increased coordination overhead.
• Bulk operations fanned out across more nodes, slowing indexing.
• Larger clusters also had a higher probability that some node would fail.

Difficult Upgrades and Restarts

• The system lacked sufficient resilience to individual node outages, making rolling restarts unsafe.
• Clusters exceeding 200 nodes and containing terabytes of data would have taken too long to drain gracefully.
• Discord therefore remained on outdated operating-system and Elasticsearch versions.
• Addressing the Log4Shell vulnerability required taking the entire search system offline while every node was restarted.

Oversized Indices

• Some indices accumulated messages from extremely large guilds.
• Each Elasticsearch index is backed by a Lucene index with a limit of approximately two billion documents.
• Once that limit was reached, all further indexing failed.
• Discord temporarily recovered by identifying and deleting guilds created primarily for message spam, but this was not viable for legitimate high-volume communities.

Moving Elasticsearch to Kubernetes

• Discord chose Kubernetes to improve operational flexibility and resource efficiency.
• The Elastic Cloud on Kubernetes (ECK) Operator could define cluster topology and configuration declaratively.
• Kubernetes would automate operating-system upgrades.
• ECK provided tools for safer rolling restarts and Elasticsearch upgrades.
• This marked Discord’s first move toward managing stateful Elasticsearch infrastructure on Kubernetes.

Smaller Multi-Cluster Cells

• Discord planned to replace very large clusters with a larger number of smaller Elasticsearch clusters.
• Smaller clusters reduce coordination overhead and limit the impact of individual node failures.
• A cell-based design also provides a more manageable scaling and operational boundary than clusters with hundreds of nodes.

Discord’s experience demonstrates that scaling Elasticsearch is not only a matter of adding nodes. Reliable operation requires isolating failures, avoiding oversized indices and fan-out-heavy batches, and designing deployment infrastructure that supports upgrades without taking search offline.