Discord Search at Trillion Scale

October 18, 202514 min read

Welcome to Engineering Unpacked, the series where I read a very smart engineering blog, open twelve tabs to understand it, and then explain it without making you do the same side quest.

Today we are unpacking Discord's post about indexing trillions of messages. The original was written by Vicki Niu. This is my plain-English breakdown, plus the search and distributed-systems lessons I am placing directly into my “scale is mostly failure management” notebook.

Discord's old message-search platform indexed billions of messages. The new one indexes trillions, runs forty Elasticsearch clusters, and answers most searches in under 100 milliseconds.

Naturally, the journey involved Redis dropping messages, 200-node clusters that could not be safely restarted, and communities so active they reached a two-billion-document ceiling.

Some servers have a general channel. Some servers test the known limits of information storage.

What Does Message Indexing Mean?

Discord stores messages so they can be displayed in a channel. That does not automatically make them fast to search.

Imagine opening every message ever posted and scanning them one by one whenever somebody searches for “the build is definitely fixed.” The results would arrive shortly after the next programming language is invented.

A search index creates additional structures optimized for finding documents. At a very simplified level, it records which words and fields appear in which messages, allowing Elasticsearch to narrow the search without reading the entire history.

The tradeoff is that every new message must travel through an indexing pipeline:

new message → queue → index worker → Elasticsearch index → searchable result

Search therefore has its own copies of data, delivery guarantees, backlogs, partitioning rules, and failure modes. The source message may be safely stored while its search index is missing or delayed.

At small scale, that can be a background job. At Discord scale, it becomes an infrastructure platform with opinions about geography, queue durability, cluster coordination, and communities that apparently type faster than Lucene was emotionally prepared for.

The original design made sense

Discord described its earlier search system in 2017. It used Elasticsearch and divided messages across indices based on their Discord server, called a guild in Discord's internal terminology, or their direct message conversation.

Keeping a guild's messages together was good for queries. Searching one guild usually meant asking one shard instead of broadcasting work across the entire cluster.

Discord ran two large Elasticsearch clusters. It also indexed messages lazily because not every user searches their history. A Redis-backed queue held indexing work, and workers pulled messages in batches to take advantage of Elasticsearch's efficient bulk API.

This design had several sensible ideas:

colocate data that is searched together;
avoid indexing data until somebody needs it;
buffer background work in a queue;
write in batches instead of one message at a time.

The system served Discord for years. It did not suddenly become foolish. The scale changed until some locally good decisions began interacting in globally painful ways.

That is the recurring plot of engineering blogs and, occasionally, my personal calendar.

The Redis Queue Became a Crisis

The real-time indexing queue used Redis. Under normal load, this worked well.

When Elasticsearch had a node failure, however, indexing slowed and the queue grew. As enough messages accumulated, Redis used more CPU until the queue itself began dropping work.

That is a particularly unpleasant failure chain:

search node fails
  → indexing slows
  → queue backlog grows
  → queue exhausts resources
  → messages disappear from the indexing pipeline

The search cluster had a temporary problem. The queue turned it into permanent missing data.

A buffer is supposed to separate the producer's speed from the consumer's speed. If the buffer cannot retain a backlog for as long as the downstream system may be unhealthy, it is not truly isolating the two systems. It is providing a short countdown before they fail together.

Discord eventually moved the pipeline to Google Cloud Pub/Sub. That gave it durable delivery and the ability to tolerate large backlogs. An Elasticsearch incident could still delay search freshness, but it would no longer silently remove messages from the future index.

Slow and recoverable beats fast and missing.

A Batch of Fifty Broke Too Much

The old workers bulk-indexed batches of messages, which sounds efficient because it is.

The problem was not the batch size. It was what the batch contained.

A single batch of fifty messages could contain messages destined for fifty different indices on fifty different Elasticsearch nodes. The bulk operation fanned out across the cluster. If one item failed because its destination node was unavailable, the entire batch was considered failed and every message was placed back on the queue.

Suppose a cluster has 100 nodes and exactly one is down. If fifty messages are distributed evenly and independently, the chance that a batch touches the failed node is roughly:

1 - (99 / 100)⁵⁰ ≈ 39.5%

One percent of the nodes being unavailable could therefore cause around forty percent of bulk operations to fail.

Nothing about the arithmetic is unfair. The architecture simply gave one failure fifty opportunities to join every batch.

The retries made the queue grow, the larger queue put pressure on Redis, and the pipeline began producing work about producing work instead of indexing messages.

This is a beautiful example of accidental coupling. Messages headed to unrelated destinations shared one success condition. A problem on node 73 caused healthy writes for nodes 4, 18, and 92 to repeat for no useful reason.

Bigger Clusters, Bigger Problems

Discord initially scaled Elasticsearch horizontally by adding indices and nodes. Eventually, its clusters grew beyond 200 nodes.

More nodes provided more capacity, but they also increased cluster-state overhead, coordination work, the number of destinations touched by bulk operations, and the probability that some node was unavailable at any given moment.

The master nodes began running out of memory. That caused indexing failures, growing queues, slower queries, and timeouts.

Worse, the platform was so sensitive to individual node availability that routine rolling restarts and software upgrades became dangerous. Gracefully draining enormous nodes would take too long, so the clusters remained on old operating-system and Elasticsearch versions.

When the Log4Shell vulnerability needed patching, Discord had to take message search fully offline for maintenance while every Elasticsearch node restarted with a safer configuration.

That is when “we cannot restart this system” stops sounding like impressive uptime and starts sounding like the system owns the company.

Operational work is not separate from scalability. A platform that can serve today's traffic but cannot safely upgrade, restart, or repair itself has already found its ceiling.

One Guild Hit Lucene's Limit

Each Elasticsearch index is backed by Lucene. A Lucene index has a maximum document count of roughly two billion.

Some enormous Discord guilds eventually posted enough messages to reach that boundary. Once an index hit the limit, new indexing operations failed.

Discord's temporary recovery option was to work with its Safety team to find spam-focused guilds and delete them from the index. That bought time, but it was not a strategy for legitimate communities with multi-billion-message histories.

The original partitioning rule kept one guild together for fast queries. It was excellent for almost every guild and physically impossible for the outliers.

This is an important scale lesson: a design can be correct for 99.99 percent of tenants and still need a separate architecture for the final 0.01 percent. The answer does not have to be making every normal tenant pay the complexity cost of the largest one.

Discord named those outliers BFGs, or Big Freaking Guilds.

I appreciate any architecture vocabulary that sounds equally at home in a design document and a boss battle.

The New Design Got Smaller

Discord rebuilt the platform around many smaller Elasticsearch clusters running on Kubernetes through the Elastic Cloud on Kubernetes operator.

Instead of treating one enormous cluster as the unit of the platform, it introduced a logical cell containing multiple smaller clusters.

Each Elasticsearch cluster stays manageable. The cell provides a higher-level grouping that Discord can use for a search workload, such as guild messages, direct messages, or BFGs.

Smaller clusters reduce cluster-state and master-node overhead. They also reduce the blast radius of an unhealthy cluster and make rolling maintenance practical. The Kubernetes operator handles declarative topology, orchestration, operating-system upgrades, and safe Elasticsearch restarts.

Today the platform runs forty Elasticsearch clusters containing thousands of indices.

Inside each cluster, Discord separates node responsibilities:

three master-eligible nodes handle coordination, with one in each zone;
at least three ingest nodes preprocess and route writes, again spread across zones;
data nodes receive enough heap for indexing and queries;
primary and replica shards live in different zones.

The roles have different resource needs, so Kubernetes schedules them onto different machine types and node pools.

This is more than “put Elasticsearch in containers.” Discord changed the failure domain and operating model. Kubernetes and the operator then automated that model.

Containers do not make a 200-node cluster less emotionally complicated by themselves.

Route first, then batch

Discord kept bulk indexing, but changed how messages are grouped.

The new Pub/Sub message router computes a Destination for every message: the Elasticsearch cluster and index where that message belongs. It maintains a channel and a Tokio task for each active destination.

Conceptually, the flow looks like this:

Pub/Sub stream
  ├─ destination A → channel A → batch A → index A
  ├─ destination B → channel B → batch B → index B
  └─ destination C → channel C → batch C → index C

Messages are routed first. Each destination task then collects its own chunk and sends a bulk operation to one Elasticsearch destination.

If index B is unavailable, batches for A and C do not fail merely because they happened to be pulled from Pub/Sub at the same time.

The system still gets the throughput benefits of batching, but the batch boundary now matches the failure boundary.

That sentence is the star of the article.

Batching is not only about putting fifty things in a box. The things in the box should share a destination, a retry policy, and a reason to succeed or fail together.

Search Shape Decides Data Shape

The cell abstraction also let Discord introduce a long-requested feature: searching across all of a user's direct messages.

The old system stored direct messages by conversation. That made searching one conversation efficient, but searching every DM would require a query to fan out across all of the user's conversations.

In the new design, guild messages remain partitioned by guild_id, while direct messages are indexed by user_id in a separate user-dm-messages cell.

This means a message between two users is indexed twice: once for each recipient. Discord spends more storage during ingestion so a future search can go directly to the data belonging to one user.

That is deliberate denormalization:

write more copies;
use more storage;
avoid unbounded query fanout;
make the important read fast and predictable.

There is no universally correct partition key. The right key depends on the query you need to answer.

“Store messages by conversation” and “store messages by user” are both sensible. They optimize different questions. Discord's cells allow those different search shapes to live in different physical layouts instead of forcing one compromise onto every workload.

BFGs get their own rules

For ordinary guilds, Discord prefers an Elasticsearch index with one primary shard. All of the guild's messages remain on one node, so a query avoids cross-shard fanout and coordination.

For a BFG with billions of messages, that same locality becomes a limitation. One Lucene index cannot grow forever, and the search is large enough to benefit from parallel work across several shards.

Discord therefore moves BFGs into a dedicated cell whose indices have multiple primary shards.

The migration is gradual and reversible:

Detect a guild approaching the Lucene document limit.
Create a new BFG index with twice the previous primary-shard count.
Dual-index new messages into the old and new locations.
Backfill historical messages while searches continue using the old index.
Switch query traffic after the new index catches up.
Stop writing to the old index once the new path proves reliable, then clean up the old data.

This is the same safe-migration pattern that keeps appearing in good infrastructure work: create the new home, mirror live changes, copy history, verify, move reads, and only then remove the old home.

The special BFG cell also protects normal guilds. Expensive, highly parallel searches for the largest communities no longer compete with everyone else on the same resources.

Outliers receive the complexity they require. The common path stays simple and fast.

The numbers after the rebuild

Discord reports that its new search infrastructure now:

indexes trillions of messages;
provides twice the indexing throughput of the legacy platform;
reduced median query latency from 500 milliseconds to under 100 milliseconds;
reduced p99 query latency from one second to under 500 milliseconds;
runs forty Elasticsearch clusters with thousands of indices;
performs rolling restarts and upgrades automatically without service impact.

The latency improvements are lovely. The ability to upgrade without an outage may be even more important.

Performance tells you the platform works on a good day. Safe maintenance tells you it can continue working next year.

System-design lessons I am stealing

Here is the aggressively condensed notebook page:

Align batch boundaries with failure boundaries. Work headed to unrelated destinations should not share one all-or-nothing result.
Use queues that survive the backlog you expect during an outage. A temporary consumer failure must not become permanent data loss.
Prefer cells when one giant cluster becomes an operating liability. Smaller failure domains simplify coordination, recovery, and upgrades.
Keep related query data together until the outliers make that impossible. Locality is valuable, but hard limits remain hard.
Let search patterns choose partition keys. Guild search and cross-DM search need different layouts.
Spend storage to save query fanout when the economics make sense. Duplicate writes can buy predictable reads.
Separate exceptional tenants from the common path. BFGs need multiple shards; almost every other guild is faster with one.
Migrate live data with dual writes and a historical backfill. Move reads only after the new index is complete and proven.
Treat maintenance as a product capability. Restarts and upgrades should be routine, not company events.
Scaling up changes probabilities. One failed node can affect far more than one percent of work when every batch fans out widely.

The five-line version

Discord's original Elasticsearch platform kept guild messages together and indexed them in bulk, but Redis could drop queued work and randomly mixed batches made one node failure poison a huge share of writes.

Its clusters grew beyond 200 nodes, became difficult to coordinate or restart, and left Discord stuck on old software until even critical maintenance required a search outage.

Discord moved to durable Pub/Sub delivery and destination-aware batching, then replaced giant clusters with Kubernetes-managed cells of smaller Elasticsearch clusters.

It partitions guild search by guild, cross-DM search by user, and multi-billion-message BFGs across multiple shards in their own cell.

The result is trillions of indexed messages, twice the indexing throughput, sub-100-millisecond median queries, and upgrades that no longer require turning search off and hoping everyone enjoys the break.

And that is how Discord scaled message search by making the important pieces smaller while giving the truly enormous guilds their own cell.