How Discord Moved Trillions of Messages to ScyllaDB

At the start of 2022, Discord was storing trillions of messages across 177 Cassandra nodes. Cassandra was chosen for its extreme write throughput, and on that front it was doing the job well. But that design comes at the cost of expensive reads, so thousands of readers piling onto a popular channel would drag down whole nodes and slow unrelated queries. And even just keeping the cluster alive was turning into a major pain. Compaction (a key Cassandra feature for keeping reads fast) would fall behind and JVM garbage-collection pauses would force operators to reboot and babysit nodes.

To fix this, Discord swapped Cassandra for ScyllaDB, a compatible reimplementation in C++ with no garbage collector. It also put Rust data services in front of the database, which route each channel's requests to one instance and merge concurrent reads for the same message into a single query. The new cluster runs just 72 nodes instead of 177, and p99 reads fell from a 40-125 millisecond range to a steady 15 milliseconds.

We picked this blog post to break down because it shows the tradeoffs of database selection at scale, and how the fixes can be a mix of optimizing inside the database and in front of it. ScyllaDB removed the JVM garbage-collection pauses and ended the compaction firefighting, and request coalescing cut the duplicate reads on hot channels, which a new engine alone could not prevent.

Bo Ingram's original post covers the nine-day migration in detail.

Discord's messages have lived in Cassandra since 2017, and by early 2022 the cluster had grown to 177 nodes holding trillions of messages. The schema, in the minimal form the post shares, is one table:

CREATE TABLE messages (
   channel_id bigint,
   bucket int,
   message_id bigint,
   author_id bigint,
   content text,
   PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

In Cassandra, the first part of the primary key is the partition key, which decides where rows physically live. A channel is one chat room inside a Discord server, and the bucket is a window of time, so a partition holds one chat room's messages for one stretch of time, kept together on disk with copies on three nodes. message_id is the clustering key, ordering rows within the partition, and message IDs are Snowflakes, chronologically sortable, so each partition keeps its messages newest first. The time bucket exists to cap partition size. Without it, a popular channel's partition would grow without bound, and huge partitions are something Cassandra handles badly. With it, a busy channel just keeps starting fresh partitions as time rolls forward.

Most of a channel's live traffic lands on its current bucket, though, and Discord's channels are wildly unequal. A private server for a few friends sends a sliver of the traffic while the big public servers with hundreds of thousands of members move orders of magnitude more, and changing databases does not change that skew.

Cassandra favors write speed over read speed. When a write arrives, Cassandra appends it to a commit log and updates a sorted structure in memory called the memtable, then tells the client it's done. No disk page gets found and rewritten in place, which is what makes writes so cheap. When the memtable fills up, it's flushed to disk as an SSTable, an immutable sorted file that never changes after it's written. Updates don't touch old SSTables either; they just land in newer ones.

Reads pay for all this convenience. The current state of a row might be spread across the memtable and several SSTables, one holding the original insert and others holding later edits, so a read has to check each of them and merge what it finds. The more SSTables pile up, the more files every read touches. A background process called compaction merges SSTables back into fewer files, and how well compaction keeps up largely decides how fast your reads are.

Now put a popular channel on top of that storage model. Thousands of people are reading the same channel at once, every one of those reads lands on the partition holding its recent messages, and the three nodes that own that partition get buried in expensive reads. That would be tolerable if the damage stopped there, but those same three nodes also hold slices of thousands of quiet channels. Reads and writes ran at quorum, where each query waits for two of a partition's three replicas to answer, so any query unlucky enough to need a buried node sat waiting behind the hot channel's traffic. This is the hot partition problem. One busy channel, and users saw higher latency on requests that had nothing to do with it.

The team also spent a lot of time keeping compaction and garbage collection under control. Cassandra compacts SSTables in the background so reads touch fewer files, and the cluster kept falling behind on it. As uncompacted SSTables accumulated, reads had to consult more files, which slowed nodes and grew the backlog further. The obvious fix is to compact faster, but compaction fights the live workload for the same disk and CPU. A node that's already slow has nothing to spare, so turning compaction up makes latency worse before it makes anything better. That's why digging out required what the team called the gossip dance: take a node out of rotation so it can compact without serving traffic, bring it back, wait while hinted handoff replays the writes it missed, then repeat node after node until the backlog was gone.

Cassandra runs on the JVM, and its garbage collector needed constant tuning. Pauses showed up as latency spikes, and the worst of them ran long enough that operators rebooted nodes and babysat them back to health. Much of the on-call toil and many of the cluster's stability problems traced back to these pauses.

ScyllaDB is a reimplementation of Cassandra in C++. It speaks the same query language and data model, but has no garbage collector and uses a shard-per-core architecture, where each CPU core owns a slice of the node's data. Eliminating garbage collection was the headline win, since GC pauses were behind so many of the latency spikes and stability incidents; better performance and faster repairs were part of the appeal too.

A new engine could not prevent hot partitions, though, and nobody on the team expected it to. ScyllaDB stores data the same way, so reads still cost more than writes, and no database change stops thousands of clients from reading the same channel at once. The remaining lever was to send the database fewer reads, so the team built request coalescing in front of it.

To get the hot partitions under control, the team put Rust services they call data services between the API monolith and the database clusters. Each exposes about one gRPC endpoint per query the API needs, business rules stay in the API layer, and the data services handle database access.

The data services coalesce identical reads that overlap in time. Someone posts an @everyone announcement on a huge server, every member gets pinged, and much of the membership opens the app within seconds to read the same message. Without coalescing, every client issues its own read for the same row, all landing on the same partition. With coalescing, the first request starts a worker task that queries the database. Requests for the same row that arrive while that task is in flight subscribe to it instead of issuing their own queries, and the worker hands the row to every subscriber when the query returns. A simplified implementation keeps a map from each query key to the task already fetching it:

// simplified sketch
async fn get_message(channel_id: u64, message_id: u64) -> Message {
    if let Some(task) = in_flight.get(&(channel_id, message_id)) {
        return task.subscribe().await; // join the query already running
    }
    let task = spawn_query(channel_id, message_id); // one real DB read
    in_flight.insert((channel_id, message_id), task.handle());
    task.await
}

Why put coalescing in a data service instead of ScyllaDB? ScyllaDB already shares lower-level work through its caches, but it doesn't offer full-query consolidation for identical CQL requests. Discord's data service has a simpler job: it knows when two RPCs are asking for the same message, and consistent hashing sends both to the same instance. They can be merged before either becomes a database request. This isn't a fundamental limitation of databases. Vitess's query consolidator, for example, does something similar in front of MySQL. The application layer gave Discord the routing key and request semantics it needed.

Coalescing only works if the duplicate requests actually meet each other, though; scattered across dozens of instances, nothing merges. So upstream, requests route to data-service instances with consistent hashing, where a hash of a routing key picks the instance and the mapping stays stable as instances come and go. For messages the routing key is channel_id, so every request for a given channel lands on the same data-service instance.

All of this shipped while Cassandra was still the primary store. Hot partitions and latency spikes still happened, just not as frequently, and the reduction kept Cassandra manageable while the team finished testing and tuning ScyllaDB.

ScyllaDB took over as the primary database in May 2022, after the team validated the new cluster by sending a small percentage of production reads to both databases and comparing the results. The messages cluster went from 177 Cassandra nodes to 72 ScyllaDB nodes, each holding 9 TB on average instead of 4. We do wonder about that density. A 9 TB node that dies takes a lot longer to restream than a 4 TB one, so consolidating onto fewer, fatter nodes trades recovery time for hardware savings. Scylla's faster repairs presumably made that math comfortable, but the post never shows the reasoning.

The p99 for fetching historical messages fell from a 40-125 millisecond range to a steady 15, and p99 inserts went from 5-70 milliseconds to a steady 5. One thing the post leaves tangled is how much of that improvement belongs to ScyllaDB versus the data services in front of it. Coalescing was already smoothing out hot partitions on Cassandra before the migration, so some slice of the headline numbers was earned by the Rust layer, and we'd have loved to see the split. The team stopped spending weekends firefighting and no longer pulled nodes out of rotation to keep the cluster up, and new workloads started landing on the database. At the end of 2022, every goal in the World Cup final showed up as a visible spike in the message-send graph, and the database didn't care.

Hot partitions are a traffic problem. A popular channel concentrates reads on whichever partition holds its messages, and swapping the engine underneath changes how expensive each of those reads is, never how many of them arrive. Coalescing and hash routing shipped while Cassandra was still the primary store, and hot-partition incidents were already thinning out before a single row had moved.

That makes the fix portable to any storage system that suffers when reads concentrate. Pick a routing key that matches where your traffic clusters, the way Discord picked the channel, and use consistent hashing so every request for that key lands on the same service instance. The duplicate requests now meet each other in one process, where merging them is easy, and a thousand simultaneous readers cost the database a single query. However large the crowd in front of it grows, the database sees load proportional to the number of distinct rows being asked for.

ScyllaDB addressed the operational problems inside the database. With no JVM there were no more garbage-collection pauses or reboot babysitting, and the new cluster ran 72 denser nodes with much steadier tail latencies.