How Slack Put Kafka in Front of Its Redis Job Queue

On its busiest days, Slack’s async job queue handles more than 1.4 billion jobs, peaking at 33,000 per second. It powers nearly everything too slow for a web request, including posting messages, sending push notifications, generating URL unfurls, triggering calendar reminders, and running billing calculations.

That queue was built entirely in Redis, which held both the backlog and the dispatch bookkeeping. That is until a database slowdown made jobs pile up until Redis hit its memory limit. At that point, the queue could neither accept new jobs nor, since dequeuing also needed a little free memory, hand out the old ones.

To prevent this from happening again, they added Kafka in front of Redis as a durable buffer plus two new Go services to move jobs in and out, keeping the application's queue logic untouched. As a result, a backlog now piles up on disk while operators throttle the flow into Redis.

We picked Slack's write-up because it's a canonical example of putting a durable log in front of a fast dispatch layer in order to handle bursts.

Slack's job queue runs on a fleet of Redis clusters, with pools of worker machines polling those clusters for new work. When something in Slack needs to happen later, the web app builds a job identifier from the job type and its arguments, then hashes it with the logical queue name to pick which Redis host the job lands on. That host runs limited deduplication, discarding the request if an identical identifier is already waiting. Workers pull jobs off the pending queue, moving each onto an in-flight list before spawning an async task to run it. Finished jobs come off the list, and failures retry until they land in a permanently-failed list that humans repair by hand.

Slack doesn't say which Redis structures they used or exactly why dequeuing cost grew with queue length, but a typical Redis task queue shows how that happens. Waiting jobs sit in a list per logical queue that enqueues push onto. Deduplication needs an index you can check by job ID, a set or a hash maintained alongside the pending jobs. Moving a job to an in-flight list before execution means it remains recoverable if the worker dies. RPOPLPUSH makes that handoff cheap, popping the oldest pending job and pushing it onto another list at constant cost. Removing one particular job by value is a different operation, a command like LREM that scans the list end to end, and a scan like that on every dequeue produces the linear cost Slack described.

Jobs run from a few milliseconds to several minutes, and the queue had scaled from Slack's earliest days without the core architecture changing. The same finite pool of RAM held both the buffer that absorbs bursts of new work and the working space where dispatch, dedup, and retries get tracked.

Resource contention in the database layer slowed job execution, so jobs accumulated in Redis until it hit its configured maximum memory. Enqueues began failing, and with them every Slack feature that depended on the queue. The dequeue path then prevented the queue from recovering. Pulling a job off the queue moves it onto a processing list, and that move needs free memory a full Redis doesn't have, so even after the database recovered the queue stayed stuck, and it took extensive manual intervention to revive.

Memory exhaustion was only one of the problems the post-mortem turned up. Every job-queue client connected to every Redis instance, a complete bipartite graph. Workers couldn't be scaled independently either, because each added worker adds polling load on Redis, so adding execution capacity could push an already overloaded Redis further under. The linear-cost dequeue made a second feedback loop, since a longer queue slows every dequeue at the moment draining matters most. And because the queue's exact semantics were never pinned down, engineers hesitated to build on it or to change dedup behavior that many jobs depended on.

The post-mortem set the requirements. The backlog had to live somewhere durable that couldn't run out of memory whether jobs flooded in or drained too slowly, scheduling needed real rate limits and priorities, and execution capacity had to scale without adding polling load on Redis. Slack put Kafka in front of Redis instead of replacing Redis with it. Why run two queue-shaped systems? Why not just make Kafka the queue?

Slack could have rebuilt the queue directly on Kafka. It's a durable append-only log, split into partitions, where each consumer tracks its own read position as an offset, and the log lives on disk, bounded by a retention window rather than by RAM, so a burst of enqueues just makes the log longer. An offset is a position in a partition, not per-job state, so per-job acknowledgment, retries, dedup, and in-flight tracking have to be built on top, retries as re-enqueues onto the topic, dedup as a small keyed store, and plenty of teams run job systems on Kafka in exactly this way. Slack had all of those semantics already built in Redis, with years of application code depending on their exact behavior, so rebuilding on Kafka alone would have turned a narrowly scoped availability fix into a rewrite.

RabbitMQ, SQS, and beanstalkd all persisted to disk, and all had the same problem, since each brings its own delivery semantics and adopting one meant the same rewrite against a different API. Turning on Redis persistence wouldn't have helped either, because a Redis dataset has to fit in RAM whether or not it's also written to disk, and the outage was a capacity problem, not a durability one. And keeping Redis alone was the design that had just failed, where the burst buffer and the dispatch workspace compete for the same RAM and overload ends in lockup rather than lag.

So Slack made what they called the "minimum viable change". The web app enqueues into Kafka, which durably buffers everything, a relay drains Kafka into Redis at a controlled rate, and Redis keeps providing the short queue and the semantics the application understands. Neither the enqueue interface nor the worker side changed. Slack first protected the enqueue path, then planned to revisit scheduling and worker scaling after running the new path in production.

Slack wrote two new stateless Go services to run the new path. Kafkagate is the enqueue path from the web app into Kafka. The interface is a plain HTTP POST, simplified here:

POST /enqueue
{
  "topic": "jobs-cluster-07",
  "partition": 12,
  "content": { "type": "url_unfurl", "args": { ... } }
}

Kafkagate holds persistent broker connections, follows partition leadership as it moves, and writes synchronously, so the caller always gets a positive ack or an error. It waits only for the partition leader to acknowledge a write, not for replication, taking the lowest latency in exchange for a small chance of losing a job if a broker dies before replicating, a tradeoff Slack judged right for most jobs. Enqueuing hosts prefer a Kafkagate in their own availability zone, failing over to others when needed. This also retired the enqueue half of the connection fan-out, since the web app now talks to a nearby gateway instead of holding connections to fifty Redis clusters.

JQRelay is the drain, one instance relaying one Kafka topic into its corresponding Redis cluster. Coordination runs through Consul, a service that provides distributed locks. On startup an instance tries to acquire the lock for a topic, and if it wins it relays all of that topic's partitions. If it ever loses the lock, it releases everything and restarts so another instance can take over, and since instances run in an auto-scaling group, replacements go through the same flow. The result is one relay responsible for each topic.

JQRelay tracks progress with Kafka's per-partition commit offsets, and the core loop, simplified, advances an offset only after the job is safely in Redis:

for job in consume(topic, partition):
    write_to_redis(job)       # retry indefinitely while Redis is down
    commit_offset(partition)  # only after the Redis write succeeds

If Redis is down, the relay retries until it returns or is replaced while the log absorbs new work behind it. If a specific job errors, JQRelay re-enqueues it to Kafka, which keeps the queue moving without losing the job. The rate limits live in Consul, picked up through its watch API. During a buildup the web app keeps enqueueing at full speed while operators turn the relay down to match what workers can execute.

The cluster ran 16 brokers, with 32 partitions per topic, replication factor 3, two days of retention, and rack-aware replication mapped to availability zones. Unclean leader election, where Kafka accepts a stale replica as leader to restore availability at the risk of losing acknowledged writes, was enabled. The team sized the cluster with load tests at expected production rates, then ran failure drills covering broker loss, a forced unclean leader election, and a full cluster restart, and every scenario met their availability goals.

The rollout itself started with double writes, every job going to both the old Redis path and the new Kafka path while JQRelay ran in what they called "shadow" mode, reading everything from Kafka and dropping it, which put real production traffic through every new component with zero user impact. They counted jobs at each hop, web app to Kafkagate to Kafka to Redis, and ran heartbeat canaries, one per partition per minute across all 1,600 partitions (50 topics of 32, one per Redis cluster), with alerts on end-to-end flow and timing. The new path served Slack internally for a few weeks before rolling out to customers one job type at a time.

One snag came from putting a second runtime into a PHP-only pipeline. JQRelay decodes and re-encodes each job's JSON, and Go by default escapes <, >, and & into unicode entities while PHP escapes /. The same data structure produced different bytes depending on the runtime, and it caused real trouble, "heartache" in the team's own word.

Slack's outage was because of one structural fact: the burst buffer and the dispatch workspace shared a single bounded pool of RAM. An in-memory queue is fast and its job semantics are easy to build, but the backlog, the dedup index, and the in-flight bookkeeping all compete for the same memory with no durable backstop, and overload ends in lockup rather than lag. The fix separated the two roles. Kafka holds the backlog on disk, where it's bounded by retention rather than RAM, and Redis holds only the short dispatch window that workers actually drain, with operators throttling the relay between them. As far as we can tell, that's still largely the system Slack runs today: Kafka for durable buffering, Redis for short-term dispatch state, workers for execution.

That assembly was fairly novel in 2017. Today, few teams of scale would start with their entire backlog in an in-memory queue, and fewer would build the Kafka-to-Redis bridge by hand, but only because modern tools have evolved to absorb the separation Slack built. Kafka with consumer groups serves as the backbone of plenty of job systems directly. SQS bundles durable buffering with acknowledgment, retries, and dead-letter handling, though application-specific scheduling, prioritization, and dedup may still live elsewhere. Durable-execution engines like Temporal go further and persist every step of a job, not just the fact that it was enqueued. Even Redis added Streams a year after this post, a log structure with consumer groups and acknowledgments, though in a conventional deployment the data still has to fit in RAM. The pattern outlived the architecture and is the thing to takeaway here: keep the backlog somewhere that can't run out of memory, and let the fast layer hold only what's in flight.