Back to Main
Learn System Design
Question Breakdowns
Advanced Topics
Get Premium
Patterns
Managing Long Running Tasks
Learn about the long running tasks pattern and how to use it in your system design
Managing Long Running Tasks
The Problem
Let's start with a problem. Imagine you run a simple website where users can view their profile. When a user loads their profile page, your server makes a quick database query to fetch their data. The whole process - querying the database, formatting the response, sending it back - takes less than 100 milliseconds. The user clicks and almost instantly sees their information. Life is good.
Now imagine that instead of just fetching data, we need to generate a PDF report of the user's annual activity. This involves querying multiple tables, aggregating data across millions of rows, rendering charts, and producing a formatted document. The whole process takes at least 45 seconds.
Sync Processing Problem
With synchronous processing, the user's browser sits waiting for 45 seconds. Most web servers and load balancers enforce timeout limits around 30-60 seconds, so the request might not even complete. Even if it does, the user experience is poor. They're staring at a loading indicator with no feedback about progress.
The PDF report isn't unique. Video uploads, for example, require transcoding that takes several minutes. Profile photo uploads need resizing, cropping, and generating multiple thumbnail sizes. Bulk operations like sending newsletters to thousands of users or importing large CSV files take even longer. Each of these operations far exceeds what users will reasonably wait for.
The Solution
Trade-offs
What you gain
What you lose
How to Implement
Message Queue
Workers
Putting It Together
When to Use in Interviews
For Example
Common Deep Dives
Handling Failures
Handling Repeated Failures
Preventing Duplicate Work
Managing Queue Backpressure
Handling Mixed Workloads
Orchestrating Job Dependencies
Conclusion
Schedule a mock interview
Meet with a FAANG senior+ engineer or manager and learn exactly what it takes to get the job.