Patterns
Managing Long Running Tasks
Learn about the long running tasks pattern and how to use it in your system design
Updated
Premium video
Sign in & purchase Premium to watch the full video.
🏃 The Managing Long-Running Tasks pattern splits API requests into two phases: immediate acknowledgment and background processing. When users submit heavy tasks (like video encoding), the web server instantly validates the request, pushes a job to a queue (Redis/RabbitMQ), and returns a job ID, all within milliseconds. Meanwhile, separate worker processes continuously poll the queue, grab pending jobs, execute the actual time-consuming work, and update the job status in a database.
The Problem
Let's start with a problem. Imagine you run a simple website where users can view their profile. When a user loads their profile page, your server makes a quick database query to fetch their data. The whole process - querying the database, formatting the response, sending it back - takes less than 100 milliseconds. The user clicks and almost instantly sees their information. Life is good.
Now imagine that instead of just fetching data, we need to generate a PDF report of the user's annual activity. This involves querying multiple tables, aggregating data across millions of rows, rendering charts, and producing a formatted document. The whole process takes at least 45 seconds.
Sync Processing Problem
With synchronous processing, the user's browser sits waiting for 45 seconds. Most web servers and load balancers enforce timeout limits around 30-60 seconds, so the request might not even complete. Even if it does, the user experience is poor. They're staring at a loading indicator with no feedback about progress.
The PDF report isn't unique. Video uploads, for example, require transcoding that takes several minutes. Profile photo uploads need resizing, cropping, and generating multiple thumbnail sizes. Bulk operations like sending newsletters to thousands of users or importing large CSV files take even longer. Each of these operations far exceeds what users will reasonably wait for.
The Solution
Trade-offs
What you gain
What you lose
How to Implement
Message Queue
Workers
Putting It Together
When to Use in Interviews
For Example
Common Deep Dives
Handling Failures
Handling Repeated Failures
Preventing Duplicate Work
Managing Queue Backpressure
Handling Mixed Workloads
Orchestrating Job Dependencies
Conclusion
Purchase Premium to Keep Reading
Unlock this article and so much more with Hello Interview Premium