Concurrency
Scarcity
Learn how to manage limited resources in low level design.
⚖️ Scarcity is about managing limited resources when demand exceeds supply. You have finite database connections, limited memory for buffers, or expensive resources that should only be created once. The constraint isn't correctness, it's that there simply aren't enough resources for everyone who wants one.
The Problem
Consider a connection pool that manages connections to a database. Each query to the database needs a connection, and connections are expensive because they consume memory on the database server, hold file descriptors, and take time to establish. So you create a pool of 5 connections at startup and reuse them across requests.
Here's what should happen when a request needs the database, each request getting a connection from the pool and executing a query:
Connection Pool
Five requests can run simultaneously, each using one connection. The sixth request, however, will need to wait until someone returns a connection before it can proceed.
But what happens when a request grabs a connection and never gives it back? This could be due to a bug in the code, or it could be a slow query that takes a long time to complete. But in any case, it will block the connection pool and prevent other requests from getting connections making them wait forever and the service will hang.
Connection Pool Blocked
The database is fine. It's handling five queries without issue. But your service is dead. Every new request blocks indefinitely waiting for a connection that's never coming back. Users see timeouts. Your monitoring shows zero errors because nothing crashed. Requests are just stuck waiting forever.
This is a scarcity problem. What happens when there aren't enough resources for everyone? Unlike correctness problems where the danger is data corruption, here the issue is limited capacity. You have 5 connections and 100 concurrent requests. Most of those requests have to wait. The question is how you manage that waiting without the system falling over.
This article walks through two solutions to scarcity problems:
- Semaphores limit how many threads can hold a resource simultaneously
- Resource pooling gives you actual resource objects, not just permission
Then we cover the four patterns where scarcity appears most often in interviews and how to apply the solutions to them:
- Limit concurrent operations like download managers and API rate limiters
- Limit aggregate consumption like bandwidth limiting and memory budgets
- Reuse expensive objects like connection pools and GPU schedulers
- Maximize utilization like work stealing, batching, and adaptive pool sizing
Solutions
Every scarcity problem comes down to enforcing a capacity limit and deciding what happens when you hit it. You need to track how many resources are in use, block new requests when the limit is reached, and wake waiting threads when resources become available.
Semaphores
Challenges
Resource Pooling (with Blocking Queue)
Challenges
Common Problems
Limit Concurrent Operations
Examples
Limit Aggregate Consumption
Examples
Reuse Expensive Objects
Examples
Maximizing Utilization
Conclusion
Purchase Premium to Keep Reading
Unlock this article and so much more with Hello Interview Premium
Currently up to 20% off
Hello Interview Premium
Reading Progress
On This Page