Key Technologies
Flink
Learn about how you can use Flink to solve a large number of problems in System Design.
Many system design problems will require stream processing. You have a continuous flow of data and you want to process, transform, or analyze it in real-time.
Stream processing is actually hard and expensive to get right. Many problems that seem like stream processing problems can actually be reduced to batch processing problems where you'd use something like Spark or (if you're ancient enough) Hadoop.
Before embarking on a stream processing solution, ask yourself the critical question: "do I really need real-time latencies?". For many problems, the answer is no and the engineers after you will thank you for saving them the ops headache.
The most basic example of this might be a service reading clicks from a Kafka topic, doing a trivial transformation (maybe reformatting the data for ingestion), and writing to a database. Easy.
Simple Kafka Stream Processing
But things can get substantially more complex from here. Imagine we want to keep track of the count of clicks per user in the last 5 minutes. Because of that 5 minute window, we've introduced state to our problem. Each message can't be processed independently because we need to remember the count from previous messages. While we can definitely do this in our service by just holding counters in memory, we've introduced a bunch of new problems.
- As one example of a problem, if our new service crashes it will lose all of its state: basically the count for the preceding 5 minutes is gone. Our service could hypothetically recover from this by re-reading all the messages from the Kafka topic, but this is slow and expensive.
- Or another problem is scaling. If we want to add a new service instance because we're handling more clicks, we need to figure out how to re-distribute the state from existing instances to the new ones. This is a complicated dance with a lot of failure scenarios!
- Or what if events come in out of order or late! This is likely to happen and will impact the accuracy of our counts.
And things only get harder from here as we add complexity and more statefulness. Fortunately, engineers have been building these systems for decades and have come up with useful abstractions. Enter one of the most powerful stream processing engines: Apache Flink.
Flink is a framework for building stream processing applications that solves some of the tricky problems like those we've discussed above and more. While we could talk about Flink for days, in this deep dive we're going to focus on two different perspectives to understanding Flink:
- First, we're going to talk about how Flink is used. There's a good chance you'll encounter a stream-oriented problem in your interview and Flink is a powerful, flexible tool for the job when it applies.
- Secondly, you'll learn how Flink works, at a high-level, under the hood. Flink solves a lot of problems for you, but for interviews it's important you understand how it does that so you can answer deep-dive questions and support your design. We'll cover the important bits.
Let's get to it!
Basic Concepts
Sources/Sinks
Streams
Operators
State
Watermarks
Windows
Basic Use
Defining a Job
Submitting a Job
Sample Jobs
Basic Dashboard Using Redis
Fraud Detection System
How Flink Works
Cluster Architecture
Job Manager and Task Managers
Task Slots and Parallelism
State Management
State Backends
Checkpointing and Exactly-Once Processing
In Your Interview
Using Flink
Lessons from Flink
Conclusion
References
Purchase Premium to Keep Reading
Unlock this article and so much more with Hello Interview Premium
Currently up to 20% off
Hello Interview Premium
Reading Progress
On This Page