Courses › System Design › Key Technologies

Apache Flink

Stateful stream processing, windows, event time, watermarks and checkpoints.

Apache Flink is a distributed stream processor: it reads unbounded event streams (usually from Kafka), keeps state, and emits results continuously with low latency. In interviews, it's the standard answer whenever you need real-time aggregation — click counts, fraud detection, trending topics, live metrics — at a scale a single consumer can't handle.

The mental model

[Kafka: clicks] -> [Source] -> [keyBy(ad_id)] -> [Window: 1 minute] -> [Aggregate: count] -> [Sink: OLAP / Redis]
  • A job is a graph of operators: sources, transformations (map, filter, keyBy, window, join) and sinks.
  • Each operator runs as many parallel tasks across the cluster. keyBy(ad_id) routes all events for one key to the same task, so per-key state is local.
  • Flink processes one event at a time (true streaming), unlike micro-batch engines, so latency is milliseconds to seconds.

State

The reason to use Flink rather than a plain Kafka consumer is managed state.

  • Operators keep keyed state — a running count per ad, the last location per driver, a session per user.
  • State lives in memory or in RocksDB on local disk, so it can be much larger than RAM.
  • Flink handles partitioning that state across tasks and redistributing it when you change parallelism.

Windows and time

Flink distinguishes event time (when the event happened) from processing time (when Flink sees it). Events arrive late and out of order, so event-time windows use watermarks — a signal that says "I've probably seen everything up to time T" — to decide when a window is complete. Events later than the allowed lateness can be dropped, sent to a side output, or used to update the result.

[!WARNING] Late data is a deep-dive favorite If an interviewer asks "what if a click arrives five minutes late?", talk about event time, watermarks and allowed lateness — and a periodic batch job that reconciles the real-time numbers against the raw log.

Fault tolerance

  • Flink takes periodic checkpoints: a consistent snapshot of all operator state plus the Kafka offsets it has read, stored in durable storage like S3.
  • On failure, it restores the last checkpoint and rewinds Kafka to the matching offsets.
  • This gives exactly-once state updates inside Flink. End-to-end exactly-once also needs a transactional or idempotent sink (e.g. upserts keyed by window and ad ID).

When to use it

  • Real-time aggregation with low latency (ad clicks, view counts, metrics)
  • Stateful event processing: fraud rules, anomaly detection, complex event patterns
  • Streaming joins and enrichment (join clicks with impressions within 30 minutes)
  • Continuous ETL from Kafka into search indexes, caches or a warehouse

When not to

  • Simple per-message work with no state — a plain consumer service is easier to run
  • Hourly or daily reports over historical data — batch with Spark or a warehouse query is simpler
  • Small scale — a single process with Redis counters may be enough

[!INTERVIEW] Ad click aggregator "Clicks land in Kafka partitioned by ad_id. A Flink job keys by ad_id, counts in one-minute tumbling event-time windows with a 30-second watermark, and upserts results into the OLAP store. Checkpoints to S3 give us exactly-once counts, and a nightly Spark job over the raw events reconciles anything the stream missed." See Design an Ad Click Aggregator and Stream Processing.