Aggregating Live Event Streams

Windows, event time and watermarks, checkpointed state, hot keys and batch reconciliation.

Clicks per ad per minute, trending hashtags, fraud signals on card payments, live dashboards of orders per region: when you need answers while events are still arriving, you process them as a stream instead of loading them into a database and querying later. The pattern comes up in designs like Design Top-K and ad-click aggregation.

Batch vs stream

Many real systems use both: a stream for fresh, approximate-enough numbers and a batch job that recomputes the exact answer later.

The shape of a streaming pipeline

Producers -> {Kafka: raw events} -> [Stream processor: windows + state] -> (OLAP / KV store)
(OLAP / KV store) -> [Dashboards & APIs]

Ingest events into a durable log (Kafka), partitioned by the key you'll aggregate on (ad ID, user ID).
Process with a stream processor that keeps state per key (running counts) and groups events into windows.
Sink aggregated results into a store built for fast reads — an OLAP database (ClickHouse, Druid, Pinot), Redis, or Cassandra.

Aggregating early is the point: 1M clicks a second becomes a few thousand (ad, minute, count) rows a minute, which is easy to store and query.

Windows

Event time vs processing time

Events arrive late and out of order: a phone was offline, a partition lagged, a retry happened. If you bucket by processing time (when the event reached you), a delayed batch of clicks lands in the wrong minute. Bucket by event time (when it actually happened) instead.

That raises a question: when is a window "done"? Stream processors use watermarks — a moving estimate that says "we've probably seen all events up to time T."

latest event time seen:  12:05:40
watermark = 12:05:40 − 30 s = 12:05:10
→ the 12:04:00–12:05:00 window can now be closed and emitted

A larger tolerance means more complete results but later output.
Allowed lateness keeps a closed window's state around a bit longer so a straggler can still update it (emitting a corrected result).
Events that arrive even later go to a side output — logged and reconciled by the batch job, never silently dropped.

State and fault tolerance

Counting needs memory: the processor keeps per-key state for every open window. To survive crashes, processors like Flink take periodic checkpoints — a consistent snapshot of all operator state *and* the Kafka offsets it corresponds to. After a failure, the job restores the last checkpoint and replays from those offsets, so the state is as if each event was counted exactly once (inside the pipeline).

The output still needs care: the sink may receive a window's result twice after a replay. Make writes idempotent — upsert keyed by (ad_id, window_start) rather than incrementing — so a rewrite replaces the value. See Safe Retries & Idempotency.

Scaling

Partition by key so each processor instance owns a slice of keys and their state; parallelism is bounded by the number of Kafka partitions.
Hot keys (one viral ad) overload one instance. Pre-aggregate with a salted key (ad42#3) across several instances, then sum the partial counts in a second stage.
Approximate when exact is too expensive: HyperLogLog for unique users, Count-Min Sketch for heavy hitters, t-digest for percentiles. They use tiny, mergeable state and are usually good enough for dashboards.

Keeping it correct enough to bill on

When the numbers drive money (ad billing, payouts), pair the stream with a batch path:

{Kafka} -> [Stream processor] -> (Fast results)
{Kafka} -> (Raw event archive) -> [Daily batch job] -> (Reconciled results)

The batch job recomputes totals from the raw archive and corrects any windows that differ. Dashboards show streamed numbers within a minute; invoices use reconciled ones.

[!INTERVIEW] What a strong answer sounds like "Click events go to Kafka partitioned by ad ID. A Flink job counts clicks per ad in 1-minute tumbling windows by event time, with a watermark 30 seconds behind and 5 minutes of allowed lateness; anything later goes to a side output. Results are upserted into ClickHouse keyed by ad and minute, so replays after a failure don't double count, and Flink checkpoints every 10 seconds. Raw events are archived to S3 and a nightly Spark job reconciles the counts we bill on."