Courses › System Design › Question Breakdowns

Design an Ad Click Aggregator

Click logging off the hot path, stream aggregation, dedup with signed impressions and batch reconciliation.

An ad click aggregator records every click on an ad, sends the user on to the advertiser's site, and lets advertisers see how many clicks each ad got, minute by minute. It's a question about high-volume event ingestion, stream aggregation and counts accurate enough to bill on.

Requirements

Functional

  • Users click an ad and are redirected to the advertiser's website
  • Advertisers can query click metrics for their ads over time, at a granularity as fine as 1 minute

Out of scope: ad targeting and serving, impressions and conversions, fraud detection beyond basic dedup (discuss as an extension), the reporting UI.

Non-functional

  • Scale: 10M active ads, 10k clicks/second at peak (~100M clicks/day)
  • Fast queries: advertiser dashboards answer in < 1 second
  • Fresh data: a click shows up in metrics within a minute or so
  • Accurate and fault tolerant: no lost clicks, no double counting; these numbers become invoices
  • The redirect itself must be fast — the user is waiting

[!TIP] Two very different workloads The write path is a firehose of tiny events (10k/s); the read path is a small number of advertisers asking aggregate questions. Say this early: it's why you aggregate on write (precompute per-minute counts) instead of counting raw rows when someone opens a dashboard.

Capacity estimates

Core Entities

  • Ad — ad ID, advertiser, redirect URL
  • Click event — click ID, ad ID, timestamp, user/IP, user agent
  • Aggregated metric — ad ID, minute bucket, click count (and unique users)

API

GET /click?adId=a123&impressionId=imp_9f2c&sig=...
→ 302 Location: https://advertiser.com/landing

GET /v1/ads/{adId}/metrics?from=2026-10-01T00:00&to=2026-10-02T00:00&granularity=1m
→ { "adId": "a123", "points": [{ "t": "2026-10-01T00:00", "clicks": 412 }, ...] }

Making the click a server-side redirect (rather than letting the browser go straight to the advertiser and sending a beacon "on the side") guarantees we see every click: if the browser blocks or drops the beacon, we've still logged it.

High-Level Design

user = client "Browser" @0,0
click = service "Click Service" @1,0
kafka = queue "Kafka\n(click events)" @2,0
flink = service "Stream Processor\n(Flink)" @3,0
olap = store "OLAP Store\n(per-minute counts)" @3,1
raw = blob "Raw Events (S3)" @2,1
batch = service "Batch Reconciler" @2,2
query = service "Query Service" @4,1
adv = client "Advertiser\nDashboard" @4,2
user -> click : GET /click
click -> kafka : log event
kafka -> flink
kafka --> raw : archive
flink -> olap : 1-min aggregates
raw -> batch
batch -> olap : corrected counts
adv -> query
query -> olap
  • Click: the click service validates the signed impression ID, writes the event to Kafka, and returns the 302 immediately. It never touches a database on the hot path.
  • Aggregate: a Flink job reads clicks partitioned by ad ID, dedups, and keeps a per-ad counter for each 1-minute tumbling window. When a window closes, it writes (adId, minute, count) to the OLAP store.
  • Archive: raw events also land in S3 (via a Kafka connector) — the source of truth for reconciliation and backfills.
  • Query: the query service reads precomputed rows from the OLAP store (ClickHouse, Druid, Pinot) and rolls minutes up into hours or days.

Deep Dives

1. Why not just count in a database?

The stream processor turns 10k writes/second into one write per active ad per minute. That's the whole trick.

2. Not counting a click twice

Duplicates come from users double-clicking, retries between the browser and the click service, and replays after a Flink restart.

  • Impression IDs: when the ad is served, the ad server generates a unique impression ID and signs it (HMAC) into the click URL. The click service rejects bad signatures, so nobody can mint fake clicks by inventing IDs.
  • Dedup in the stream: Flink keeps a set of impression IDs seen in the last few minutes (keyed state with a TTL) and drops repeats. One impression can produce at most one billable click.
  • Exactly-once processing: Flink checkpoints its state and Kafka offsets together, and the sink writes idempotently (upsert keyed by adId + minute), so a restart that reprocesses a few seconds of events overwrites rows instead of adding to them.

[!WARNING] Exactly-once is end to end, or not at all Flink's checkpointing only gives exactly-once *inside* the job. If the sink blindly INCREMENTs, a replay double counts. Make the output idempotent: write the full count for a window, keyed by window, so writing it twice is harmless.

3. Freshness *and* billing-grade accuracy

Streaming gives fresh numbers but can be wrong at the edges (late events, bugs, a bad deploy). Run a reconciliation batch as well:

[Kafka] -> [Flink] -> (OLAP: fast counts)
[Kafka] -> (S3 raw events) -> [Hourly / daily batch job] -> (OLAP: corrected counts)

The batch job (Spark) recomputes counts from raw events in S3, compares them with what the stream wrote, and overwrites any window that differs. Dashboards show streamed numbers within a minute; invoices use reconciled numbers.

4. Late-arriving events

A phone goes through a tunnel and its click reaches us 3 minutes late. Use event time (when the click happened), not processing time, and give each window a watermark with allowed lateness (e.g. 5 minutes): the window is emitted on time and re-emitted with an updated count if late events arrive. Anything later still is picked up by the batch reconciliation.

5. Scaling the stream

  • Kafka partitions by ad ID so all clicks for one ad go to one Flink task (needed for per-ad state). 10k/s needs only a handful of partitions; provision more for growth.
  • Hot ads: a Super Bowl ad can take a big share of all clicks and overload one partition. Salt the key for known-hot ads (adId:0…adId:N), count each sub-key separately and sum at query time.
  • Click service is stateless; scale horizontally behind a load balancer. If Kafka is briefly unavailable, buffer locally on disk rather than dropping clicks.

6. Fast queries over long ranges

A query for "last 90 days at 1-minute granularity" would read ~130k points. Pre-roll hourly and daily tables (a scheduled job or the OLAP store's materialized views) and pick the coarsest granularity that answers the query. Index the tables by (adId, time); OLAP stores keep these columnar and compressed.

What's expected at each level

  • Mid-level: a working click → queue → aggregate → query pipeline, and why counting raw rows on read doesn't scale.
  • Senior: stream processing with tumbling windows, idempotent dedup with signed impression IDs, a reconciliation path for accuracy.
  • Staff: end-to-end exactly-once reasoning, event time vs. processing time and late data, hot-key mitigation, and how billing uses reconciled numbers while dashboards use streamed ones.