Courses › System Design › Key Technologies

Kafka

Topics, partitions, consumer groups, ordering, retention and delivery guarantees.

Apache Kafka is a distributed, durable, append-only log. Producers write events to it; consumers read them at their own pace. It shows up in interviews as a message queue (decoupling services, buffering bursts) and as an event stream (feeding analytics, search indexes and stream processors).

Core concepts

[Producers] -> {Topic: orders (partitions 0..N)} -> [Consumer Group A]
{Topic: orders (partitions 0..N)} -> [Consumer Group B]
  • Topic — a named stream of events, e.g. orders or clicks.
  • Partition — a topic is split into partitions, each an ordered, append-only log stored on a broker. Partitions are the unit of parallelism and ordering.
  • Offset — each message's position in its partition. Consumers track the offset they've processed.
  • Broker — a Kafka server. A cluster has many; each partition is replicated across several brokers.
  • Consumer group — a set of consumers that share the work of reading a topic. Each partition is assigned to exactly one consumer in the group, so the group processes each message once. Different groups read the same topic independently.

Partitioning and ordering

Kafka guarantees order within a partition, not across a topic. Producers choose the partition by key:

key = order_id   → all events for one order land in the same partition, in order
key = user_id    → all of a user's events are ordered relative to each other
no key           → spread round-robin, no ordering guarantee

[!INTERVIEW] Pick the key deliberately The partition key determines both ordering and load balance. Keying by a hot entity (one celebrity, one viral video) overloads a single partition. Mention how you'd handle that — e.g. salting the key with a suffix for very hot entities and aggregating downstream.

The number of partitions caps the parallelism of a consumer group: with 12 partitions, a 13th consumer sits idle.

Durability and retention

  • Each partition has a leader and follower replicas on other brokers. With acks=all, a write is acknowledged only after all in-sync replicas have it.
  • Messages are retained for a configured time or size (days or weeks), regardless of whether they've been consumed. Consumers can rewind and replay — invaluable for reprocessing after a bug.
  • Log compaction keeps only the latest value per key — useful for changelog topics.

Delivery semantics

In practice: at-least-once delivery + idempotent consumers (dedupe by event ID, or use upserts). Say this explicitly.

When to use Kafka

  • Decoupling — the order service publishes OrderPlaced; email, inventory and analytics consume independently.
  • Buffering bursts — absorb a traffic spike and let consumers catch up, instead of overwhelming a database.
  • Event streaming — clickstreams, logs, metrics feeding stream processors (Flink) or data lakes.
  • Change data capture — database changes streamed to caches, search indexes and warehouses.

When not to

  • Simple task queues with per-message acknowledgements, delays and retries are often easier with SQS, RabbitMQ or a Redis-based queue.
  • Request/response — Kafka is asynchronous by design.
  • Low volume — operating a Kafka cluster is real work; a managed queue may be enough.

Handling failures in consumers

  • Retries — retry transient errors with backoff.
  • Retry topics / dead-letter queue — move messages that keep failing aside so they don't block the partition.
  • Consumer lag — monitor it; growing lag means consumers can't keep up. Add partitions and consumers, or speed up processing.

[!TIP] Kafka is a good answer for "what if the database is slow?" Writing events to Kafka first and having consumers write to the database at a controlled rate is a standard way to scale writes and survive spikes. See Scaling Writes.