Courses › System Design › Question Breakdowns

Design a Distributed Message Queue

Partitioned append-only logs, ISR replication and high-water marks, consumer groups, offsets and delivery guarantees.

Design a distributed message queue like Kafka (or a queue service like SQS): producers publish messages to topics, consumers read them at their own pace, and the system stores everything durably across a cluster. It's a question about the append-only log, partitions for ordering and parallelism, replication and leader election, consumer groups and offsets, and delivery guarantees. (For when to *use* a queue, see Message Queues and Kafka; this breakdown is about *building* one.)

Requirements

Functional

  • Producers publish messages (key, value) to a topic
  • Consumers subscribe to topics; multiple consumer groups each get every message, and consumers within a group share the work
  • Messages with the same key are delivered in order
  • Messages are retained for a configured time (e.g. 7 days) and can be replayed from an earlier position

Out of scope: per-message ack/visibility semantics of SQS-style queues (discuss as a variant), cross-cluster replication, exactly-once transactions (mention).

Non-functional

  • Throughput: ~1M messages/second in, several times that out (many consumer groups)
  • Durable: acknowledged messages survive broker failures
  • Low latency: publish-to-consume in tens of milliseconds
  • Scalable: add brokers and partitions to grow
  • Highly available: losing a broker doesn't stop producers or consumers

[!TIP] It's a log, not a queue The core idea that makes Kafka fast: messages aren't deleted when consumed. Each partition is an append-only log on disk; consumers just remember an offset. Sequential disk writes and reads, the OS page cache, and zero-copy transfers make this extremely fast, and replay comes for free.

Capacity estimates

Core Entities

  • Topic — name, partition count, replication factor, retention
  • Partition — an ordered, append-only log; one leader replica and followers
  • Message — offset (position in the partition), key, value, timestamp, headers
  • Consumer group — members, partition assignments, committed offset per partition

API

produce(topic, key, value, acks = all) → (partition, offset)
subscribe(group, topics)                → partitions assigned to this consumer
poll()                                  → batch of messages from assigned partitions
commit(partition, offset)
seek(partition, offset | timestamp)     → replay

High-Level Design

p = client "Producers" @0,1
b1 = store "Broker 1\n(leader P0, follower P1)" @1,0
b2 = store "Broker 2\n(leader P1, follower P2)" @1,1
b3 = store "Broker 3\n(leader P2, follower P0)" @1,2
ctrl = service "Controller\n(metadata, leader election)" @2,0
g1 = client "Consumer Group A" @2,1
g2 = client "Consumer Group B" @2,2
p -> b1 : hash(key) → partition
p -> b2
p -> b3
b1 <-> b3 : replicate P0
ctrl --> b1
ctrl --> b2
ctrl --> b3
g1 -> b1
g1 -> b2
g2 -> b3
  • A topic is split into partitions; the producer picks one by hash(key), so a key's messages stay in order on one partition.
  • Each partition has a leader broker that accepts writes and followers that replicate it.
  • A controller (a Raft quorum in modern Kafka's KRaft mode) stores metadata and elects new leaders when brokers fail.
  • Consumers in a group split the partitions among themselves, read sequentially, and commit offsets.

Deep Dives

1. The partition log on disk

Each partition is a series of segment files (e.g. 1 GB each) plus a sparse index from offset to file position.

  • Writes append to the active segment — sequential I/O, batched and compressed by the producer.
  • Reads for recent data come from the OS page cache; the broker sends file bytes straight to the socket (zero-copy sendfile).
  • Retention deletes whole old segments — no per-message deletes. Log compaction (an alternative policy) keeps only the latest value per key, turning a topic into a durable changelog.

2. Replication and durability

  • Followers fetch from the leader. The set of followers that are caught up is the in-sync replica set (ISR).
  • With acks=all, the leader acknowledges a write only after every ISR member has it. With min.insync.replicas = 2 and replication factor 3, an acknowledged message survives any single broker failure.
  • A message is visible to consumers only up to the high-water mark (replicated to all ISR), so consumers never read data that could vanish on failover.

[!WARNING] Unclean leader election loses data If every in-sync replica is gone, electing an out-of-sync follower keeps the partition available but drops the messages it never received. Disable unclean election for anything that matters, accepting unavailability instead.

3. Leader election and metadata

The controller tracks partition leaders and ISRs. When a broker stops heartbeating, the controller picks a new leader for its partitions from their ISRs and publishes new metadata; clients refresh metadata and reconnect. Spread leaders evenly across brokers so load balances.

4. Consumer groups and offsets

  • Each partition is consumed by one consumer per group, which is what preserves ordering; parallelism is capped by the partition count.
  • Consumers commit offsets (stored in an internal compacted topic). After a crash, a consumer resumes from the last commit.
  • Rebalancing: when consumers join or leave, partitions are reassigned. Cooperative rebalancing moves only the partitions that must move, instead of stopping the whole group.

5. Delivery guarantees

6. Queue semantics (SQS-style variant)

A work queue needs per-message acknowledgment: a message is leased to one consumer (visibility timeout), deleted on ack, and redelivered on timeout, with a dead-letter queue after N failures. No ordering by default, no replay, but no "one consumer per partition" limit and no head-of-line blocking by a slow message.

7. Scaling

Add brokers and move partitions to them (partition reassignment copies data in the background). Choose partition counts for peak consumer parallelism (you can add partitions later, but that changes hash(key) → partition for existing keys). Use tiered storage to keep old segments in object storage cheaply.

What's expected at each level

  • Mid-level: topics and partitions, producers writing and consumers reading, data stored on disk and replicated.
  • Senior: append-only segments and offsets, ISR replication with acks=all, consumer groups and rebalancing, at-least-once vs. exactly-once.
  • Staff: high-water marks and unclean election, controller quorum and failover, log compaction, partition-count trade-offs, tiered storage, and queue-vs-log semantics.