Courses › System Design › Patterns

Broadcasting with Pub/Sub

Queues vs topics, choosing a broker, ordering keys, delivery guarantees and live fan-out.

In publish/subscribe, producers send messages to a topic without knowing who receives them, and every subscriber to that topic gets a copy. It decouples services — the order service publishes OrderPlaced once, and billing, shipping, email and analytics each react on their own — and it's the backbone of real-time features like chat and live updates.

Queues vs pub/sub

Two different delivery shapes, often confused:

Many systems do both. In Kafka, each consumer group gets every message on a topic (pub/sub), and within a group, partitions are split among its consumers (a work queue). Google Pub/Sub, SNS+SQS and RabbitMQ exchanges express the same combination differently.

[Order Service] -> {Topic: orders}
{Topic: orders} -> [Billing (group A)]
{Topic: orders} -> [Shipping (group B)]
{Topic: orders} -> [Analytics (group C)]

Choosing a system

The key question is "can a subscriber miss messages?" If a missed message is just a missed UI update the client will refresh anyway, ephemeral pub/sub is fine. If missing it means a customer isn't charged or an order isn't shipped, use a durable log with acknowledgments.

Ordering

  • Global ordering across a whole topic doesn't scale. Order is usually guaranteed per partition (Kafka) or per ordering key (Google Pub/Sub).
  • So choose the key that needs ordering: order_id for order events, conversation_id for chat. All events for one order land in one partition, in order.
  • Consumers must still tolerate duplicates, and with retries possibly reordering, compare versions or sequence numbers (see Safe Retries & Idempotency).

Delivery semantics and failures

  • Acknowledge after processing, not on receipt — otherwise a crash loses the message.
  • Retries with backoff, then a dead-letter queue for messages that keep failing (a "poison message" shouldn't block everything behind it).
  • Backpressure: a durable log lets slow consumers fall behind without affecting producers; monitor consumer lag and scale consumers (up to the number of partitions).
  • Retention as replay: with Kafka you can rewind a consumer group to reprocess events after fixing a bug, or bootstrap a brand-new service from history.

Pub/sub for real-time fan-out

A classic use: users are connected via WebSockets to many different servers, and an event must reach whichever servers hold the right connections.

p = service "Chat Service" @0,1
ps = queue "Redis Pub/Sub\n(channel per room)" @1,1
w1 = service "WS Server 1" @2,0.3
w2 = service "WS Server 2" @2,1.7
u1 = client "Alice" @3,0.3
u2 = client "Bob" @3,1.7
p -> ps : publish room:42
ps -> w1, w2 : subscribed to room:42
w1 -> u1
w2 -> u2

Each connection server subscribes to the channels of the users or rooms it currently serves, and unsubscribes when they disconnect. The message is persisted in the database first; pub/sub just delivers it to whoever is online, and offline users fetch it on reconnect. See Real-time Updates.

Topic design

  • Per event type (orders, payments) for service integration — a manageable number of topics, partitioned by entity ID.
  • Per entity (room:42, user:9) for real-time routing — fine in Redis Pub/Sub, where channels are nearly free, but not in Kafka, where millions of topics would overwhelm the cluster.
  • Schemas: publish versioned, backward-compatible message schemas (a schema registry with Avro or Protobuf), so producers can evolve without breaking subscribers.

[!WARNING] Don't lose events between your DB and the broker "Write to the database, then publish" fails when the process dies in between. Use the transactional outbox or CDC so publishing follows the commit reliably.

[!INTERVIEW] What a strong answer sounds like "Order events go to a Kafka topic partitioned by order ID, so each order's events stay in sequence. Billing, fulfillment and analytics each have their own consumer group, so they consume independently and a slow analytics job can't hold up shipping. Consumers ack after processing, dedupe by event ID and send repeated failures to a dead-letter topic. For live order-status updates in the app, a small service forwards those events into Redis Pub/Sub channels per user, which our WebSocket servers subscribe to."