Courses › System Design › Question Breakdowns

Design WhatsApp

Real-time 1:1 and group messaging, delivery receipts, ordering, presence and offline users.

Design a messaging app like WhatsApp or Messenger. It's a test of real-time communication, routing messages across servers, and reliable delivery to users who are frequently offline.

Requirements

Functional

  • Users can start 1:1 and group chats (groups up to ~100 members)
  • Users can send and receive messages in real time
  • Users receive messages sent while they were offline (for up to 30 days)
  • Users see delivery and read receipts

Out of scope: voice/video calls, media beyond simple attachments, end-to-end encryption details.

Non-functional

  • Low latency delivery (< 500ms) for online users
  • Guaranteed delivery — messages must not be lost
  • Scale: ~100B messages a day, hundreds of millions of users online at peak
  • Messages delivered in order within a conversation
  • Consistent history: a user sees the same conversation on every device

[!TIP] Back-of-the-envelope WhatsApp has reported ~100B messages a day. That's ~1.2M messages/second on average, with evening peaks around twice that. A text message with its envelope is ~250 bytes, so ~25 TB/day. Since we keep messages for 30 days (for offline users and new devices), the store holds ~750 TB, about 2.2 PB with three replicas. Throughput, not total size, is what the store has to be chosen for.

Core Entities

  • User, Conversation (1:1 or group), Membership, Message, Receipt

API

Most traffic flows over a persistent WebSocket connection, with message types rather than REST endpoints:

{ "type": "send_message", "conversationId": "c1", "clientMsgId": "uuid", "body": "hi" }
{ "type": "ack", "messageIds": ["m1", "m2"] }
{ "type": "new_message", "conversationId": "c1", "messageId": "m1", "seq": 1042, "body": "hi" }
{ "type": "receipt", "messageId": "m1", "status": "read", "userId": "u7" }

High-Level Design

a = client "Client A" @0,0
cs1 = service "Chat Server 1" @1,0
ps = queue "Pub/Sub" @2,0
cs7 = service "Chat Server 7" @3,0
b = client "Client B" @4,0
db = store "Messages DB" @1,1
reg = cache "Connection\nRegistry" @2,1
notif = service "Push Notifications\n(APNs / FCM)" @3,1
a <-> cs1 : WebSocket
cs1 -> ps
ps -> cs7
cs7 <-> b : WebSocket
cs1 -> db
cs1 -> reg
cs1 --> notif : B offline

Sending a message:

  • Client A sends the message over its WebSocket to chat server 1.
  • Server 1 persists the message (assigning a per-conversation sequence number), then acks to A — the message is now durable.
  • Server 1 looks up each recipient's connection (registry: userId → serverId) or publishes to each recipient's channel.
  • The recipient's server pushes the message over B's WebSocket.
  • B acks; the server records delivery and notifies A (delivered receipt).

Offline recipients: the message stays in an undelivered inbox for that user. When they reconnect, the client asks for everything after its last-seen sequence number per conversation (or the server drains the inbox).

a = client "User A" @0,0
s = service "Chat Server" @1.5,0
db = store "Messages DB" @1.5,1
b = client "User B" @3,0
a -> s : 1. send
s -> db : 2. persist
s -> b : 3. deliver
b --> s : 4. ack

Once B acks, the server sends A a delivered receipt, and later a read receipt when B opens the chat.

[!NOTE] Ack before or after storing? Acking the sender after the write is durable is the safe default: an acked message can't be lost. Acking first and persisting in the background is faster, but needs a retry log for writes that fail. Say which one you chose and why.

Deep Dives

1. Push, not pull

Polling can't meet a 500ms target without hammering the servers with empty responses, so an online client keeps one WebSocket open and the server writes to it the moment a message arrives. Long polling is the fallback for networks that block WebSockets. Each chat server keeps an in-memory map userId → socket for its own connections.

A phone in the background loses its socket within seconds (the OS suspends the app). That gap is what push notifications fill (section 7); the socket only serves apps that are on screen or were just used.

How many chat servers? Assume ~400M users online at the evening peak. A server built for connections (epoll or Erlang/BEAM, little memory per socket) holds hundreds of thousands to a couple of million; WhatsApp famously ran ~2M per box. At a conservative 200k per server, that's ~2,000 chat servers. The count is set by connections and memory, not CPU.

2. Routing messages between servers

With hundreds of millions of concurrent connections spread over thousands of servers:

  • Pub/sub per user (e.g. Redis): each server subscribes to channels for its connected users; senders publish to the recipient's channel. Simple and fast.
  • Connection registry + direct RPC: look up which server holds the user and send directly. Fewer moving parts per message at scale, but the registry must be kept accurate as users reconnect.

Either way, the database is the source of truth. If a real-time hop fails, the message is still delivered on the next sync.

3. Storing messages

Messages are write-heavy and always read by conversation in time order — a natural fit for Cassandra (or DynamoDB):

PRIMARY KEY ((conversation_id, bucket), seq)  -- newest-first clustering

Undelivered message pointers live in a per-user inbox table, deleted once acked. Messages older than the retention window are expired with TTLs.

Why not Postgres? At over a million inserts a second, a B-tree store spends its time on random index updates and vacuuming deleted rows. An LSM-based store appends writes and sorts them in the background, and its clustering order makes "the 50 messages before seq 1,042" one sequential read. That matches the workload exactly.

Partitioning: the partition key is (conversation_id, bucket), where a bucket is a week or a fixed message count, so a busy group never builds one unbounded partition. Spread across 4 TB nodes, ~2.2 PB of replicated data is ~550 nodes, plus headroom.

The phone is the cache. Clients keep their full history locally (SQLite) and only ask the server for messages after their cursor. Opening a chat reads from the device; the server is mostly a delivery queue plus a 30-day backlog, which is why its storage stays bounded.

4. Ordering and duplicates

  • Phone clocks can't order messages: two people in a group who send at the same moment from phones whose clocks differ by a few seconds would see the two messages in different orders.
  • Instead, the server that persists a message assigns the conversation's next sequence number (one writer per conversation, or a conditional insert). Every device sorts by it, so everyone sees the same order, and a gap in the numbers tells a client it missed something and should re-sync.
  • The client's clientMsgId makes sends idempotent: a retried send doesn't create a duplicate.

5. Groups

For groups of ≤100, fan out on send: persist once per conversation and push to each online member. Very large groups or channels would need fan-out via queues and batching.

The chat server that receives a group message persists it once under the group's conversation, reads the member list (cached, since membership rarely changes), and publishes one delivery per member to that member's channel. Members' own chat servers do the last hop. Read receipts in groups are aggregated ("read by 12") rather than sent as 99 separate events.

6. Multiple devices and presence

  • Each device has its own connection and its own last-seen sequence; the server fans out to all of a user's devices.
  • Presence is derived from the socket: connected means online. Store it as a Redis key with a ~60-second TTL, refreshed by heartbeats.
  • Subscribe on view: a client subscribes to the presence of the one contact whose chat is open, not to all 300 contacts. That turns an n² broadcast into a handful of subscriptions per user.
  • "Last seen" is a timestamp written at most once a minute per user, not on every heartbeat.
  • Typing indicators are fire-and-forget over the socket and never stored.

7. Failures, offline users and push notifications

  • Chat server crashes: its ~200k clients reconnect at once. Have clients retry with jittered exponential backoff, or the reconnect storm takes down the next server too. They re-sync from their cursors, so nothing is lost.
  • Durability: a message is acked to the sender only after a quorum write (2 of 3 replicas in different zones).
  • Offline users: the message waits in the inbox, and the notification service sends a push through APNs or FCM using the device token registered at login. With end-to-end encryption the push carries no message text, just "new message"; the app wakes, connects and fetches it.

What's expected at each level

  • Mid-level: WebSockets, persistence before delivery, offline delivery on reconnect.
  • Senior: cross-server routing, a write-optimized message store with a good partition key, ordering and idempotency.
  • Staff: multi-device sync, presence at scale, connection-server failure and reconnect storms, and regional deployment.