Replication

Single-leader, multi-leader and leaderless replication, quorums, replication lag and failover.

Replication means keeping copies of the same data on several machines. It serves three goals: availability (survive a machine or zone failure), read scaling (spread reads over copies) and latency (put a copy near users). Almost every database you'll name in an interview replicates, and the strategy it uses determines what can go wrong. Databases introduces the basics; this page goes deeper.

Three replication strategies

Single leader (primary–replica)

[Writes] -> (Leader) -> (Follower 1)
(Leader) -> (Follower 2)
[Reads] -> (Follower 1)

All writes go to one leader, which streams its change log to followers. Reads can go to any copy.
Used by PostgreSQL, MySQL, MongoDB, Redis and Kafka partitions.
Simple and conflict-free, because one node orders all writes. The leader is a write bottleneck, which you solve with sharding: one leader per shard.

Multi-leader

Several nodes (usually one per region) accept writes and replicate to each other.
Good for multi-region writes with low local latency and for offline clients (a calendar app on your phone is effectively a leader).
Write conflicts are the cost: two regions update the same row concurrently. You need a resolution rule — last-writer-wins, merge functions, or CRDTs.

Leaderless

Clients (or a coordinator) write to several replicas directly and read from several. Used by Cassandra, DynamoDB internally and Riak.
Quorums: with N replicas, a write waits for W acknowledgements and a read queries R replicas. If W + R > N, every read overlaps the latest write. A typical setting is N=3, W=2, R=2.
Stale replicas are repaired by read repair (fix it when a read notices) and anti-entropy (background comparison using Merkle trees).

Synchronous vs asynchronous

Synchronous — the leader waits for a follower before acknowledging. No data loss on failover, but each write is slower and a slow follower blocks writes.
Asynchronous — acknowledge immediately and replicate in the background. Fast, but the last few writes can be lost if the leader dies.
Semi-synchronous — one synchronous follower, the rest asynchronous. A common compromise.
Quorum-based (Raft, Paxos) — wait for a majority. Survives minority failures without losing acknowledged writes. See Consensus.

Replication lag and its anomalies

With asynchronous followers, a read can return old data. The classic problems:

Read-your-writes — a user updates their profile, refreshes, and sees the old version because the read hit a lagging follower. Fix: read your own recent writes from the leader (e.g. for 1 minute after a write), or wait until the follower has caught up to the write's log position.
Monotonic reads — two refreshes hit different followers and time seems to go backwards. Fix: pin each user to one replica.
Consistent prefix — an answer appears before its question because two partitions replicated at different speeds. Fix: keep causally related writes in the same partition.

These guarantees are described in Consistency Models.

Failover

When the leader dies, a follower must be promoted:

Detect the failure — usually a missed heartbeat timeout (see Gossip & Failure Detection). Too short causes needless failovers; too long extends downtime.
Elect a new leader — ideally the most up-to-date follower, chosen by consensus.
Redirect clients and make the old leader step down if it comes back.

[!WARNING] Split brain If the old leader is only partitioned, not dead, two nodes may accept writes at once. Prevent it with a consensus-based election and fencing — a monotonically increasing epoch number that storage rejects from stale leaders.

[!INTERVIEW] Explaining your read path "Writes go to the primary with one synchronous replica in another zone, so failover loses nothing. Reads go to async read replicas, except a user's own profile reads for 30 seconds after they edit it, which go to the primary so they always see their change."