Courses › System Design › Key Technologies
ZooKeeper & etcd
Consensus-backed coordination: leader election, membership, locks and fencing tokens.
ZooKeeper and etcd are coordination services: small, strongly consistent, highly available key-value stores that distributed systems use to agree on things. You rarely store application data in them. Instead, they answer questions like "who is the leader?", "which servers are alive?", "who holds this lock?" and "what's the current config?" In interviews, they're the answer when correctness depends on every node agreeing.
How they work
- A small ensemble of 3 or 5 nodes replicates every write with a consensus protocol — ZAB in ZooKeeper, Raft in etcd.
- A write is committed once a majority (quorum) acknowledges it. A 5-node ensemble tolerates 2 failures; it stays available as long as a majority is up.
- All writes go through the leader, so they're linearizable — every client sees them in the same order.
- The data set is small (megabytes) and kept in memory. These systems are built for correctness and availability, not throughput.
[!NOTE] CP by design In CAP terms they're CP: a minority partition refuses writes rather than risk two leaders. That's exactly the property you want from a coordinator.
Key primitives
What they're used for
Leader election
Each candidate creates an ephemeral sequential node under /election. The one with the lowest sequence number is the leader; the others watch the node just before theirs. If the leader crashes, its session expires, its node disappears, and the next candidate takes over — no split brain.
Service discovery and membership
Each server registers an ephemeral node (or a leased key) when it starts. Clients watch the directory to learn which servers are alive. Crashed servers drop out automatically when their session or lease expires.
Distributed locks
The same pattern as leader election, per resource. Because ZooKeeper's sequence numbers and etcd's revisions are monotonically increasing, they double as fencing tokens: the protected resource rejects writes carrying an older token. That makes these locks safer than a Redis lock for correctness-critical work. See Design a Distributed Lock Service.
Configuration and metadata
Small, critical metadata that every node must agree on: shard-to-server maps, feature flags, cluster membership. Kubernetes stores all cluster state in etcd; older Kafka versions kept broker and partition metadata in ZooKeeper (newer ones use the built-in KRaft protocol).
Where Chubby fits
Google's Chubby (2006) is the lock service this family descends from. It's built on Paxos and designed for coarse-grained locks held for minutes or hours. It introduced the ideas both systems use: client sessions kept alive by heartbeats (KeepAlives), small files used as locks and config, change notifications, and sequencers, an early form of fencing token. ZooKeeper was Yahoo's open-source take on the same idea, and etcd is a newer Raft-based version of it. Chubby is internal to Google, so you won't choose it in an interview. If an interviewer or a paper mentions it, treat it as a ZooKeeper equivalent.
When not to use them
- Application data, high write rates or large values — they're slow on purpose and hold little data
- Caching or rate-limit counters — use Redis
- Locks where an occasional double-execution is harmless — a Redis lock is simpler
[!INTERVIEW] Shard ownership in a job scheduler "Each scheduler node registers an ephemeral node in ZooKeeper. A leader, elected the same way, assigns job partitions to live nodes and writes the assignment to /assignments. Nodes watch that path. If a node dies, its session expires, the leader sees the membership change and reassigns its partitions, and the epoch number acts as a fencing token so a zombie node can't run jobs it no longer owns."