Clocks & Ordering

Clock skew, Lamport and vector clocks, hybrid clocks, TrueTime and ordering events safely.

"Which happened first?" sounds trivial, but in a distributed system each machine has its own clock, and none of them agree exactly. Deciding the order of events — for last-writer-wins, for log ordering, for "newest message first" — is a common source of subtle bugs, and a favorite follow-up question.

Physical clocks drift

Server clocks are synchronized with NTP, typically to within a few milliseconds — but sometimes tens or hundreds of milliseconds off, and they can jump backwards when corrected.
Wall-clock time (time.now()) is for displaying and approximate ordering. Use a monotonic clock for measuring durations and timeouts, because it never jumps backwards.
Two events on different machines a few milliseconds apart can't be reliably ordered by their timestamps.

[!WARNING] Last-writer-wins can lose data If two replicas resolve conflicts by timestamp and one clock runs 50 ms fast, its writes "win" even when they happened earlier. LWW is fine for data where losing a concurrent update is acceptable (a profile photo), not for counters or carts.

Logical clocks

Instead of measuring time, logical clocks capture causality: "A happened before B" if A could have influenced B.

Lamport timestamps

Each node keeps a counter. Increment it on every event; attach it to every message; on receiving a message, set the counter to max(local, received) + 1.
If A happened before B, then L(A) < L(B). Ties are broken by node ID to get a total order everyone agrees on.
They can't tell you whether two events were concurrent — a smaller timestamp doesn't prove causality.

Vector clocks

Each node keeps a vector of counters, one per node. Comparing two vectors tells you whether one event happened before the other or they were concurrent.
Used to detect conflicting writes that need merging (Dynamo-style stores, Riak). The cost is a vector that grows with the number of writers.

Hybrid logical clocks (HLC)

Combine physical time with a logical counter: timestamps stay close to wall-clock time but always respect causality.
Used by CockroachDB, YugabyteDB and MongoDB to order transactions without special hardware.

Bounded-uncertainty clocks

Google's TrueTime uses GPS and atomic clocks to report time as an interval [earliest, latest] with a small, known uncertainty. Spanner waits out that uncertainty before committing, which gives globally ordered, externally consistent transactions. AWS offers a similar bounded clock (ClockBound).

Practical ordering techniques

[!TIP] Let a single owner assign the order The most robust way to order events is to avoid comparing clocks at all: send everything that must be ordered through one sequencer — a partition leader, a database primary, or a per-entity counter — and use its sequence numbers.

[!INTERVIEW] Message ordering in chat "I won't trust client timestamps for ordering. Each chat is owned by one partition, which assigns a monotonically increasing sequence number as messages arrive. Clients sort by sequence number and use it to detect gaps and fetch missing messages after reconnecting. The client timestamp is only shown to users."