Caching at Scale

Sharding a cache cluster, replicas and failover, hot keys, cold starts and stampedes.

One Redis box gets you surprisingly far — around 100k operations per second and tens of gigabytes of memory. A distributed cache is what you reach for when the hot data no longer fits on one machine, one machine can't serve the traffic, or losing that one machine would take your database down with it. The Caching guide covers *what* to cache and how to keep it fresh; this pattern is about running the cache as a cluster.

When one node stops being enough

Do the math out loud before you add nodes:

Memory: 50M users × 2 KB profile = 100 GB. That's past a comfortable single node, so partition.
Throughput: 500k reads/sec at peak is ~5 busy Redis nodes, even if the data would fit on one.
Availability: if a cache outage would send 100% of reads to a database sized for 5% of them, the cache is now a critical dependency and needs replicas.

Partitioning the keyspace

Every key must map to exactly one node, and every client must agree on that mapping.

Where the routing logic lives is a second choice: in a smart client (one hop, but every client library must agree), or a proxy tier such as Twemproxy or Envoy (thin clients, one extra hop).

[!TIP] Keep related keys together Multi-key operations only work when the keys live on the same node. Redis Cluster lets you force that with hash tags: cart:{user42}:items and cart:{user42}:total hash only on user42, so they land in the same slot.

Replication and failover

Give each shard a primary and one or two replicas. Writes go to the primary and replicate asynchronously, so a failover can lose the last few writes — fine for a cache, since the database still has the truth.

app = service "App Servers\n(smart client)" @0,1
p1 = cache "Shard A\nprimary" @1.5,0
r1 = cache "Shard A\nreplica" @2.7,0
p2 = cache "Shard B\nprimary" @1.5,1
r2 = cache "Shard B\nreplica" @2.7,1
p3 = cache "Shard C\nprimary" @1.5,2
r3 = cache "Shard C\nreplica" @2.7,2
db = store "Database\n(source of truth)" @0,2.4
app -> p1, p2, p3
p1 --> r1
p2 --> r2
p3 --> r3
app --> db : on miss

You can also read from replicas to spread load on hot shards, as long as the feature tolerates slightly stale values.

Hot keys

Partitioning spreads *keys*, not *traffic*. One viral post or celebrity profile can pin a single shard at 100% CPU while the rest idle. Fixes, cheapest first:

Local (in-process) cache with a 1–5 second TTL on each app server for the handful of hottest keys. Absorbs most of the load at the cost of a few seconds of staleness.
Key replication — write the value under post:123#0 … post:123#7 and have readers pick a random suffix. Spreads one key over eight shards.
Read from replicas for the shard that owns the hot key.

Detect hot keys with sampling in the client or the cache's own tooling (redis-cli --hotkeys), rather than guessing.

When nodes come and go

A node dies: its replica is promoted (seconds). Without replicas, that slice of keys is cold and those reads fall through to the database at once.
Adding capacity: with consistent hashing or slots, only the moved keys miss. Add nodes *before* peak, not during it.
A full cold start (deploying a new cluster, a region failover) is the dangerous one. Warm the cache from a snapshot or replay recent traffic, and rate-limit how fast misses can hit the database.

[!WARNING] The thundering herd When a hot key expires, thousands of requests miss at the same moment and all query the database. Use request coalescing (one request per key fetches; the rest wait on it), a short lock on refill (SET lock:key NX EX 5), or jittered TTLs so keys don't all expire together.

Consistency with the database

A distributed cache doesn't change the basic choices — cache-aside with TTLs plus delete-on-write is still the default — but it adds failure modes:

Delete, don't update, on write. Two concurrent writers updating the cache can leave it on the older value; deleting forces the next read to fetch the truth.
Invalidations can be lost if a cache node is briefly unreachable. A TTL is your safety net: every key should have one.
For stricter freshness, drive invalidation from the database's change stream (CDC) instead of application code, so no write path can forget.

Eviction and memory

Size the cluster for the working set, not the whole dataset. Use an LRU or LFU eviction policy (allkeys-lru / allkeys-lfu in Redis) and watch the eviction rate: steady evictions of keys that are read again soon mean the cluster is too small.

[!INTERVIEW] What a strong answer sounds like "Profiles are ~100 GB with 400k reads/sec, so I'll run a Redis Cluster of 6 shards, each with one replica, using hash slots so we can reshard without a cold cache. Reads are cache-aside with a 1-hour TTL and we delete on update. Celebrity profiles get a 2-second in-process cache on each API server so one hot key can't melt a shard. If a shard fails, its replica takes over; if we lose a whole cluster, we rate-limit misses to protect Postgres while it warms."