Resilience Patterns

Timeouts, retries with jitter, circuit breakers, bulkheads, load shedding and backpressure.

Most large outages aren't caused by a component dying — they're caused by a component getting slow, and the slowness spreading. Requests pile up, threads and connections run out, retries multiply the load, and a single struggling dependency takes down everything upstream. These patterns stop that cascading failure. They pair with Fault Tolerance, which covers redundancy and failover.

Timeouts

Every network call needs a timeout. Without one, a hung dependency holds a thread or connection forever.
Set timeouts from the dependency's latency distribution — e.g. a little above its p99 — not a round number like 30 seconds.
Use deadlines that propagate: if the user's request has 800 ms left, downstream calls get what remains, not a fresh full timeout each. gRPC supports this natively.

Retries with backoff and jitter

Retry only transient errors (timeouts, 503s, connection resets), never 4xx client errors.
Retry only idempotent operations, or make them idempotent with an idempotency key — see Safe Retries & Idempotency.
Use exponential backoff (100 ms, 200 ms, 400 ms…) with jitter (randomize each delay) so clients don't retry in synchronized waves.
Cap attempts (2–3), and use a retry budget — e.g. retries may add at most 10% extra load.

[!WARNING] Retry amplification If each of four layers retries 3 times, one failing request at the bottom becomes 3⁴ = 81 attempts. Retry at one layer (usually closest to the client or at the edge proxy), not at every layer.

Circuit breakers

Closed -> Open
Open -> Half-open
Half-open -> Closed
Half-open -> Open

Closed — calls flow normally while the breaker counts failures.
Open — after the error rate crosses a threshold, calls fail fast without hitting the dependency, giving it time to recover.
Half-open — after a cool-down, a few trial calls go through. Success closes the breaker; failure re-opens it.

Combine with a fallback: cached data, a default value, or a degraded feature. Proxies like Envoy and libraries like Resilience4j implement this.

Bulkheads

Isolate resources per dependency or per tenant — separate connection pools, thread pools or queues — so one slow dependency or noisy customer can only exhaust its own share. Named after the watertight compartments in a ship's hull.

Load shedding and backpressure

Load shedding — when overloaded, reject excess requests quickly (HTTP 503 or 429) instead of accepting everything and timing out on all of them. Shed low-priority traffic first (prefetches, analytics) to protect critical traffic (checkout).
Backpressure — let a slow consumer signal producers to slow down: bounded queues, TCP flow control, reactive streams, or consumer lag on Kafka that pauses ingestion.
Rate limiting — cap clients before they overload you. See Rate Limiters.
Bounded queues — an unbounded queue only hides overload until memory runs out, and every request in it is already stale.

Handling thundering herds

Request coalescing — when many requests miss the cache for the same key, let one fetch it and share the result.
Jittered TTLs so cached items don't all expire at once.
Gradual warm-up of new instances and staggered client reconnects after an outage.

Putting it together

[!INTERVIEW] Protecting checkout "Checkout calls the recommendations service with a 150 ms timeout behind a circuit breaker; on failure it simply shows no recommendations. Calls to the payment provider retry once with jitter using an idempotency key. Each dependency has its own connection pool, and when the service is over capacity the gateway sheds browse traffic before checkout traffic."