Availability & Fault Tolerance

Nines, SLOs and error budgets, redundancy, failover models, multi-region and blast radius.

Everything fails eventually: disks, machines, racks, whole data centers, dependencies and deployments. Availability is the fraction of time your system works; fault tolerance is how you keep it working when parts of it don't. In interviews, the non-functional requirement "highly available" should turn into specific choices about redundancy, failover and how the system degrades.

Measuring availability

An SLI is what you measure (successful requests ÷ total requests). An SLO is your target (99.9% over 30 days). An SLA is a contract with penalties, set below your SLO.
The gap between 100% and the SLO is the error budget: spend it on releases and experiments; when it's exhausted, slow down.
Dependencies multiply: a request that needs five services at 99.9% each is about 99.5% available. Reduce hard dependencies on the critical path.

Redundancy: no single points of failure

Run at least two of everything on the request path — load balancers, app servers, databases — spread across availability zones.
Stateless services are easy: put several instances behind a load balancer with health checks.
Stateful components need replication and a failover plan.
Plan for N+1 or N+2 capacity: losing one zone of three means the other two must absorb 50% more traffic.

Failover models

Two numbers to state for disaster recovery: RPO (how much data you can lose — determined by replication lag) and RTO (how long recovery can take — determined by the failover model).

Multi-region

Route users to the nearest healthy region with DNS or anycast.
Read-local, write-to-home: replicate data everywhere for reads, but send writes for each user or tenant to one home region to avoid conflicts.
Fully active–active writes require multi-leader or globally consistent databases (Spanner, DynamoDB Global Tables) and a conflict strategy.

Limiting the blast radius

Cells / shards — split users into independent cells, each a full copy of the stack. A bad deploy or poisoned request takes down one cell, not everyone.
Bulkheads — separate thread pools, connection pools and queues per dependency so one slow dependency can't exhaust shared resources.
Staged rollouts — deploy to a canary, then one zone, then one region, with automatic rollback on error-rate increases. Most outages are caused by changes, not hardware.

Graceful degradation

Decide in advance what the system does when a dependency fails:

Serve stale cached data if the database is slow.
Disable non-critical features (recommendations, "who's online") to protect core flows (checkout, messaging).
Queue writes and process them later instead of failing.
Return a default response — a generic home feed instead of a personalized one.

See Resilience Patterns for timeouts, retries, circuit breakers and load shedding.

[!INTERVIEW] Turn "highly available" into specifics "The target is 99.99% for posting and reading messages. App servers are stateless across three zones. The database has a synchronous replica in a second zone, so RPO is zero and automated failover is under a minute. If the recommendations service fails, the feed falls back to a chronological list instead of erroring."