Throttling with Rate Limits

Token buckets and sliding windows, shared counters in Redis, and failing open vs closed.

Rate limiting caps how many requests a client can make in a window of time. It protects your services from abuse and runaway clients, keeps one noisy tenant from starving everyone else, and controls cost when you pay per call downstream. It shows up as a component in almost every design — and as a full question on its own in Design a Rate Limiter.

Decide three things first

Who is being limited? The key you count by: user ID or API key for authenticated traffic, IP address for anonymous traffic, or a combination (per user *and* per endpoint).
What's the limit? "100 requests per minute per user", "10 login attempts per hour per IP", "1,000 writes per second per tenant".
What happens when it's exceeded? Usually reject with HTTP 429 Too Many Requests and a Retry-After header. Sometimes queue, delay or degrade instead.

The algorithms

limit = 100/min, now = 12:01:15 (25% into the current minute)
previous window (12:00): 80 requests
current window  (12:01): 30 requests
estimate = 30 + 80 × (1 − 0.25) = 90  → allowed (90 < 100)

Token bucket is the usual default in interviews: it's cheap, it allows the short bursts real clients produce (a page load fires 10 calls at once), and it's what most API gateways and cloud providers implement.

Where to put the limiter

Client -> [API Gateway + Rate Limiter] -> [Services]
[API Gateway + Rate Limiter] -> (Redis counters)

At the gateway / edge — the common answer. One place, applied before any expensive work, shared by every service.
In each service — for limits only that service understands (e.g. "5 password resets per day").
In the client — politeness and backoff, never enforcement: clients can't be trusted.

Making it distributed

With 50 gateway instances, each one counting locally would allow 50× the limit. The counters need to be shared:

Central Redis is the standard answer. A fixed window is one INCR plus EXPIRE; a token bucket or sliding window runs as a small Lua script so the read-and-update is atomic.
Shard the counters by the limit key (consistent hashing over a Redis cluster) when one node can't handle the request rate.
Latency budget: a Redis round trip is ~0.5 ms in the same zone — acceptable on every request.

[!TIP] Trading accuracy for speed At very high volume, each gateway can keep a local counter and sync with Redis every 100 ms or so, allowing a small overshoot in exchange for no network hop on the hot path. Say explicitly that you're accepting, say, 5% over the limit.

Failure modes

Redis is down. Choose deliberately: fail open (let traffic through, keep the product up — the usual choice for general APIs) or fail closed (reject — right for login attempts or expensive operations).
Race conditions. "Read count, check, then increment" in two round trips lets concurrent requests slip past the limit. Do it in one atomic operation.
Clock skew between gateways matters for window boundaries. Use Redis's clock (inside the Lua script) rather than each server's.

Tell clients what's happening

Return the limit state on every response so well-behaved clients can pace themselves:

HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 100
RateLimit-Remaining: 0
Retry-After: 30

Clients should retry with exponential backoff and jitter, otherwise everyone retries at the same moment and you're rate-limiting a self-inflicted wave.

Beyond simple limits

Tiered limits per plan (free: 60/min, paid: 1,000/min), looked up with the API key.
Multiple limits at once: 10/sec *and* 1,000/hour, to stop both bursts and sustained scraping.
Cost-based limits: a search query costs 5 tokens, a cache read costs 1.
Load shedding: when the system itself is overloaded, drop low-priority traffic first, regardless of per-client limits.

[!INTERVIEW] What a strong answer sounds like "I'd enforce limits at the API gateway with a token bucket per API key — 100 tokens, refilling at 10 per second — stored in Redis and updated by a Lua script so it's atomic. Anonymous traffic is limited per IP. Over the limit we return 429 with Retry-After. If Redis is unavailable, the general API fails open, but login and payment endpoints fail closed."