Rate Limiters

Token bucket and window algorithms, where limits run, and distributed counters in Redis.

A rate limiter decides whether a request is allowed, based on how many requests the same client has made recently. It's a small component that shows up everywhere: in front of public APIs, on login endpoints, around expensive downstream calls. This page covers the building block; Throttling with Rate Limits goes deeper on distributed counting, and Design a Rate Limiter is the full interview question.

What it protects against

Abuse: credential stuffing, scraping, spam.
Noisy neighbors: one tenant's runaway script starving everyone else.
Overload: shedding load before your database falls over.
Cost: capping calls to paid third-party APIs, or GPU time per user for an LLM product.

The algorithms

[!TIP] Default to token bucket It's simple, cheap, and matches how people think about limits: "100 requests per minute, with bursts of 20 allowed." It's what AWS, Stripe and most API gateways use.

def allow(bucket, now, rate, capacity):
    elapsed = now - bucket.last_refill
    bucket.tokens = min(capacity, bucket.tokens + elapsed * rate)
    bucket.last_refill = now
    if bucket.tokens >= 1:
        bucket.tokens -= 1
        return True
    return False

Where it runs

At the edge (CDN or WAF): cheap blocking of obvious abuse by IP.
In the API gateway: the usual place, keyed by API key or user ID, before requests reach your services.
In the service itself: for limits that need business context, such as "5 password resets per account per hour".

Distributed limits

With many gateway nodes, each one needs to see the same count. The standard answer is a shared counter in Redis, updated with an atomic operation or a small Lua script so check-and-decrement can't race. Each check adds a network hop (well under a millisecond inside a data center).

At very high volume, nodes can keep local counters and sync with Redis periodically, accepting a small overshoot in exchange for fewer round trips.

Decide what happens when Redis is unreachable: fail open (allow traffic, risk overload) for most APIs, or fail closed for security-sensitive endpoints such as login.

Responding to limited clients

Return HTTP 429 Too Many Requests with a Retry-After header, and ideally RateLimit-Limit / RateLimit-Remaining headers so well-behaved clients can slow down before they hit the wall. Clients should retry with exponential backoff and jitter.

[!INTERVIEW] Name the key, the algorithm and the store "The gateway applies a token bucket per API key — 100 requests per second, bursts of 200 — with counters in Redis via a Lua script. Over the limit, we return 429 with Retry-After. If Redis is down, we fail open and fall back to per-node local limits."