Courses › System Design › Question Breakdowns

Design a Webhook Delivery System

At-least-once delivery, backoff with jitter, isolating bad endpoints, signatures and SSRF defense.

Design the system a platform like Stripe or GitHub uses to notify customers' servers about events (payment.succeeded, push) by calling their HTTPS endpoints. You don't control the receivers: they're slow, flaky, down for hours or misconfigured. It's a question about reliable at-least-once delivery, retries with backoff, isolating bad endpoints, ordering, and security (signatures).

Requirements

Functional

  • Customers register endpoints (URL + event types to receive)
  • When an event occurs, every matching endpoint receives an HTTP POST with the event payload
  • Failed deliveries are retried with backoff for up to ~3 days, then the endpoint is flagged/disabled
  • Customers can see delivery attempts, and replay events manually
  • Receivers can verify that a webhook really came from us

Out of scope: the systems producing the events; customer-side handling.

Non-functional

  • At-least-once delivery: no event is ever silently dropped
  • Scale: ~1B events/day (~12k/s, peaks ~50k/s), ~1M endpoints
  • Timely: most deliveries within seconds of the event
  • Isolation: one slow or dead endpoint must not delay anyone else's webhooks
  • Secure: payloads signed; we never become a tool for attacking internal networks (SSRF)

[!TIP] The hard part is other people's servers Producing events is easy. The design is dominated by the fact that ~5–10% of endpoints are failing at any moment, some take 30 seconds to respond, and some hang forever. Every decision — timeouts, queues per endpoint, backoff, circuit breakers — is about not letting the bad ones hurt the good ones.

Capacity estimates

Core Entities

  • Endpoint — customer, URL, subscribed event types, signing secret, status (active / failing / disabled)
  • Event — ID, type, payload, created at
  • Delivery — event, endpoint, status, attempt count, next attempt at
  • Attempt — delivery, timestamp, HTTP status, latency, error

API

POST /v1/webhook_endpoints   { "url": "https://shop.example.com/hooks", "events": ["payment.*"] }
GET  /v1/webhook_endpoints/{id}/deliveries?status=failed
POST /v1/events/{eventId}/redeliver   { "endpointId": "we_9" }

# what the receiver gets
POST https://shop.example.com/hooks
Webhook-Id: evt_7a1
Webhook-Timestamp: 1760190000
Webhook-Signature: v1,Hk3...   (HMAC-SHA256 of id.timestamp.body)
{ "id": "evt_7a1", "type": "payment.succeeded", "data": { ... } }

High-Level Design

prod = service "Event Producers" @0,0
bus = queue "Event Stream\n(Kafka)" @1,0
fan = service "Fan-out Service" @2,0
ep = store "Endpoints DB" @2,1
q = queue "Delivery Queues\n(sharded by endpoint)" @3,0
w = service "Delivery Workers" @4,0
retry = store "Retry Schedule\n(next attempt at)" @3,1
log = store "Attempt Log" @4,1
cust = client "Customer Endpoints" @5,0
prod -> bus
bus -> fan
fan -> ep : who subscribes?
fan -> q : one delivery per endpoint
w -> q : pull
w -> cust : signed POST
w -> log
w -> retry : on failure
retry -> q : when due
  • Producers write events to a durable stream.
  • The fan-out service looks up subscribed endpoints and creates one delivery per (event, endpoint).
  • Delivery workers POST the signed payload with a strict timeout. A 2xx is success; anything else (or timeout) is a failure.
  • Failures are written to a retry schedule with the next attempt time; a scheduler re-enqueues them when due.
  • Every attempt is logged for the customer dashboard and for manual replay.

Deep Dives

1. Retries with exponential backoff

A typical schedule: immediately, then ~5s, 5m, 30m, 2h, 5h, 10h, 10h… for up to ~3 days, each with random jitter so thousands of retries don't hit a recovering endpoint at the same instant.

  • Retries live in a time-indexed store (DB table indexed by next_attempt_at, or a delay queue) — not in a worker's memory, so a worker crash loses nothing.
  • Retry on timeouts, connection errors, 5xx and 429. Treat most 4xx as failures too (the customer's code may be broken) but record them distinctly; 410 Gone can disable the endpoint immediately.
  • After exhausting retries, mark the delivery failed, email the customer, and keep it replayable.

2. Isolating bad endpoints

If all deliveries share one queue, a backlog of retries to a dead endpoint — or a slow endpoint holding workers for 30s each — delays everyone.

  • Strict timeouts (e.g. 10–15s total) so no call ties up a worker for long.
  • Shard queues by endpoint (or hash of endpoint into many partitions) and cap concurrency per endpoint (e.g. 10 in-flight requests), so one endpoint can only use its share of workers.
  • Circuit breaker per endpoint: after N consecutive failures, stop sending live traffic and probe occasionally; send everything queued when it recovers. Endpoints failing for days are disabled.
  • Use async I/O workers (one worker can hold thousands of in-flight HTTP calls) since the work is almost entirely waiting.

3. At-least-once, and helping receivers be idempotent

Duplicates are unavoidable: the receiver may process the event and time out before responding. We can't fix that, so we make it easy to handle: every delivery of an event carries the same event ID, and docs tell receivers to dedup by it.

[!WARNING] Never ack the event before the delivery is durable The fan-out service must commit deliveries (to the queue or DB) before committing its offset on the event stream. Otherwise a crash between the two drops events for some endpoints — the one failure webhooks are not allowed to have.

4. Ordering

Strict per-endpoint ordering means one failing event blocks every later event for that endpoint (head-of-line blocking). Most platforms don't guarantee order and instead include a timestamp and resource version in the payload, so receivers can ignore stale updates — or send "thin" events and let receivers fetch the latest state via the API. If ordering is a hard requirement, partition by (endpoint, resource ID) and deliver serially within each partition.

5. Security

  • Signatures: HMAC-SHA256 over id.timestamp.body with a per-endpoint secret. Receivers verify in constant time and reject timestamps older than ~5 minutes (replay protection). Support two active secrets during rotation.
  • SSRF protection: customers choose the URLs, so a malicious one could target http://169.254.169.254 or internal hosts. Resolve DNS and block private/reserved IP ranges, re-check after redirects (or don't follow redirects), and send from a dedicated egress network with a published IP list.
  • HTTPS only; never include secrets in payloads.

6. Scaling the fan-out

Most events go to 0–2 endpoints, but some (a platform-wide event in a multi-tenant integration) go to thousands. Cache endpoint subscriptions by (customer, event type); for large fan-outs, write deliveries in batches. The event stream partitioned by customer keeps each customer's fan-out on one consumer.

What's expected at each level

  • Mid-level: endpoints table, a queue of deliveries, workers that POST with retries, an attempt log.
  • Senior: exponential backoff with jitter in a durable retry store, per-endpoint isolation and timeouts, HMAC signatures with timestamps, at-least-once with event IDs for dedup.
  • Staff: circuit breakers and endpoint health lifecycle, ordering trade-offs and head-of-line blocking, SSRF defense, durable fan-out handoff, async I/O worker economics.