Courses › System Design › Question Breakdowns

Design a Notification System

Priority isolation, preferences and quiet hours, idempotent delivery, campaign throttling and provider failover.

A notification system is the shared service every other team uses to reach users: push notifications, SMS and email. The product teams say *what* to tell *whom*; this service decides *how* and *when*, and makes sure it happens exactly once. It's a question about reliable asynchronous delivery, user preferences, third-party providers that fail and throttle, and bulk sends that would otherwise overwhelm everything.

Requirements

Functional

  • Internal services can send a notification to a user on push, SMS or email (or "best channel")
  • The system respects user preferences: opted-out categories, preferred channels, quiet hours
  • Services can schedule notifications and send bulk campaigns to millions of users
  • Senders can track delivery status (sent, delivered, opened, failed)

Out of scope: in-app notification inbox UI, composing marketing campaigns, the push providers themselves.

Non-functional

  • Scale: ~1B notifications/day; bulk campaigns of 50M users
  • Latency: transactional notifications (OTP codes, "your driver has arrived") delivered within seconds
  • At-least-once delivery, no visible duplicates: a user never gets the same notification twice
  • Isolation: a 50M-user marketing campaign must never delay a password-reset code
  • Availability: degrade gracefully when a provider (APNs, Twilio, SES) is slow or down

[!TIP] Priority is the design Say early that notifications come in very different classes — critical (OTP, security alerts), transactional (order updates) and promotional (campaigns) — and that each class gets its own queues, workers and rate limits. Most deep-dive answers follow from that split.

Capacity estimates

Core Entities

  • Notification request — id, idempotency key, sender service, user (or segment), template, data, priority, scheduled time
  • User preferences — per category: allowed channels, opt-outs, quiet hours, time zone
  • Device / contact — push tokens per device, phone number, email, verification status
  • Template — per channel and locale
  • Delivery attempt — notification, channel, provider, status, timestamps, error

API

POST /v1/notifications
{
  "idempotencyKey": "order-9001-shipped",
  "userId": "u_42",
  "category": "order_updates",
  "priority": "transactional",
  "template": "order_shipped",
  "data": { "orderId": "9001", "eta": "Tue" },
  "channels": ["push", "email"],
  "sendAt": null
}
→ 202 { "notificationId": "n_8f3" }

POST /v1/campaigns   { "segmentId": "...", "template": "...", "sendAt": "..." }
GET  /v1/notifications/{id}   → status per channel

High-Level Design

svc = client "Product Services" @0,1
api = service "Notification API" @1,1
prefs = store "Preferences\n+ Contacts" @1,2
router = service "Router\n(prefs, dedupe, quiet hours)" @2,1
sched = store "Scheduled Store" @1,0
q = queue "Queues per\nchannel × priority" @3,1
push = service "Push Workers" @4,0
email = service "Email Workers" @4,1
sms = service "SMS Workers" @4,2
prov = note "APNs / FCM / SES / Twilio" @5,1
svc -> api
api --> sched : sendAt in future
api -> router
router -> prefs
router -> q
q -> push, email, sms
push -> prov
email -> prov
sms -> prov
  • Accept: the API validates the request, stores it with its idempotency key, and returns 202 immediately.
  • Decide: the router loads the user's preferences and contacts, drops opted-out categories, picks channels, applies quiet hours (delaying non-critical sends) and renders the template.
  • Queue: one message per channel goes onto a queue for that channel and priority.
  • Deliver: channel workers call the providers, record the attempt and retry failures.
  • Track: provider callbacks (delivered, bounced, opened) update the delivery status.

Deep Dives

1. Isolating priorities

Critical queues have reserved worker capacity and their own provider rate-limit budget. Promotional sends get whatever is left and are paused entirely during incidents.

2. No duplicates with at-least-once delivery

Duplicates come from senders retrying, from queue redelivery and from workers crashing after calling the provider.

  • API level: the sender's idempotencyKey is stored with a unique constraint; a repeated request returns the original notificationId.
  • Worker level: before calling the provider, the worker records (notificationId, channel) as sending with a conditional write; a redelivered message sees it's already in progress or sent and stops.
  • Provider level: pass the notification ID as the provider's idempotency key where supported.

A tiny window remains where a worker crashes after the provider accepted the message but before recording it. For push and email that rare duplicate is acceptable; say so. See Safe Retries & Idempotency.

3. Preferences, quiet hours and frequency caps

  • Preferences are read on every send, so cache them (Redis) with invalidation on update.
  • Quiet hours are evaluated in the user's time zone. Non-critical notifications are moved to the scheduled store for the end of quiet hours; critical ones go out anyway.
  • Frequency caps ("max 3 promotional pushes a day") use per-user counters with a TTL in Redis.
  • Collapse and digest: ten "someone liked your photo" events in a minute become one notification ("Ana and 9 others…") by buffering per user for a short window.

4. Scheduled and bulk sends

[Campaign] -> [Segment resolver: user IDs in batches] -> {Bulk queue} -> [Router] -> {Promotional queues} -> [Workers]
  • Scheduled notifications sit in a store indexed by send time; a poller moves due items into the pipeline. See Scheduling Future Work.
  • Campaigns are expanded in batches of ~1,000 users, not 50M individual API calls, and throttled to a target rate so they don't overwhelm providers or the app servers that users open after tapping the push.
  • Spread big campaigns over time and by time zone (send at 10 a.m. local).

5. Providers fail and throttle

  • Each provider has rate limits (APNs connections, Twilio per-number throughput, SES sending quotas). Workers use a token bucket per provider shared across workers.
  • Retries with exponential backoff and jitter for transient errors; give up after a deadline that depends on priority (an OTP code is useless after 5 minutes).
  • Permanent errors clean up data: an invalid push token is deleted, a hard email bounce marks the address undeliverable.
  • Failover: keep a second SMS and email provider and switch on error-rate thresholds using a circuit breaker. See Resilience Patterns.
  • Messages that exhaust their retries go to a dead-letter queue for inspection.

6. Delivery tracking

  • Every attempt is written to a delivery log (Cassandra or DynamoDB keyed by notification ID; also streamed to a warehouse for analytics).
  • Provider webhooks report delivered, bounced and opened; push opens come from the app itself.
  • Senders query status by notification ID, and aggregated metrics (delivery rate per channel and provider) drive alerting.

What's expected at each level

  • Mid-level: an asynchronous pipeline with queues and per-channel workers, user preferences, retries.
  • Senior: priority isolation, idempotency at the API and worker, quiet hours and scheduling, provider rate limits and failure handling.
  • Staff: campaign throttling and time-zone spreading, digesting and frequency caps, multi-provider failover, delivery analytics, and how the platform protects itself and downstream services from bulk traffic.