Courses › System Design › Question Breakdowns

Design a Market Data Feed

Feed arbitration and gap recovery, snapshot + sequenced updates, conflation for slow clients, bars and corrections.

Design a real-time market data service like the price feeds behind the Bloomberg Terminal or a trading app: take millions of price updates per second from exchanges, and deliver live quotes and charts to hundreds of thousands of users, each watching their own list of symbols. It's a fan-out question with finance's twists: bursty, ordered updates, gap detection and recovery, conflation for slow consumers, and historical bars for charts.

Requirements

Functional

  • Ingest trades and quotes from many exchanges/venues for ~100k instruments
  • Users subscribe to symbols (a watchlist) and receive live updates (last price, bid/ask, volume)
  • Users can load charts: OHLC bars (1-minute to daily) and recent trades
  • Professional users can request full-tick data; retail users get conflated updates

Out of scope: news, analytics, order entry (see Design a Stock Exchange), licensing/billing per exchange (mention entitlements).

Non-functional

  • Ingest: ~5M updates/second at the open, bursting to 10M+
  • Low latency: professional feed within milliseconds; retail apps within ~100ms–1s
  • Ordered and complete: per instrument, updates arrive in order; gaps are detected and repaired
  • Scale out to clients: ~500k concurrent connections, each subscribed to ~50 symbols
  • Entitlements: users only see data from exchanges they're licensed for

[!TIP] Slow consumers are the real problem A phone on a train can't take 1,000 updates/second for a hot stock. The server must conflate — send only the latest price per symbol at the client's pace — instead of queueing every tick and falling minutes behind. Interviewers wait for this.

Capacity estimates

Core Entities

  • Instrument — symbol, venue(s), type, tick size
  • Tick — instrument, type (trade/quote), price, size, venue, exchange sequence, timestamps
  • Snapshot — latest state per instrument (last, bid, ask, day OHLC, volume)
  • Bar — instrument, interval, open, high, low, close, volume
  • Subscription — connection, instruments, delivery mode (full / conflated), entitlements

API

WS /v1/stream
  → { "op": "subscribe", "symbols": ["AAPL", "MSFT", "BTC-USD"], "mode": "conflated" }
  ← { "type": "snapshot", "symbol": "AAPL", "last": 189.52, "bid": 189.51, "ask": 189.53, "seq": 99120 }
  ← { "type": "update", "symbol": "AAPL", "last": 189.55, "seq": 99131 }

GET /v1/bars?symbol=AAPL&interval=1m&from=...&to=...

High-Level Design

ex = client "Exchange Feeds\n(multicast)" @0,1
fh = service "Feed Handlers\n(normalize)" @1,1
bus = queue "Internal Bus\n(partitioned by symbol)" @2,1
snap = cache "Snapshot Cache\n(latest per symbol)" @3,0
fan = service "Distribution Servers\n(conflation)" @3,1
bars = service "Bar Builder" @3,2
ts = store "Tick & Bar Store\n(time series)" @4,2
users = client "Terminals / Apps" @4,1
ex -> fh
fh -> bus
bus -> snap
bus -> fan
bus -> bars
bars -> ts
bus --> ts : raw ticks
users <-> fan : WebSocket
fan -> snap : on subscribe
users -> ts : chart history
  • Feed handlers consume each exchange's feed (often UDP multicast), detect gaps using the exchange's sequence numbers, and normalize messages into one internal format.
  • An internal bus partitioned by symbol keeps each symbol's updates in order.
  • A snapshot cache keeps the latest state per symbol, so new subscribers get a full picture instantly.
  • Distribution servers hold client connections, track subscriptions and push updates — full-tick or conflated.
  • A bar builder aggregates ticks into OHLC bars; ticks and bars go to a time-series store for charts.

Deep Dives

1. Gaps and recovery from exchange feeds

Exchange feeds are UDP multicast with per-channel sequence numbers: fast, but packets get lost.

  • Exchanges send two identical feeds (A/B) on separate networks; the handler arbitrates — takes whichever copy of each sequence number arrives first.
  • If both miss a sequence number, request it from the exchange's retransmission service, or rebuild state from a snapshot feed.
  • Downstream, every normalized message carries its own sequence per symbol so consumers can detect gaps too.

2. Subscribe = snapshot + updates

A new subscriber needs the current state, then every change after it, with nothing missed in between:

  • Register the subscription (start buffering updates for that symbol).
  • Send the snapshot with its sequence number.
  • Send buffered and live updates with a higher sequence; drop older ones.

Clients that detect a gap (sequence jumps) re-request the snapshot.

3. Conflation for slow consumers

For each connection, keep a per-symbol "latest value" slot instead of a queue:

{Updates for AAPL} -> [Overwrite latest AAPL slot] -> [Send when socket is writable or every 250 ms] -> Client

A fast client gets nearly every update; a slow one gets the latest price at its pace, never a growing backlog. Professional feeds that need every tick use dedicated, well-provisioned connections (and disconnect consumers that fall too far behind).

4. Fan-out at scale

  • Distribution servers subscribe to the bus only for symbols their clients want, then fan out locally — popular symbols (AAPL, BTC) are delivered once per server, not once per client.
  • Consistent hashing or a subscription registry spreads clients across servers; servers are regional, close to users.
  • Entitlements are checked at subscribe time (exchange licenses per user), and delayed data (15-minute) is served from a separate delayed stream for unlicensed users.

5. Bars and history

The bar builder keeps the current bar per symbol per interval in memory and closes it on the interval boundary (late trades correct the bar). Store ticks and bars in a time-series / columnar store (kdb+, ClickHouse, or a TSDB), partitioned by date and symbol; charts read bars, rarely raw ticks.

[!NOTE] Corrections happen Exchanges cancel or correct trades after the fact (busted trades). Bars and stored ticks must accept corrections, and clients should receive a correction message rather than silently changed history.

What's expected at each level

  • Mid-level: ingest, a pub/sub fan-out over WebSockets, latest price per symbol, stored history for charts.
  • Senior: partitioning by symbol for ordering, snapshot + sequenced updates on subscribe, conflation for slow clients, bar aggregation.
  • Staff: A/B feed arbitration and retransmission, full-tick vs. conflated tiers, entitlements and delayed data, corrections, and regional fan-out economics.