Courses › System Design › Question Breakdowns

Design a Price Tracker

Demand-prioritized crawling under a budget, change-only price history, indexed alert matching and data quality.

A price tracker (like CamelCamelCamel or Honey) monitors product prices on e-commerce sites, shows price history charts, and alerts users when a product drops below their target price. It's a question about scheduling a huge crawl with limited budget (you can't check every product every minute), storing and serving price history, and efficiently matching price changes against millions of alerts.

Requirements

Functional

  • Users can search for or paste a product and see its price history chart
  • Users can set an alert: notify me when the price drops below $X
  • The system tracks prices for products over time
  • Users get notified (email or push) when an alert fires

Out of scope: coupon codes, buying through the tracker, sites other than a few large retailers.

Non-functional

  • Scale: ~500M tracked products, ~50M active alerts, ~10M users
  • Freshness: products with active alerts or high interest checked at least hourly; the long tail daily or weekly
  • Politeness: stay within retailers' rate limits and terms (official APIs where possible)
  • Alert latency: notify within minutes of a detected drop, and never notify twice for the same drop
  • Read-heavy charts: history pages are viewed far more than they change

[!TIP] The crawl budget is the design You can't refresh 500M products every hour. Say early that crawl frequency should be prioritized by demand — products with alerts and page views get checked often, the long tail rarely — and that price changes are what you store, not every observation.

Capacity estimates

Core Entities

  • Product — id, retailer, retailer product id (ASIN), title, image, current price, last checked, crawl priority
  • Price point — product, price, currency, observed time (stored on change)
  • Alert — user, product, target price, status (active / triggered), last notified price
  • User — contact details, notification preferences

API

GET  /v1/products/search?q=                   → products[]
POST /v1/products/resolve   { url }            → { productId }   (adds it to tracking if new)
GET  /v1/products/{id}/history?range=1y        → [{ t, price }]
POST /v1/alerts             { productId, targetPrice }
GET  /v1/alerts             → alerts[]

High-Level Design

user = client "Users" @0,1
api = service "API" @1,1
prod = store "Products DB" @2,0
hist = store "Price History\n(time-series)" @2,1
alerts = store "Alerts\n(by product, target)" @2,2
sched = service "Crawl Scheduler\n(priority queue)" @3,0
fetch = service "Fetchers\n(per-retailer limits)" @4,0
changes = queue "Price Changes" @4,1
matcher = service "Alert Matcher" @3,2
notif = service "Notifications" @4,2
user -> api
api -> prod, hist, alerts
sched -> fetch
fetch -> prod : update current price
fetch -> changes : on change
changes -> hist
changes -> matcher
matcher -> alerts
matcher -> notif
  • Schedule: the scheduler decides which products are due, based on each product's priority, and hands them to fetchers.
  • Fetch: fetchers call retailer APIs or fetch pages within per-retailer rate limits, extract the price, and compare with the current price.
  • Record changes: only changes are written to price history and published to a stream.
  • Match alerts: the alert matcher finds alerts for that product whose target is now met and sends notifications.
  • Serve: charts read the history store through a cache.

Deep Dives

1. Prioritizing the crawl

  • Each product gets a next check time computed from a priority score: number of active alerts, recent page views, how often its price changes, and whether it's near an alert threshold.
  • The scheduler pulls due products from a time-ordered index (a sorted set or a next_check index) and pushes them to per-retailer queues.
  • Products whose price changes often get checked more often (adaptive); products that never change back off.
  • Spread checks with jitter so they don't all hit at the top of the hour.

This is the same frontier idea as a web crawler. See also Crawling & Large Data Pipelines.

2. Fetching politely and reliably

  • Prefer official product APIs and affiliate feeds; they're cheaper and more reliable than scraping.
  • Enforce per-retailer rate limits with a token bucket per retailer shared across fetchers.
  • Parsers are per retailer and break when layouts change — monitor extraction failure rates per retailer and alert.
  • Validate prices before accepting them (a $0.01 or 100× jump is usually a parsing error or a third-party seller glitch); require confirmation by a second fetch before alerting.

[!WARNING] Alerting on a bad parse A parsing bug that reads "$1,299" as "$1" will fire alerts to thousands of users. Sanity-check large drops and re-fetch to confirm before notifying.

3. Storing price history

  • Store a point only when the price changes, plus periodic "still this price" markers so charts know the data is fresh.
  • Access pattern: one product's history over a time range. A wide-column store keyed by (product_id, time) (Cassandra, Bigtable) or a time-series database fits well. See Time-Series Databases.
  • Precompute summary stats per product (all-time low, 30-day average) on write so product pages don't scan history.
  • Cache rendered chart data for popular products with a TTL matching their crawl interval.

4. Matching price changes to alerts

For each price change, find alerts on that product with target_price >= new_price that haven't already fired.

  • Store alerts indexed by (product_id, target_price). A change is one range query: WHERE product_id = ? AND target_price >= ? AND status = 'active'.
  • Popular products may have 100k alerts; process matches in batches and hand them to the notification system in bulk.
  • Mark alerts as triggered (or record the price they last fired at) with a conditional update so a redelivered price-change event doesn't notify twice. Re-arm when the price goes back above the target, if the user wants repeated alerts.

5. Crowd-sourced prices

A browser extension sees the price whenever a user visits a product page. Use those observations as free fetches:

  • Extensions report (product, price, time); the server treats them as low-trust observations and confirms surprising ones with its own fetch.
  • They keep popular products fresh without spending crawl budget, and they let the system discover new products to track.

What's expected at each level

  • Mid-level: a scheduler and fetchers, a price history store, alerts checked when prices change, notifications.
  • Senior: demand-based crawl prioritization, per-retailer rate limiting, storing only changes, indexed alert matching and idempotent notifications.
  • Staff: adaptive crawl scheduling under a fixed budget, data quality checks before alerting, crowd-sourced observations and trust, and the cost trade-offs of freshness versus fetch volume.