Courses › System Design › Question Breakdowns

Design Metrics Monitoring & Alerting

Series and cardinality, compressed time-series storage, downsampling, fast queries and alerting that survives outages.

Design a metrics monitoring and alerting system like Prometheus, Datadog or Google's Monarch: collect CPU, memory, request rates and custom metrics from hundreds of thousands of machines (or GPUs), store them as time series, power dashboards, and page engineers when something breaks. It's a question about high-volume time-series ingestion, cardinality, compression and downsampling, fast range queries, and alerting that's reliable when everything else is failing.

Requirements

Functional

  • Hosts and services emit metrics (name + labels + numeric value over time)
  • Engineers query metrics over time ranges with aggregation (avg cpu by cluster over 6h) for dashboards
  • Teams define alert rules (error rate > 5% for 5 minutes) and get notified (page, Slack, email)
  • Data is retained for a long time at reduced resolution

Out of scope: logs and traces (mention as siblings), anomaly-detection ML, the dashboard UI.

Non-functional

  • Ingest: ~10M data points/second from ~500k hosts (or GPU nodes)
  • Query latency: dashboards in < 1s for recent data
  • Alerting latency: alert within ~1 minute of the condition
  • Highly available, especially alerting — the monitoring system must work during an outage
  • Cost-efficient storage: years of data

[!TIP] Cardinality is the hidden scale The number of data points isn't the hardest part — it's the number of unique series (metric name × every label combination). One careless label like user_id or request_id multiplies series by millions and takes the system down. Mention cardinality limits early.

Capacity estimates

Core Entities

  • Series — metric name + label set (http_requests_total{service="api", region="us", code="500"}) → series ID
  • Sample — series ID, timestamp, value
  • Alert rule — query, threshold, duration, severity, routing
  • Alert — rule, state (pending / firing / resolved), labels, started at

API

POST /v1/write   (batched samples, protobuf)  [{ "labels": {...}, "samples": [[1760190000, 0.73], ...] }]

GET  /v1/query_range?query=sum by (region)(rate(http_requests_total{code="500"}[5m]))&start=...&end=...&step=60

POST /v1/rules   { "expr": "error_ratio > 0.05", "for": "5m", "severity": "page", "route": "team-api" }

High-Level Design

hosts = client "Hosts / GPUs\n(agents)" @0,1
ing = service "Ingestion Gateways" @1,1
kafka = queue "Write Buffer\n(Kafka)" @2,1
tsdb = store "TSDB Shards\n(hot, in memory + disk)" @3,1
cold = blob "Object Storage\n(downsampled blocks)" @4,1
query = service "Query Service" @3,0
dash = client "Dashboards" @4,0
alert = service "Alert Evaluator" @3,2
notify = service "Notifier\n(page, Slack)" @4,2
hosts -> ing : push every 10 s
ing -> kafka
kafka -> tsdb
tsdb -> cold : compaction
dash -> query
query -> tsdb
query -> cold
alert -> tsdb
alert -> notify
  • Agents on each host collect metrics and push batches every ~10s (or the system pulls/scrapes them, Prometheus-style).
  • Ingestion gateways validate, enforce cardinality limits, and write to a Kafka buffer that absorbs bursts and lets the TSDB be restarted safely.
  • TSDB shards (sharded by series) keep recent data in memory and compressed blocks on disk; older blocks are downsampled and moved to object storage.
  • Queries fan out to the shards holding the series and merge results.
  • The alert evaluator runs rules every ~30–60s and sends firing alerts to the notifier, which dedups, groups and routes them.

Deep Dives

1. Push or pull?

Either works; at large scale, a push-based agent feeding a buffer is common, with absence-of-data alerts to catch silent hosts.

2. Storing time series

  • Series ID = hash of the sorted label set. An inverted index maps each label pair to the series IDs that have it (region=us → [ids]), so {service="api", code="500"} is a posting-list intersection.
  • Samples are appended per series into chunks (e.g. 2 hours). Gorilla-style compression — delta-of-delta timestamps and XOR-encoded floats — gets ~1.4 bytes per sample, ~10× smaller than raw.
  • Recent chunks live in memory with a write-ahead log for crash recovery; full chunks are flushed into immutable blocks and compacted.

3. Downsampling and retention

Keep raw 10s data for ~15 days, 1-minute rollups (min/max/avg/sum/count) for ~3 months, 1-hour rollups for years. Compaction jobs build the rollups and store them in object storage; the query planner picks the coarsest resolution that satisfies the requested step.

4. Cardinality control

  • Enforce per-tenant limits on active series and reject or drop new series beyond them.
  • Flag labels whose distinct values explode (IDs, timestamps, raw URLs) at ingestion.
  • Pre-aggregate at the agent (e.g. histograms instead of per-request series).

[!WARNING] Don't alert through the system you're monitoring If the alert evaluator depends on the same overloaded cluster, a big outage silences your alerts exactly when you need them. Run alerting independently (separate shards or replicas, minimal dependencies), and add a dead man's switch: an always-firing heartbeat alert whose absence pages someone.

5. Alerting

  • Rules are evaluated on a schedule; a condition must hold for its for duration to fire (avoids flapping).
  • The notifier deduplicates (the same alert from replicas), groups related alerts (100 hosts down → one page), applies silences and on-call routing, and escalates unacknowledged pages.
  • Run evaluators in an HA pair; the notifier dedups their duplicate alerts.

6. Query performance

Dashboards repeat the same queries every refresh: cache query results by (query, aligned time range); precompute expensive aggregations with recording rules (a rule that writes a new, small series). Shard queries by time range and series and merge partial aggregates.

What's expected at each level

  • Mid-level: agents sending metrics, a time-series store, dashboards querying it, threshold alerts.
  • Senior: series identity and inverted label index, compression and chunking, downsampling with retention tiers, a buffer for ingestion bursts, alert dedup and grouping.
  • Staff: cardinality governance, alerting independence and dead man's switches, push vs. pull trade-offs, recording rules, and multi-tenant isolation at 100M active series.