Ingesting Time-Series Metrics

High-volume ingestion, compressed time blocks, downsampling, retention and the cardinality trap.

Metrics, sensor readings, stock ticks, server CPU every 10 seconds: time-series data is a stream of (series, timestamp, value) points that arrives constantly, is almost never updated, and is mostly queried as aggregates over recent time ranges. Those properties let you build something far cheaper than a general-purpose database — if you design around them.

What makes time-series different

Write-heavy and append-only. 10,000 servers × 500 metrics every 10 seconds = 500,000 points per second, around the clock. Points are never edited.
Reads are ranges and rollups. "p99 latency for the checkout service over the last 6 hours, per minute" — not "fetch point #8,124,551".
Recent data is hot. Dashboards and alerts read the last hour constantly; last month's data is read rarely and at coarse resolution.
Old data loses detail value. Nobody needs 10-second CPU samples from last year. Hourly averages are enough.

The data model

A series is a metric name plus a set of labels (tags); each series is a list of timestamped values.

http_requests_total{service="checkout", region="us-east", status="500"}
  12:00:00  1,042
  12:00:10  1,051
  12:00:20  1,063

[!WARNING] Cardinality is the real limit Every unique combination of label values is a separate series. Labels with a handful of values (region, status) are fine. A label like user_id or request_id turns one metric into millions of series and will exhaust memory in the index. Keep high-cardinality detail in logs or traces, and keep metrics to bounded labels.

The ingestion pipeline

Agents -> {Kafka} -> [Ingesters] -> (TSDB: hot, recent blocks)
(TSDB: hot, recent blocks) -> [Compactor + Downsampler] -> (Object storage: old blocks)

Agents on each host collect and batch points (send every few seconds, not per point), and buffer locally if the backend is unreachable.
Kafka absorbs bursts and decouples collection from storage, so a slow database doesn't make agents drop data. Partition by series so one series' points stay in order.
Ingesters buffer the recent window of each series in memory, append to a write-ahead log for durability, and periodically flush compressed blocks.
Compaction merges small blocks into larger ones and downsampling produces 1-minute and 1-hour rollups for older data.

Pull vs push: Prometheus pulls (scrapes each target's /metrics endpoint), which makes it easy to tell when a target is down; most hosted systems accept pushes, which works better for short-lived jobs and across firewalls.

Storage tricks

Write-optimized storage. An LSM-tree or a purpose-built TSDB turns a flood of small writes into large sequential ones (see Scaling Writes). Wide-column stores like Cassandra also work, with the series as the partition key and time as the clustering key.

Time-partitioned blocks. Store data in blocks of, say, 2 hours. Queries skip blocks outside their time range, and retention becomes "delete whole old blocks" — no expensive row-by-row deletes.

Compression. Consecutive points are extremely similar, so compress each series in its block:

Timestamps use *delta-of-delta* encoding: scrapes every 10 seconds produce deltas of 10, 10, 10… so the delta-of-delta is almost always 0 and fits in a single bit.
Values are XOR'd with the previous value; slowly-changing floats share most of their bits, so only the few differing bits are stored.

Facebook's Gorilla paper reported an average of 1.37 bytes per point with these techniques, down from 16 bytes raw — roughly 12× smaller.

500k points/s × 86,400 s/day ≈ 43 billion points/day
× 1.4 bytes ≈ 60 GB/day of raw-resolution data
keep 15 days raw (≈ 900 GB), then 1-minute rollups for a year

Downsampling and retention

Store several aggregates per rollup bucket — min, max, sum, count — not just the average: you can't recompute a max from averages, and the sum and count let you combine buckets correctly.

Querying fast

Inverted index on labels finds the matching series (service="checkout" AND status="500") before any data is read.
Pick the resolution from the time range: a 1-year chart reads hourly rollups, never raw points.
Pre-aggregate expensive queries with recording rules, or cache rendered dashboard panels for a few seconds.
Percentiles don't add up. You can't average p99s across servers; store histograms (bucket counts) or sketches, which can be merged.

Alerting

Alert rules run every 15–60 seconds against recent data. Evaluate them on the hot, in-memory tier so alerting keeps working when the long-term store is slow, and alert on the absence of data too — a silent host is often a dead one.

[!INTERVIEW] What a strong answer sounds like "We're ingesting about 500k points per second. Agents batch every 10 seconds into Kafka, partitioned by series ID. Ingesters keep the last two hours per series in memory, write a WAL, and flush compressed 2-hour blocks — delta-of-delta timestamps and XOR'd values, roughly 1.4 bytes a point. A compactor downsamples to 1-minute and 1-hour rollups and moves old blocks to S3. We cap label cardinality per metric so one bad deploy can't create millions of series."