Courses › System Design › Key Technologies

Bigtable & HBase

Sorted wide-column stores: row-key design, tablets, LSM storage and avoiding hot spots.

Google Bigtable and its open-source clone Apache HBase are wide-column stores built for massive, sorted key-value data with high write throughput — billions of rows, petabytes, millions of operations per second. Bigtable backs Google Search, Maps and Analytics. In interviews they fit time-series, IoT, user-activity history and any design where you scan ranges of a well-chosen row key.

Data model

  • A table is a sorted map: row key → column family → column qualifier → timestamp → value.
  • Rows are sorted lexicographically by row key. That single fact drives every design decision.
  • Column families are declared up front and stored together; columns within a family are created on the fly, so rows can have millions of sparse columns.
  • Each cell can keep several timestamped versions, with garbage collection by count or age.
  • Single-row operations are atomic; there are no multi-row transactions or secondary indexes.
row key = <device_id>#<reverse_timestamp>
sensor-0042#9223370270000000000   → cf:temp=21.4, cf:humidity=0.53
sensor-0042#9223370270000060000   → ...

Scanning the prefix sensor-0042# returns that device's readings, newest first, because the timestamp is reversed.

Architecture

  • Tables are split into contiguous key ranges — tablets in Bigtable, regions in HBase — served by tablet/region servers. Hot or large ranges are split automatically.
  • Writes go to a commit log and an in-memory buffer, then are flushed to immutable sorted files on a distributed file system (Colossus for Bigtable, HDFS for HBase) and compacted later — an LSM tree, like Cassandra. See Write-Ahead Logging (WAL).
  • Because storage is shared, a failed server's ranges can be reassigned in seconds without copying data.
  • Unlike Cassandra's leaderless design, each key range has exactly one serving node, which gives strong consistency per row.

Row-key design

[!WARNING] Avoid hot spots A row key that starts with a timestamp or a sequential ID sends every new write to the last tablet, which then becomes a bottleneck. Lead with a well-distributed field (device ID, user ID, or a hash), then append the time.

  • Promote fields into the key that you filter on, in the order you filter: user_id#event_type#timestamp.
  • Salt extremely hot prefixes with a small bucket number, and scan all buckets when reading.
  • Keep keys short — they're stored with every cell.
  • Need a second access pattern? Maintain a second table keyed differently — writes to both are your responsibility.

Bigtable/HBase vs Cassandra vs DynamoDB

When to use it

  • Time-series and IoT data at very high write rates
  • User activity history, web crawl data, financial ticks — anything read by key ranges
  • Large analytical datasets that also need low-latency key lookups (Bigtable integrates with BigQuery, Dataflow and Spark)

When not to

  • Secondary indexes, ad-hoc queries or joins
  • Multi-row transactions
  • Small data — the minimum cluster is overkill

[!INTERVIEW] Web crawler storage "Crawled pages go into Bigtable keyed by reversed domain plus path — com.example.www/blog/post-1 — so all pages from one site sit next to each other for per-domain scans. A contents column family keeps the last three versions of each page by crawl timestamp, and an anchors family stores inbound link text as columns named by the linking URL." See Design a Web Crawler.