Courses › System Design › Question Breakdowns

Design OTA Firmware Updates

A/B partitions and bootloader rollback, code signing, staged rollouts with automatic halts, delta downloads.

Design an over-the-air (OTA) update system that ships new firmware and software to millions of devices — cars, phones, routers, IoT sensors — safely. A bad update can brick devices that nobody can physically reach, so this is a question about staged rollouts with automatic halts, atomic installs with rollback (A/B partitions), code signing, delta updates over constrained networks, and devices that lose power mid-install.

Requirements

Functional

  • Engineers publish a signed release (firmware image + metadata) for a device model/hardware revision
  • Releases roll out in stages (e.g. 1% → 10% → 50% → 100%) to targeted groups (model, region, current version)
  • Devices check for, download, verify and install updates, then report the result
  • Failed installs roll back automatically; operators can pause or roll back a rollout

Out of scope: building the firmware, app-store updates, per-device feature flags (mention them).

Non-functional

  • Never brick a device: a failed or interrupted update must leave the device working
  • Security: devices only install images signed by the vendor; keys are protected
  • Scale: ~10M devices; a release of ~500 MB (full) or ~50 MB (delta)
  • Constrained devices: flaky and metered networks, limited storage and battery, devices offline for weeks
  • Observability: rollout health visible within minutes, with automatic halt on regressions

[!TIP] Safety first, scale second Distributing 50 MB to 10M devices is a CDN problem you can solve in a sentence. The interview is about what can go wrong on the device — power loss mid-write, a boot loop, a malicious image — and how the rollout notices and stops. Lead with A/B partitions, signatures and staged rollout.

Capacity estimates

Core Entities

  • Release — version, target hardware, image (or deltas from known versions), signature, notes
  • Rollout — release, target cohort, stages, current percentage, health thresholds, status (running / paused / halted / done)
  • Device — ID, model, hardware revision, current version, region, last check-in, update state
  • Install report — device, release, outcome (success / download failed / verify failed / install failed / rolled back), logs

API

# device → server
GET  /v1/devices/{id}/update?current=4.2.1&hw=rev-c   → 204 | { "release": "4.3.0", "url": "https://cdn.../4.2.1-4.3.0.delta", "sha256": "...", "signature": "..." }
POST /v1/devices/{id}/reports   { "release": "4.3.0", "outcome": "success", "bootTimeMs": 8120 }

# operator
POST /v1/rollouts   { "release": "4.3.0", "cohort": { "model": "x2", "region": ["us"] }, "stages": [1, 10, 50, 100] }
POST /v1/rollouts/{id}/pause

High-Level Design

eng = client "Release Engineers" @0,0
sign = service "Signing Service\n(HSM keys)" @1,0
rel = blob "Release Artifacts\n(full + deltas)" @2,0
cdn = cache "CDN" @3,0
roll = service "Rollout Service" @1,1
db = store "Devices & Rollouts DB" @2,1
dev = client "Devices\n(A/B partitions)" @3,1
health = service "Health Monitor\n(auto-halt)" @2,2
eng -> sign : build
sign -> rel
rel -> cdn
eng -> roll : start rollout
roll -> db
dev -> roll : check in
dev -> cdn : download
dev -> roll : report
roll -> health
health -> roll : halt
  • Publish: a build is signed by the signing service (keys in an HSM), and full images plus deltas from common prior versions are stored and pushed to a CDN.
  • Roll out: the rollout service decides, per device check-in, whether this device is in the current stage's cohort and returns the right artifact.
  • Install: the device downloads (resumably), verifies the signature and hash, writes the image to its inactive partition, and reboots into it.
  • Report: the device reports success or failure; the health monitor compares cohorts and halts the rollout on regressions.

Deep Dives

1. A/B partitions: never brick

The device has two system partitions. It runs from A and writes the update to B while running normally.

[Download + verify] -> [Write to inactive slot B] -> [Mark B "try once"] -> [Reboot into B] -> [Health check passes] -> [Mark B good]
[Reboot into B] -> [Boot fails or health check fails] -> [Bootloader falls back to A]
  • Power loss during the write? A is untouched; the device just retries.
  • The bootloader tries B a limited number of times; if B never marks itself healthy (boots, connects, passes self-tests), it falls back to A automatically.
  • Devices without room for two copies use a small recovery partition that can always re-download and reflash.

2. Signing and verification

  • Images are signed with a private key held in an HSM; devices hold the public key in secure boot storage and verify before installing (and the bootloader verifies before booting — secure boot chain).
  • Include the target hardware and version in the signed metadata to prevent installing the wrong image or downgrading to a vulnerable version (anti-rollback counters in hardware).
  • Frameworks like The Update Framework (TUF)/Uptane (used in cars) separate roles and keys so one compromised server or key can't push malicious firmware.

3. Staged rollout and automatic halt

  • Assign devices to a stable bucket (hash(deviceId) % 100); stage 1% = buckets 0, stage 10% = buckets 0–9, and so on. Cohort filters (model, region, version) narrow it further.
  • After each stage, wait a soak period, then compare the cohort's metrics with devices still on the old version: install failure rate, rollbacks, crash rate, boot time, battery, connectivity.
  • If thresholds are breached, halt automatically and alert. Rolling forward with a fix is often safer than pushing a downgrade.

[!WARNING] A halt must stop new installs everywhere at once Devices that already received an "update available" response may install hours later. Give every update offer an expiry and re-check the rollout status right before installing, so a halted release stops even on devices mid-flow.

4. Delivery over bad networks

  • Delta updates (binary diffs from a specific source version) cut size ~10×; fall back to the full image if the device's version has no delta.
  • Resumable, chunked downloads with per-chunk hashes; downloads only on Wi-Fi/charging for phones, or within a car's allowed data budget.
  • The CDN carries the bytes; the rollout pace (not CDN capacity) limits bandwidth.

5. Check-ins at scale

10M devices checking in every few hours is ~1k/s — modest, but synchronized check-ins (all devices after a regional power outage) spike. Add jitter to check-in intervals, and for urgent security fixes, send a push (MQTT/push notification) telling devices to check in now rather than polling faster.

6. Fleet visibility

Store each device's current version and last report; dashboards show version distribution and rollout progress per cohort. Devices offline for weeks get the latest release (or a required stepping-stone version) when they return.

What's expected at each level

  • Mid-level: a server that tells devices which version to install, downloads from a CDN, devices reporting status.
  • Senior: A/B partitions with bootloader fallback, signature verification, staged rollout by device buckets, delta and resumable downloads.
  • Staff: automatic halt from cohort comparisons, offer expiry and re-checks, anti-rollback and key compromise resilience (TUF/Uptane), push-triggered urgent updates, and recovery paths for devices that can't store two images.