Courses › System Design › Question Breakdowns

Design Smart Grid Device Control

Millions of MQTT devices, unacknowledged commands, closed-loop demand response and owner overrides.

Design a platform that controls millions of home energy devices — smart thermostats, home batteries, EV chargers, water heaters — to help the power grid: when demand is about to exceed supply, the utility asks for a 500 MW reduction in a region within minutes, and the platform sheds load by sending commands to devices. It's an IoT control-plane question about millions of intermittently connected devices, commands that may be delayed, rejected or never acknowledged, closed-loop control toward a target, and respecting the homeowner's final say.

Requirements

Functional

  • Devices connect, report telemetry (power draw, state, battery charge) and receive commands (setpoint change, pause charging, discharge battery)
  • Grid operators create demand-response events: region, target reduction (MW), start, duration
  • The platform selects devices, dispatches commands, tracks the actual reduction and adjusts to hit the target
  • Homeowners can opt out or override any command at any time, and set comfort limits
  • Operators see live progress and after-the-fact reports for settlement

Out of scope: device firmware, energy trading markets, billing customers for incentives (mention it).

Non-functional

  • Scale: ~10M devices; telemetry every ~1 minute (~170k messages/s); events targeting millions of devices
  • Timely: commands reach most online devices in seconds; a full event ramps up within ~5 minutes
  • Unreliable edges: devices go offline, reconnect, reject commands (owner override) or never acknowledge
  • Safety: never violate owner limits or device safety constraints; never create a new spike (e.g. all devices turning back on at once)
  • Highly available during the exact moments the grid is stressed (heat waves, storms)

[!TIP] Command ≠ outcome Sending "reduce" to a million devices doesn't mean the grid sees 500 MW less. Some are offline, some owners override, some were already idle. Treat it as a control loop: dispatch, measure actual reduction from telemetry, and dispatch more (or release some) until you hit the target. That framing separates strong answers from "fan out a message".

Capacity estimates

So 500 MW needs ~250k–500k participating devices after non-response — select more than that.

Core Entities

  • Device — ID, owner, type, region/feeder, capabilities, flexibility (kW available), connection status, last telemetry
  • Owner preferences — enrollment, comfort limits (e.g. max 3°F setback), opt-out windows
  • Grid event — region, target MW, start, duration, status
  • Command — device, event, action, parameters, expiry, status (sent / acked / rejected / expired / overridden)

API

POST /v1/grid-events   { "region": "feeder-114", "targetMw": 500, "start": "2026-07-21T17:00Z", "durationMin": 120 }
GET  /v1/grid-events/{id}   → { "targetMw": 500, "measuredMw": 462, "devicesActive": 391204 }

# device ↔ platform over MQTT (persistent TLS connections)
topic devices/{id}/telemetry   → { "kw": 1.8, "mode": "cool", "setpoint": 72, "t": ... }
topic devices/{id}/commands    ← { "cmdId": "c9", "action": "setback", "deltaF": 3, "until": "...", "expiresAt": "..." }
topic devices/{id}/acks        → { "cmdId": "c9", "result": "applied" | "rejected_by_owner" }

High-Level Design

dev = client "Devices\n(thermostats, batteries, EVs)" @0,1
gw = service "Device Gateway\n(MQTT brokers)" @1,1
tel = queue "Telemetry Stream" @2,2
agg = service "Aggregator\n(kW per region)" @3,2
reg = store "Device Registry +\nPreferences" @2,0
ops = client "Grid Operator" @4,0
ev = service "Event Manager" @3,0
disp = service "Dispatcher\n(control loop)" @3,1
cmd = store "Command Log" @2,1
dev <-> gw
gw -> tel
tel -> agg
ops -> ev : create event
ev -> disp
disp -> reg : select devices
disp -> cmd
disp -> gw : commands
agg -> disp : measured reduction
  • Devices keep persistent MQTT connections to a gateway fleet and stream telemetry into a durable log.
  • Aggregators compute live load and available flexibility per region and feeder.
  • An operator creates an event; the dispatcher selects devices, records commands in a command log, and sends them through the gateways.
  • Devices apply commands (within owner limits) and ack; telemetry shows the actual effect.
  • The dispatcher compares measured vs. target every minute and adds or releases devices until the event ends, then staggers the return to normal.

Deep Dives

1. Connecting millions of devices

  • MQTT over TLS: lightweight, persistent, built for constrained devices and flaky links; supports QoS 1 (at-least-once) delivery and retained messages.
  • A broker cluster of tens of nodes each holding ~100k–500k connections; devices reconnect with jittered backoff, so a broker restart doesn't create a reconnect storm.
  • A connection registry (device → broker) lets the dispatcher route commands; or each broker subscribes to command topics for its devices.

2. Commands that may never be acknowledged

Every command carries a command ID, an expiry and absolute parameters ("setpoint 75°F until 19:00"), not relative ones ("+3°F"), so a duplicate delivery is harmless.

  • Offline devices: queue the command until expiry; when the device reconnects, deliver it only if still valid. Stale commands are worse than none.
  • No ack: after a timeout, mark it unknown — don't assume it worked. Telemetry is the ground truth.
  • Owner override: the device reports overridden; the platform respects it immediately and doesn't retry on that device for the rest of the event.

[!WARNING] The owner has the final word Never design a retry loop that fights the homeowner. An override ends participation for that device; comfort and safety limits are enforced on the device as well as in the platform, so a buggy or compromised server can't push a thermostat to 90°F.

3. Closed-loop dispatch

[Measured reduction from telemetry] -> [Compare with target] -> [Select / release devices] -> [Dispatch commands]
  • Start by dispatching target × an over-subscription factor (e.g. 1.3×), estimated from historical response rates per device type.
  • Rank devices by flexibility and low owner impact (batteries before thermostats; homes not already near their comfort limit); rotate participation so the same homes aren't always picked (fairness and owner retention).
  • Measured reduction is computed against a baseline (what the device would have drawn), not zero.

4. Avoiding new spikes

If 500k air conditioners turn back on at 19:00:00, the "snapback" creates a fresh peak. Stagger event start and end over several minutes with per-device random offsets, and release devices in waves while watching regional load. The same jitter applies to device reconnects and firmware updates.

5. Telemetry pipeline

Telemetry goes into Kafka partitioned by region; stream processors maintain per-region and per-feeder aggregates (current kW, available flexibility) in memory for the dispatcher. Raw data goes to a time-series store with downsampling (1-minute for 30 days, 15-minute for years) for settlement reports and baseline models.

6. Reliability and security

  • Run the dispatcher active-passive per region with the command log as the source of truth; a failover resumes the loop from the log without re-sending everything.
  • Devices fall back to safe local behavior if they lose connectivity during an event (finish the event at its expiry, then return to normal).
  • Strong device identity (per-device certificates), signed commands, least-privilege operator roles and an audit log — this is critical infrastructure.

What's expected at each level

  • Mid-level: persistent device connections, a telemetry pipeline, a service that fans out commands to devices in a region.
  • Senior: MQTT with QoS and offline queuing, idempotent absolute commands with expiry, owner overrides, telemetry as ground truth.
  • Staff: closed-loop control with over-subscription and baselines, staggering to prevent snapback, fairness in device selection, regional failover, and security for critical infrastructure.