Courses › System Design › Question Breakdowns

Design ML Model Distribution

Bandwidth math, peer-to-peer and tree distribution of weights, integrity, safe activation and rollback.

Design the system that ships a new ML model version — hundreds of GBs of weights — to thousands of GPU servers so they can start serving it, quickly and safely. It's an infrastructure question about bandwidth math, avoiding the object-storage bottleneck with peer-to-peer / tree distribution, integrity, and safe activation and rollback.

Requirements

Functional

  • Publish a model version (weights + config + tokenizer) to a registry
  • Distribute it to a target set of GPU workers (e.g. every inference server for that model, in every region)
  • Workers verify the files and activate the new version only when it's complete and healthy
  • Support staged rollout (canary → 10% → 100%) and rollback to the previous version

Out of scope: training, model evaluation, the inference engine itself.

Non-functional

  • Scale: models of 100 GB – 1 TB; 1,000–10,000 workers across several regions
  • Fast: fleet-wide distribution in minutes, not hours; a new or restarted worker cold-starts quickly
  • Correct: a worker never serves partially downloaded or corrupted weights
  • Efficient: don't saturate object storage, cross-region links or the serving network
  • Resilient: worker churn and slow nodes don't stall the rollout

[!TIP] Do the bandwidth math first 500 GB to 1,000 workers is 500 TB of transfer. From one storage service capped at, say, 100 Gbps (~12.5 GB/s), that's ~11 hours. But each worker has its own ~25–100 Gbps NIC: if workers forward data to each other, the fleet's total upload capacity grows with the number of workers, and the time is close to one worker's download time (500 GB at ~3 GB/s ≈ 3 minutes). That observation is the whole design.

Capacity estimates

Core Entities

  • Model version — name, version, manifest (list of files → chunks → SHA-256), size, status
  • Chunk — hash, size, storage location
  • Rollout — model version, target fleet, stage, percentage, health gates
  • Worker — model versions present on local disk, active version, health

API

POST /v1/models/{name}/versions          (register a manifest after upload)
POST /v1/rollouts   { "model": "chat-l", "version": "2026-10-10", "target": "serving/us-*", "stages": [1, 10, 100] }
GET  /v1/rollouts/{id}   → { "stage": 2, "workersReady": 812, "workersFailed": 3 }
POST /v1/rollouts/{id}/rollback

# worker agent ↔ tracker
GET  /v1/manifests/{version}
GET  /v1/peers?version=...&chunk=...    → peers that have this chunk

High-Level Design

reg = blob "Model Registry\n(object storage)" @0,1
ctl = service "Rollout Controller" @1,0
trk = service "Tracker\n(who has which chunk)" @2,0
seed = service "Regional Seed /\nCache Nodes" @1,1
w1 = service "GPU Worker\n(agent)" @2,1
w2 = service "GPU Worker\n(agent)" @3,1
w3 = service "GPU Worker\n(agent)" @2.5,2
ctl -> trk : version, target set
reg -> seed : once per region
seed -> w1
seed -> w2
w1 <-> w2 : chunks
w1 -> w3
w2 -> w3
w1 --> trk : have / need
w2 --> trk
w3 --> trk
  • Training uploads the weights to the registry as content-addressed chunks (~16 MB) plus a manifest of their hashes.
  • The rollout controller decides which workers should fetch which version, stage by stage.
  • Each region pulls the chunks once into seed/cache nodes; workers then fetch chunks from seeds and from each other, coordinated by a tracker that knows which worker has which chunk.
  • A worker's agent verifies every chunk hash, assembles the files on local NVMe, loads the model into GPU memory, runs a health check, and only then reports ready for traffic.

Deep Dives

1. Distribution strategies

Chunking + pipelining is what makes trees and swarms fast: a worker starts forwarding chunk 1 while it's still downloading chunk 2, so the depth of the tree adds only a few chunk-times, not whole-model times.

2. The swarm in practice

  • The tracker tells each worker which peers hold which chunks; workers fetch rarest chunks first so every chunk is replicated quickly.
  • Topology awareness: prefer peers in the same rack, then the same zone; send one copy across a region boundary.
  • Stragglers: hedge the last few chunks by requesting them from multiple peers and taking the first.
  • Cap per-worker upload bandwidth so distribution doesn't starve live inference traffic on serving nodes.

[!NOTE] GPU-to-GPU transfer Inside a cluster with fast interconnects (InfiniBand/RDMA, NVLink), a worker that already has the model in GPU memory can broadcast it to new workers directly (NCCL broadcast, RDMA reads), skipping disk entirely. That's how some platforms cut cold starts for autoscaled replicas from minutes to seconds.

3. Integrity

  • The manifest lists each chunk's SHA-256, and the manifest itself is signed. Every chunk is verified on receipt; a bad chunk is discarded and refetched from someone else, and the peer that sent it is reported.
  • Chunks are content-addressed, so versions that share files (same tokenizer, unchanged layers) share chunks and transfer nothing new.

4. Safe activation

  • Download into a new directory; never overwrite the active version (keeps instant rollback possible).
  • After assembly: verify all hashes, load into GPU memory alongside or after draining the old version, run a smoke test (fixed prompts with expected outputs), then flip the active pointer atomically and announce readiness to the load balancer.
  • Keep the previous version on disk for rollback.

5. Staged rollout and rollback

The controller moves through stages (canary 1% → 10% → 100%) gated on health metrics (error rate, latency, quality evals). Rollback is just "activate previous version" — fast because its weights are still on local disk. Distribution can run ahead of activation: pre-stage the new weights to the whole fleet while only the canary serves it.

6. Worker churn and cold starts

New workers (autoscaling, replacements) join the swarm, fetch from peers that already have the model, and pin it in a local cache. With a warm local cache, restarts load from NVMe in seconds; without, peers serve the bytes at LAN speed rather than hitting object storage.

What's expected at each level

  • Mid-level: chunked downloads from object storage with checksums, a regional cache, a rollout controller.
  • Senior: bandwidth math showing why central fan-out fails, tree or P2P distribution with pipelining, signed manifests, atomic activation and rollback.
  • Staff: topology-aware swarms, straggler hedging, protecting serving traffic, GPU-to-GPU transfers for cold starts, content-addressed dedup across versions, and decoupling pre-staging from activation.