Courses › System Design › Question Breakdowns

Design a Remote Devbox

Ephemeral compute with persistent volumes, warm pools and prebuilds, idle stop, tunnels and isolation.

Design a remote development environment service like GitHub Codespaces or an internal "devbox" platform: engineers get an on-demand machine with the repo checked out and tools installed, connect from a browser IDE, VS Code or SSH, and stop and resume it without losing work. It's a question about fast provisioning, separating what must persist from what can be thrown away, idle cost control, isolation and routing connections to the right machine.

Requirements

Functional

  • A developer can create a devbox from a template (image, machine size, repo) and get a ready environment
  • Connect via browser IDE, local IDE (SSH / remote extension) or terminal, and preview web apps on forwarded ports
  • Stop and resume a devbox; their work (files, uncommitted changes) persists
  • Devboxes auto-stop when idle and are deleted after a retention period
  • Admins set templates, quotas, machine types and see audit logs and cost

Out of scope: the IDE itself, building the base images (a CI pipeline does that), source control hosting.

Non-functional

  • Fast: a new devbox ready in < 60s (ideally seconds from a warm pool); resume in < 30s
  • Scale: ~10k engineers (internal) or millions of users (product); ~100k running devboxes at peak
  • Isolation: one user's code and credentials are inaccessible to others; devboxes run arbitrary code
  • Durable work: a host failure must not lose committed or uncommitted files in the workspace
  • Cost: compute is the dominant cost — idle machines must not keep running

[!TIP] Start from what persists Split the devbox into ephemeral compute (the VM/container — replaceable at any time) and a persistent workspace volume (home directory + repo). Once those are separate, stop/resume, host failure, resizing and cost control all become "attach the same volume to a new machine".

Core Entities

  • Template — base image, default machine size, setup scripts, repo
  • Devbox — owner, template, machine size, state, current host, volume ID, last activity
  • Volume — persistent disk / snapshot, size, region
  • Host — capacity, running devboxes, health

API

POST /v1/devboxes   { "template": "web-monorepo", "size": "8cpu-32gb", "branch": "main" }
→ 202 { "devboxId": "db_42", "state": "provisioning" }

GET  /v1/devboxes/{id}               → { "state": "running", "url": "https://db-42.dev.example.com" }
POST /v1/devboxes/{id}/stop
POST /v1/devboxes/{id}/start
DELETE /v1/devboxes/{id}

The devbox state machine: provisioning → running → stopping → stopped → starting → running, plus failed and deleted.

High-Level Design

dev = client "Browser IDE /\nVS Code / SSH" @0,1
gw = service "Connection Gateway\n(auth + routing)" @1,1
api = service "Devbox API\n(control plane)" @1,0
db = store "Devbox State DB" @2,0
sched = service "Scheduler /\nProvisioner" @3,0
pool = service "Warm Pool\n(pre-booted VMs)" @4,0
host = service "Devbox VM\n(agent inside)" @3,1
vol = store "Persistent Volume\n(workspace)" @4,1
img = blob "Image Registry +\nRepo Snapshots" @4,2
dev -> api : create / start / stop
api -> db
api -> sched
sched -> pool : take a VM
sched -> host : attach volume, start
host -> vol
img --> host : image layers
dev -> gw
gw -> host : tunnel
host --> api : heartbeat + activity
  • The control plane (API + state DB + scheduler) records the desired state and drives each devbox through its state machine.
  • Provisioning takes a pre-booted VM from a warm pool, attaches the user's persistent volume (created from a template snapshot), and starts the dev container.
  • An agent in the devbox heartbeats, reports activity and runs setup.
  • A connection gateway authenticates the user and tunnels IDE/SSH/port traffic to the right VM.

Deep Dives

1. Fast provisioning

Cold start is the sum of: get a VM (30–90s), pull a multi-GB image, clone a big repo, install dependencies. Attack each:

Prebuilds are the big win for large monorepos: minutes of setup become seconds.

2. Stop, resume and host failures

  • Stop: flush and detach the volume, release the VM (back to the pool or terminated). Only the volume costs money while stopped.
  • Resume: schedule onto any host with capacity in the volume's zone and re-attach. Memory state (running processes) is lost unless you support VM snapshot/hibernate, which is slower and more expensive — offer it only if needed.
  • Host failure: the volume is network-attached and replicated (EBS/PD-style), so the scheduler restarts the devbox elsewhere. Take periodic volume snapshots so even a volume loss has a recent restore point.

3. Idle detection and cost control

  • The agent reports activity: IDE connections, keystrokes, terminal sessions, running processes the user marked as "keep alive".
  • No activity for e.g. 30 minutes → auto-stop. Stopped for 30 days → snapshot to cold storage, then delete.
  • Per-team quotas (running devboxes, total vCPU) and per-size limits enforced in the control plane before scheduling.

[!WARNING] Don't trust "no SSH connection" as idle A developer may start a 2-hour test run and disconnect. Count running user processes and explicit keep-alives, and warn in the IDE before stopping. Losing someone's long-running job once destroys trust in auto-stop.

4. Isolation and credentials

  • Each devbox is a VM or microVM (not a shared-kernel container) — it runs arbitrary code and often needs Docker inside.
  • Network egress via a controlled NAT; no access to other devboxes or internal production networks unless explicitly granted.
  • Credentials (git, cloud) are short-lived tokens injected by the agent and refreshed, scoped to the user; never baked into images or snapshots.

5. Connecting to the right machine

Devboxes change hosts on every resume, and often sit in private subnets. The gateway terminates TLS for https://db-42.dev.example.com (and port forwards like 3000-db-42…), authenticates the user (SSO), looks up the devbox's current host in the state DB, and proxies through a reverse tunnel the agent opened outbound. Firewalls stay closed, and moving hosts is invisible to the user.

6. Reconciling desired and actual state

Like Kubernetes, the control plane stores desired state ("db_42 should be running") and a reconciler loop compares it with actual state from agent heartbeats and the cloud API, retrying idempotent steps (create VM, attach volume, start container) until they match. Crashed provisioning steps resume instead of leaking VMs; a sweeper deletes orphaned VMs and volumes.

What's expected at each level

  • Mid-level: a control plane that creates VMs/containers from templates, persistent storage for files, a state machine, idle auto-stop.
  • Senior: ephemeral compute + persistent volume, warm pools and prebuilt snapshots, gateway routing through outbound tunnels, short-lived credentials.
  • Staff: reconciliation loops and orphan cleanup, capacity planning per region and size, VM-level isolation trade-offs, hibernation vs. cold resume, cost visibility and quotas for organizations.