Courses › System Design › Question Breakdowns

Design a CI/CD System

Webhooks to DAGs, pull-based runners with leases, stuck jobs, isolation, caches and tenant fairness.

Design a CI/CD system like GitHub Actions or GitLab CI: a push to a repository triggers a pipeline of jobs (build, test, deploy) defined in a config file, run on a fleet of isolated runners, with live logs and results. It's a question about turning events into a DAG of jobs, scheduling onto workers, isolation, stuck jobs and multi-tenant fairness.

Requirements

Functional

  • A push or pull request triggers the pipeline defined in the repo's config (.ci.yml)
  • A pipeline is a DAG of jobs (build → test in parallel → deploy); a job runs a list of steps in a container
  • Developers see job status and live logs, and can cancel or re-run a job
  • Jobs can store artifacts and cache dependencies between runs
  • Deploy jobs can require manual approval and use secrets

Out of scope: the source-control system itself, building a container registry, the deployment targets (we call their APIs).

Non-functional

  • Scale: ~100k pipelines/hour at peak, ~1M jobs/hour, tens of thousands of concurrent jobs
  • Fast start: a job starts within ~10 seconds of becoming runnable (when capacity exists)
  • Isolation: one tenant's job can't read another's code, secrets or caches
  • Reliable: no job silently lost; stuck jobs are detected and recovered
  • Fair: a monorepo kicking off 5,000 jobs can't starve everyone else

[!TIP] Two layers: control plane and data plane The control plane (webhooks, pipeline state, scheduling) is cheap, stateful and must be consistent. The data plane (runners executing jobs) is expensive, elastic and untrusted. Draw the line early; most deep dives sit on one side of it.

Core Entities

  • Pipeline run — repo, commit SHA, trigger, status
  • Job — pipeline run, name, dependencies, runner labels, status, attempt, assigned runner, lease
  • Step log — job, chunked log lines (in object storage)
  • Artifact / cache entry — key, job, blob location, size, expiry
  • Runner — labels (os, arch, gpu), status, current job, last heartbeat

API

POST /webhooks/scm           (push / pull_request events from GitHub, GitLab …)
GET  /v1/pipelines/{id}       → { status, jobs: [{ id, name, status, needs: [...] }] }
GET  /v1/jobs/{id}/logs?follow=true   → text/event-stream
POST /v1/jobs/{id}/cancel
POST /v1/jobs/{id}/rerun

# runner ↔ control plane
POST /v1/runners/poll    { labels } → { job } | 204 (long poll)
POST /v1/jobs/{id}/heartbeat
POST /v1/jobs/{id}/complete { status, artifacts }

High-Level Design

scm = client "Git Host" @0,0
hook = service "Webhook Service" @1,0
pipe = service "Pipeline Service\n(parse DAG)" @2,0
db = store "Pipelines & Jobs DB" @2,1
sched = service "Scheduler" @3,1
q = queue "Job Queues\n(per label / tenant)" @3,0
run = service "Runners\n(ephemeral VMs)" @4,0
logs = blob "Logs, Artifacts,\nCache (S3)" @4,1
ui = client "Developer UI" @1,2
api = service "API / Log Stream" @2,2
scm -> hook : push event
hook -> pipe
pipe -> db : jobs (pending)
sched -> db : ready jobs
sched -> q
run -> q : long-poll
run -> logs
run -> db : status, heartbeat
ui -> api
api -> db
api -> logs
  • Trigger: the webhook service verifies the event signature, dedups by delivery ID and enqueues it.
  • Plan: the pipeline service fetches the config at that commit, expands it into jobs and dependency edges, and stores them as pending.
  • Schedule: jobs whose dependencies have all succeeded become queued and go to a queue matching their runner labels.
  • Run: a runner long-polls, claims a job, starts a fresh container/VM, runs the steps, streams logs, uploads artifacts and reports the result.
  • Advance: each completion triggers the scheduler to re-check the DAG and release newly unblocked jobs.

Deep Dives

1. Executing the DAG

Store the DAG explicitly: jobs plus job_dependencies(job_id, needs_job_id). Keep a counter of unfinished dependencies per job; when a job succeeds, decrement its dependents' counters in the same transaction, and any that reach zero become queued. If a job fails, mark downstream jobs skipped (unless they run if: always()).

This event-driven approach avoids polling every pipeline, and the transaction keeps two parallel completions from both releasing (or both missing) a job.

2. Dispatching to runners: push or pull?

Pull is the common answer. A runner's poll claims a job with a conditional update (status queued → running, runner = me, lease_until = now()+60s), so two runners can't take the same job.

3. Stuck jobs

Jobs get stuck in three ways, each with its own detector:

  • Runner died: it stops heartbeating. A reaper finds running jobs with an expired lease and requeues them (attempt + 1) or fails them after N attempts. Fencing by attempt number stops a zombie runner from reporting a stale result.
  • Job hangs: the runner is alive but the step never finishes. Enforce a per-job timeout (default e.g. 6 hours, configurable) enforced by the runner *and* checked by the reaper.
  • Never scheduled: no runner has the requested labels (gpu: a100). Surface "waiting for a runner" in the UI and alert after a threshold instead of waiting silently.

4. Isolation and secrets

Jobs run untrusted code (any pull request can change the build script). Each job gets a fresh, ephemeral environment — a microVM (Firecracker) or a VM per job, destroyed afterwards — not a shared long-lived container.

  • Secrets are injected at runtime from a vault, scoped to the repo/environment, masked in logs, and not exposed to pull requests from forks.
  • Deploy jobs get short-lived cloud credentials via OIDC tokens instead of stored long-lived keys.

5. Logs, artifacts and caches

  • Live logs: the runner streams log chunks to a log service that appends them to a buffer (Redis stream) for live viewers and flushes chunks to object storage for the archive. The UI tails via SSE.
  • Artifacts: uploaded to S3 with a presigned URL; metadata in the DB; expire after N days.
  • Dependency cache: keyed by a hash of the lockfile (npm-${hash(package-lock.json)}); restored at job start, saved at the end. Scoped per repo and branch so a malicious PR can't poison main's cache.

6. Scaling runners and fairness

  • Autoscale the runner pool on queue depth per label; keep a small warm pool of booted VMs so start time stays under ~10s.
  • Per-tenant concurrency limits (e.g. 500 concurrent jobs on the paid tier) and fair queuing across tenants, so a monorepo spike waits its turn.
  • Cancel superseded runs: a new push to the same branch cancels the old pipeline's queued jobs — a big saving in practice.

What's expected at each level

  • Mid-level: webhook → pipeline → jobs → workers → status, with logs stored somewhere sensible.
  • Senior: explicit DAG execution with dependency counters, pull-based runners with leases and heartbeats, stuck-job handling, ephemeral isolated environments.
  • Staff: multi-tenant fairness and quotas, secret scoping and fork security, cache poisoning, warm pools vs. cost, and control-plane/data-plane separation for reliability.