Courses › System Design › Question Breakdowns

Design GitHub

Replicated Git repositories on file servers, atomic ref updates, fork networks, clone storms and pull requests.

Design a code hosting platform like GitHub: users push and pull Git repositories, browse code in the web UI, open pull requests with reviews and checks, and fork repositories — for ~100M+ repositories, some of them gigantic. It's a question about storing Git repositories on a fleet of file servers, replication of a filesystem-based data structure, routing every operation to the right replica, and the web/metadata side (pull requests, permissions, webhooks) built around it.

Requirements

Functional

  • Users can create repositories and push / pull / clone over HTTPS and SSH
  • Users can browse files, commits, blame and diffs in the web UI
  • Users can open pull requests, review and comment, and merge (with required checks)
  • Users can fork a repository
  • Pushes trigger webhooks and CI (see Design a CI/CD System and Webhook Delivery)

Out of scope: issues, Actions runners, packages, code search (mention it), Copilot.

Non-functional

  • Durable: pushed commits are never lost — replicated synchronously
  • Consistent: after a push succeeds, every subsequent read sees it (refs must never go backwards)
  • Scale: ~400M repositories, ~100M developers; tens of thousands of Git operations/second; clones of large repos are heavy CPU work
  • Fast web reads: file views and diffs in < 500ms
  • Highly available: a file server failure doesn't block pushes or reads

[!TIP] A repository is a directory, not rows Git stores a repository as files (objects, packfiles, refs) that the Git tools operate on directly. You can't put it in a relational database. The design is about placing repositories on file servers, replicating them, and routing each Git operation to a server that holds the repo — the approach GitHub's Spokes system takes.

Capacity estimates

Core Entities

  • Repository — owner, name, visibility, fork network ID, replica locations, checksum
  • Ref — branch/tag name → commit SHA (in the repo itself)
  • Pull request — repo, head and base refs, author, state, reviews, checks, merge commit
  • Permission — user/team, repo, role

API

# Git protocol (smart HTTP / SSH)
GET  /{owner}/{repo}.git/info/refs?service=git-upload-pack
POST /{owner}/{repo}.git/git-receive-pack            (push)

# REST
POST /v1/repos/{owner}/{repo}/pulls   { "head": "feature", "base": "main", "title": "..." }
PUT  /v1/repos/{owner}/{repo}/pulls/{n}/merge   { "method": "squash" }
GET  /v1/repos/{owner}/{repo}/contents/{path}?ref=main

High-Level Design

dev = client "Git Clients\n(HTTPS / SSH)" @0,1
web = client "Web UI / API" @0,0
front = service "Git Frontends\n(auth, routing)" @1,1
app = service "Web App / API" @1,0
route = store "Repo Routing\n(repo → replicas)" @2,0
fs1 = store "File Server A\n(replica)" @3,0
fs2 = store "File Server B\n(replica)" @3,1
fs3 = store "File Server C\n(replica)" @3,2
meta = store "Metadata DB\n(users, PRs, perms)" @1,2
hooks = queue "Push Events" @2,2
dev -> front
web -> app
front -> route
app -> route
front -> fs1 : reads go to one
front -> fs2 : writes go to all
front -> fs3
app -> meta
fs1 --> hooks : post-receive
  • Git frontends authenticate the user, check permissions, look up the repository's replicas, and proxy the Git protocol to file servers.
  • Reads (fetch, clone) go to one healthy replica; writes (push) go to all replicas through a coordinated, transactional ref update.
  • The web app reads repository content (files, diffs, blame) through an RPC service on the file servers and keeps users, PRs and permissions in a metadata database.
  • A successful push emits events for webhooks, CI and PR updates.

Deep Dives

1. Replicating Git repositories

Each repository lives on 3 file servers in different racks. A push must be atomic across them:

  • The frontend streams the pushed objects to all replicas.
  • Each replica checks the push applies (the old ref values match — no concurrent push won).
  • A three-phase ref update: replicas vote, then commit the new ref values together. If a replica fails mid-push, the push still succeeds with a majority, and the lagging replica is marked out of date until repaired.
  • A checksum of all refs per replica lets the system detect divergence cheaply and repair from a healthy copy.

Objects are content-addressed and immutable, so copying them is safe and idempotent; only refs need careful coordination.

2. Routing and failover

The routing table maps repo → [replicas, health, checksum]. Reads pick a nearby, healthy, up-to-date replica; when a server dies, reads move to other replicas immediately, and a background job creates a new third replica elsewhere. Spreading a server's repositories across many different peers means re-replication after a failure is fast and parallel.

3. Forks and storage deduplication

A popular repo can have 100k forks that share almost all objects. Store a fork network together: forks share one object store (Git alternates), with per-fork refs. Fork creation becomes nearly free, and storage isn't multiplied. All forks of a network are placed on the same replica set.

4. Big repos and clone storms

Generating a packfile for a large clone is CPU-heavy. Cache packfiles for common requests (full clone of the default branch), use bitmap indexes to compute reachable objects quickly, and spread clone load across replicas. A CI fleet cloning the same monorepo thousands of times per hour can be served from the cache.

[!WARNING] Refs must never go backwards If a read is served by a replica that missed the latest push, a user may fetch an older main and their next push could appear to rewrite history. Only route reads to replicas whose ref checksum is current, and mark lagging replicas out of rotation until repaired.

5. Pull requests and merges

PR metadata (title, reviews, comments, required checks) lives in the relational database. The diff and mergeability are computed from the repository by Git on the file servers (cached by base SHA, head SHA). Merging creates a merge commit through the same replicated push path, after verifying branch protection rules (approvals, passing checks) in the metadata DB.

6. Web reads at scale

File views, blame and diffs are computed from Git objects — immutable, so cache aggressively by (repo, commit SHA, path). Rendered views for popular repos can be served from a cache layer for a long time because a commit's content never changes.

What's expected at each level

  • Mid-level: repos stored on servers, Git operations through a frontend with auth, metadata in a database, pull requests as records.
  • Senior: replicated repositories with a routing table, atomic multi-replica ref updates, reads from one replica and writes to all, caching by immutable SHAs.
  • Staff: checksums for divergence detection and repair, fork networks with shared object stores, packfile and bitmap caching for clone storms, and the consistency rules that keep refs from going backwards.