Courses › System Design › Question Breakdowns

Design Dropbox

Chunking, delta sync, dedup, conflict handling and notifying every device.

Design a file hosting and sync service like Dropbox or Google Drive: users keep a folder on each of their devices, and any change on one device shows up on the others. The interesting parts aren't storing files, they're moving as few bytes as possible (chunking, deltas, dedup) and keeping every device in sync, including devices that were offline.

Requirements

Functional

  • A folder on each device mirrors the cloud copy: a change on one device appears on the others
  • Users can share folders with other people, who see them in their own tree
  • Edits made offline sync when the device reconnects, without losing anyone's work
  • Large files (tens of GB) upload in resumable pieces

Out of scope: real-time collaborative editing (Google Docs style), full version history (we'll mention snapshots as an extension).

Non-functional

  • No data loss, ever: confirmed uploads are durable, and conflicting edits keep both versions
  • Convergence: every device ends up with the same tree
  • Bandwidth-efficient: a one-line edit to a 2 GB file must not re-upload 2 GB
  • Sync latency of a few seconds between online devices

[!TIP] Back-of-the-envelope 100M users storing ~8 GB each is ~800 PB before dedup; identical installers, attachments and shared folders typically cut that by a quarter or more. 40M daily users each change ~50 MB of files, but chunk-level sync uploads only the changed chunks, roughly a tenth: ~200 TB of new chunks a day. With ~2.5 devices per daily user, about 100M clients wait for change notifications, so the notification tier, not storage, sets the server count.

Core Entities

  • File: id, owner, path, size, latest version, list of chunk hashes
  • Chunk: hash (content address), size, storage location
  • Workspace / Folder, Device, User, Share (who can read or write what)

API

POST /v1/files/upload-session { path, size, chunkHashes[] }  → { missingChunks[], uploadUrls[] }
PUT  {uploadUrl}               (chunk bytes, straight to block storage)
POST /v1/files/commit          { path, chunkHashes[], baseVersion } → { version }
GET  /v1/changes?cursor=        (long poll) → { changes[], cursor }
GET  /v1/files/{id}/chunks      → { chunks[]: { hash, downloadUrl } }

The client says which chunks make up the file; the server replies with the ones it doesn't already have. Only those get uploaded.

High-Level Design

c = client "Client\n(desktop / mobile)" @0,1
lb = service "Load Balancer" @1,1
blk = service "Block Servers" @2,0
meta = service "Metadata Servers" @2,1
sync = service "Sync Service" @2,2
s3 = blob "Block Storage\n(chunks)" @3,0
db = store "Metadata DB" @3,1
q = service "Notification\nService" @1,3
c -> lb
lb -> blk, meta
blk -> s3 : chunks
meta -> db
sync -> db
sync -> q : namespace pings
c --> q : long poll
  • Block servers check which chunks are new and hand out signed URLs, so chunk bytes go straight to and from block storage (S3-like).
  • Metadata servers own files, versions, chunk lists and each namespace's change journal.
  • The sync service turns each commit into a small "namespace changed" ping for the devices that mount it.

Deep Dives

1. The client does a lot of the work

The client is a sync engine, not a thin UI. It keeps a local journal (SQLite) recording, for every file, its path, its chunk hashes and the server version it last agreed on. The OS reports file events (inotify, FSEvents, ReadDirectoryChangesW); the client then:

  • Waits for the file to stop changing (an editor may save several times in a second).
  • Re-hashes it in 4 MB chunks (SHA-256) and compares the list with its journal.
  • Uploads only the chunks the server says it's missing, then commits the new chunk list against the version it started from.

Because the journal is local, the client works offline and knows exactly what to send when it reconnects.

a = client "Client A" @0,0
blk = service "Block Servers" @1,0
s3 = blob "Block Storage" @2,0
meta = service "Metadata Servers" @1,1
sync = service "Sync Service" @2,1
b = client "Client B" @3,0.5
cc = client "Client C" @3,1.5
a -> blk : 1. new chunks
blk -> s3
a -> meta : 2. commit
meta -> sync : 3. changed
sync -> b, cc : 4. notify

Clients B and C then fetch the new chunk list and download only the chunks they're missing.

2. Chunking: why 4 MB?

Chunks are the unit of transfer, retry and dedup:

  • A failed upload retries one chunk, not the whole file.
  • An edit re-uploads only the changed chunks.
  • Identical chunks are stored once, even across users.

The size is a trade-off you can put numbers on. A user with 1 TB stored has 250k chunk rows at 4 MB, but 16M at 64 KB, so small chunks multiply metadata and request counts. Large chunks waste bandwidth: changing one byte re-uploads the whole chunk. 4 MB (the block size Dropbox has described publicly) suits typical documents and media. Fixed-size chunks are simple; content-defined chunking (boundaries set by a rolling hash) keeps chunks stable when bytes are inserted near the start of a file.

3. Telling clients about changes

Separate "something changed" from "what changed":

  • What changed lives in a change journal per namespace (a user's root folder or a shared folder): an append-only list of (seq, path, new version). A device stores the last seq it applied as its cursor and asks for everything after it. A laptop that was off for a month replays the journal from its cursor; nothing has to be queued for it while it's away.
  • Something changed is a tiny ping. Desktop clients hold a long poll (or a WebSocket) to a notification service that answers "namespace 42 is past seq 1,180" and nothing more. Phones don't keep connections open: a push notification (APNs/FCM) wakes the app, which then reads the journal.

This keeps the notification tier stateless and cheap: it never holds per-device message queues, only who is listening for which namespaces.

4. Downloading on other devices

When a device sees a new version, it fetches the new chunk list, compares it with the chunks it already has locally, and downloads only the missing ones from block storage using signed URLs. Phones usually skip this step until the user opens the file, to save storage and data.

5. Consistency and conflicts

A commit is a compare-and-set: UPDATE files SET version = N + 1, chunks = … WHERE id = ? AND version = N, plus an append to the namespace's change journal, in one transaction. Keep a namespace's files and journal on the same shard (next section) so that transaction never spans machines.

Commits carry a baseVersion. If someone else committed first (both edited offline), the commit fails and the client keeps both copies ("report (Ada's conflicted copy).docx"). That's what Dropbox does, and it never loses data.

6. Deduplication

Each chunk is addressed by its hash, so the server can tell before upload whether it already has it:

  • Before uploading, the client sends its chunk hashes and uploads only the ones the server doesn't know (the API above). This saves the upload and the storage at once.
  • Shared chunks need reference counts: a chunk can be deleted only when no file version in any namespace still points to it. Garbage-collect in the background, and never delete a chunk the same day its count reaches zero, in case a commit is still in flight.

[!WARNING] Dedup leaks information If "do you already have chunk X?" answers yes instantly, a user can probe whether *anyone* has stored a given file. Scope cross-user dedup carefully, or make the upload behave the same either way.

7. Scaling the metadata

Shard metadata by namespace. Every commit, listing and journal read touches one namespace, so each stays on one shard. A shared folder is its own namespace that several users mount, so sharing never needs a cross-shard join. The cost: moving a file between namespaces is a copy plus a delete, done as two commits.

A namespace with millions of files (a large company's shared drive) can outgrow a shard. Watch for it and move such a namespace to a dedicated shard; hashing users across shards won't help there.

For downloads, a chunk that suddenly gets popular (an installer shared across a company) can be served through a CDN with signed URLs, since chunks are immutable.

8. Sharing and permissions

Store an access list per file/folder (owner, readers, writers) in the metadata DB and check it on every metadata and chunk request. Signed, short-lived download URLs keep block storage private.

What's expected at each level

  • Mid-level: separates file bytes from metadata, uploads/downloads through storage, notifies other devices.
  • Senior: chunking with hashes, delta sync, change journals with cursors, conflict handling, dedup with reference counts.
  • Staff: the client sync engine in depth, offline reconciliation, metadata sharding, dedup privacy, cost at petabyte scale.