Courses › System Design › Question Breakdowns

Design Pastebin

Metadata vs. blob storage, key generation, expiry and caching immutable content.

Pastebin lets users paste a block of text (code, logs, config) and get back a short link anyone can open. It looks like a URL shortener with a payload attached, and that's how to approach it: the new part is where the content lives, since pastes can be megabytes while their metadata is a few hundred bytes.

Requirements

Functional

  • Users can paste text and get back a short link
  • Anyone holding the link can read the paste, raw or rendered
  • Pastes expire (default 30 days); users can choose another lifetime, including "burn after first read"

Out of scope: user accounts and dashboards, syntax highlighting, editing pastes after creation.

Non-functional

  • Unguessable links: pastes often hold secrets by accident (tokens in logs), so links must not be enumerable
  • Durable until expiry, and actually gone after it
  • Fast reads: a paste opens in under ~200ms, including from CI logs and chat previews
  • Read-heavy: a paste dropped in a team channel gets opened ~20 times

[!TIP] The estimate that matters 300k pastes/day is only ~3.5 writes/second and ~70 reads/second. Sizes are skewed: most pastes are a few KB of code, but log dumps pull the mean to ~25 KB, so storage grows ~7.5 GB/day, ~2.7 TB a year. Traffic is trivial; bytes and retention drive the design, which is what pushes paste bodies out of the database.

Cap paste size (e.g. 2 MB): that covers any reasonable log excerpt and stops the service from becoming free file hosting.

Core Entities

  • Paste metadata: key, content key (pointer to the blob), created at, expires at, owner (optional)
  • Paste content: the text itself, stored as an object

API

POST /v1/pastes
{ "content": "…", "expiresIn": "7d", "burnAfterRead": false }
→ 201 { "url": "https://paste.ly/aZ3kQ9" }

GET /v1/pastes/{key}
→ 200 { "content": "…", "createdAt": "…", "expiresAt": "…" }

DELETE /v1/pastes/{key}
→ 204

Rate-limit creation per account or IP so one script can't fill the storage, and reads per IP so nobody can scan for keys.

High-Level Design

Split storage in two: a metadata database for the small rows you query by key, and object storage (S3) for the paste bodies. Each scales on its own terms, and object storage is far cheaper per GB.

user = client "Client" @0,1
lb = service "Load Balancer" @1,1
app = service "Paste Service" @2,1
kgs = service "Key Generator" @2,0
meta = store "Metadata DB" @3,1
blob = blob "Object Storage\n(paste bodies)" @3,2
cache = cache "Cache" @3,0
user -> lb
lb -> app
app -> kgs : new key
app -> meta
app -> blob
app -> cache

Write: generate a key, upload the content to object storage under that key, insert the metadata row, return the URL. Upload the blob first: a metadata row pointing at a missing blob is a broken paste, while an orphaned blob is just garbage to collect later.

Read: look up the metadata (cache first), check it hasn't expired, then fetch the content. For large pastes, return a short-lived presigned URL or serve through a CDN instead of streaming the bytes through your servers.

Deep Dives

1. Generating keys

Unguessability rules out a counter, so the real choice is between random keys and content hashes:

  • Hash the content: identical pastes collapse to one key, which tells an attacker whether a given secret was ever pasted. Rule it out.
  • Random keys + unique constraint *(recommended)*: draw 8 characters from a cryptographic random source and insert with a unique constraint, retrying on the rare conflict.

Do the collision math instead of assuming it's a problem. 8 base62 characters give 62⁸ ≈ 2.2 × 10¹⁴ keys. Five years at 300k/day is ~550M pastes, so even at the end a new key collides with probability ~550M / 2.2 × 10¹⁴ ≈ 1 in 400,000. A retry loop that almost never runs is simpler than any key-handing service.

Length is also a security setting: with ~10¹⁴ possible keys and a rate limit of 10 guesses/second per IP, scanning for live pastes is hopeless. Shorter keys would make it practical.

2. Expiration and cleanup

  • Store expires_at and check it on every read, so an expired paste is never served even if it hasn't been deleted yet (lazy expiration).
  • Delete for real. Because pastes leak secrets, expiry is a privacy promise: a scheduled job removes expired metadata and blobs, and object-storage lifecycle rules catch anything it misses.
  • Burn after reading needs an atomic claim: UPDATE pastes SET burned = true WHERE key = ? AND NOT burned and serve only if one row changed, otherwise two quick readers could both see it.
  • Purge the CDN copy when a paste is deleted early, or give CDN responses a TTL shorter than the remaining lifetime.

3. Scaling reads

Reads cluster in the first hours after a paste is shared, then die off. Cache metadata and small bodies in Redis for a few hours after creation; a day of new pastes is ~7.5 GB, so even caching all of them is affordable. Put a CDN in front of content: a paste never changes, so it caches perfectly until it expires (minus the early-deletion caveat above).

4. Partitioning the metadata

At ~110M rows a year, one Postgres instance holds years of metadata, so don't shard on day one. When you must, shard by a hash of the key, since every read is a lookup by key. A managed key-value store (DynamoDB, with its built-in TTL handling deletion) is an equally good choice.

What's expected at each level

  • Mid-level: separates metadata from content, generates unique keys, handles expiration.
  • Senior: justifies key length with collision and guessing math, gets the order of writes right, caching and CDN for immutable content.
  • Staff: cost of storage over a decade, abuse prevention (size caps, rate limits, malware scanning), presigned URLs and data-retention policy.