Courses › System Design › Patterns

Handling Large Blobs

Presigned URLs, multipart uploads, CDNs and keeping blobs out of your DB.

Photos, videos, documents and backups don't belong in your database or flowing through your API servers. Large binary objects ("blobs") need their own path through the system.

The rule: metadata in the database, bytes in blob storage

Store files in object storage (Amazon S3, Google Cloud Storage, Azure Blob) and keep a metadata row in your database: owner, size, content type, storage key, status.

Uploading with presigned URLs

Routing a 2 GB upload through your API servers ties up connections and bandwidth for minutes. Instead, let the client upload directly to blob storage:

Client -> [API Server] -> (Metadata DB)
Client -> (Blob Storage)
(Blob Storage) -> {Upload Event} -> [Processing Workers]
  • The client asks the API for an upload URL. The API creates a metadata row (status = 'pending') and returns a presigned URL — a time-limited, signed URL granting permission to upload one object.
  • The client uploads the file directly to blob storage.
  • Blob storage emits an event when the upload completes (or the client confirms); the system marks the file uploaded and starts any processing.

Large files: multipart and resumable uploads

For files over ~100 MB:

  • Split the file into chunks (e.g. 5–100 MB) and upload them in parallel with multipart upload.
  • A failed chunk is retried on its own; an interrupted upload resumes from the last completed chunk.
  • Hashing chunks enables deduplication and delta sync — only changed chunks are re-uploaded (this is how Dropbox-style sync works).

Downloading

  • Serve files through a CDN so popular content is cached near users.
  • Protect private files with signed URLs (or signed cookies) that expire.
  • Support range requests so video players and download managers can fetch portions.

Processing pipelines

Uploaded files often need work: thumbnails, transcoding, virus scanning, text extraction. Trigger an asynchronous pipeline from the upload event (see Managing Long-Running Tasks), and track status in the metadata row so clients can show progress.

Things to mention

  • Consistency between metadata and blob — handle orphaned uploads (metadata exists, upload never finished) with a cleanup job based on status and age.
  • Security — validate content type and size; scan for malware; never trust the client's claimed type.
  • Cost — move rarely accessed files to cheaper storage tiers with lifecycle rules.

[!INTERVIEW] Say it early As soon as a design involves user files, say: "Files go straight to S3 with presigned URLs; the database only stores metadata." It removes a whole class of follow-up questions.