Courses › System Design › Key Technologies

Amazon S3

Presigned and multipart uploads, event notifications, storage classes, lifecycle rules and prefix limits.

Amazon S3 is the default object store in system design interviews — and Google Cloud Storage and Azure Blob Storage work almost the same way. The general idea (metadata in the database, bytes in a bucket) is covered in Blob Storage. This page covers the S3-specific features you can name in a deep dive.

The basics

  • Buckets and keys — s3://media-prod/videos/9001/1080p.mp4. Keys are flat strings; the slashes only look like folders.
  • Objects up to 5 TB, written and read whole over HTTP. You can read a byte range with a Range header, but you can't modify part of an object.
  • Durability of eleven nines — data is spread across at least three availability zones.
  • Strong read-after-write consistency for every PUT, DELETE and list since 2020.

Features worth naming

Presigned URLs

Your API signs a time-limited URL that lets the client PUT or GET one specific object directly, so large files never pass through your servers. This is the standard upload and download path.

Multipart upload

Split large files into parts (5 MB – 5 GB each, up to 10,000 parts) and upload them in parallel. A failed part is retried on its own, and the upload can be resumed. Required above 5 GB and recommended above ~100 MB. See Handling Large Blobs.

Event notifications

An object being created or deleted can trigger SQS, SNS, EventBridge or Lambda. That's how "upload finished → start transcoding" works without the client having to call your API.

[Client] -> (S3: presigned multipart PUT) -> {S3 event} -> (SQS) -> [Transcoding workers] -> (S3: outputs)

Storage classes and lifecycle rules

Lifecycle rules move objects to cheaper classes over time ("to IA after 30 days, Glacier after 1 year") and delete expired ones, including incomplete multipart uploads.

Versioning, replication and locking

  • Versioning keeps every version of an object, so you can recover from overwrites and deletes.
  • Cross-region replication copies objects to another region for disaster recovery or lower read latency.
  • Object Lock makes objects write-once-read-many (WORM) for compliance.

Performance

  • Each prefix supports roughly 3,500 writes and 5,500 reads per second, and S3 scales by splitting busy prefixes. Spread very hot workloads across many prefixes, e.g. a hash at the start of the key.
  • Latency per request is tens of milliseconds, so put a CDN in front for user-facing downloads.
  • Use byte-range GETs to download large objects in parallel or to stream video segments.

When not to use it

  • Small, frequently updated records — use a database
  • Querying or searching across objects — keep metadata in a database, or use Athena/S3 Select for occasional analytics over files
  • Low-latency random writes or file-system semantics — use block storage (EBS) or a file system (EFS)

[!INTERVIEW] Photo sharing uploads "The client asks our API for a presigned multipart upload. The API creates a photos row with status pending and returns the URL. The client uploads directly to S3, an S3 event lands on SQS, and a worker generates thumbnails and marks the row ready. Originals move to Infrequent Access after 90 days, and CloudFront serves all reads." See Design Dropbox and Instagram.