Courses › System Design › Question Breakdowns

Design Sora (Text-to-Video)

Long GPU jobs, tiered fair queues, preemption at checkpoints, failure recovery, safety and CDN delivery.

Design a text-to-video service like Sora: users type a prompt, and minutes later get a generated video they can view, remix and share. Each generation occupies expensive GPUs for a long time, demand far exceeds supply, and users on different plans expect different service. It's a question about long-running GPU jobs, queueing and fair scheduling, priority tiers and preemption, failure recovery with checkpoints, safety checks and delivering large outputs.

Requirements

Functional

  • Users submit a prompt (plus options: duration, resolution, aspect ratio, optional image/video input) and get a job ID
  • Users see status and progress (queued, position, generating %, done) and can cancel
  • Finished videos are stored and playable/downloadable; users can browse their library and share videos
  • Plans have quotas (credits) and priorities (free / paid / enterprise)

Out of scope: model training, the diffusion model internals, social feed ranking.

Non-functional

  • Long jobs: a generation takes 1–10 minutes of GPU time, depending on length and resolution
  • Scale: ~1M generations/day at peak demand, against a fixed GPU fleet — the system is almost always capacity-bound
  • Fair: no single user or tenant monopolizes GPUs; paid tiers get faster service
  • Reliable: GPU or worker failures don't silently lose jobs, and users aren't charged for failures
  • Safe: prompts and outputs pass content moderation; outputs carry provenance (C2PA metadata, watermark)

[!TIP] It's a scheduler problem with a model inside The model is a black box that turns (prompt, params) into frames on one or more GPUs. What you design is everything around it: an async job API, a queue with tiers and fairness, GPU workers with checkpoints, and output delivery through a CDN. Separate the cheap, elastic control plane from the scarce GPU data plane early.

Capacity estimates

Demand spikes (a launch, a viral trend) exceed supply by multiples. Queueing is the normal state, so its fairness and transparency matter.

Core Entities

  • Job — user, prompt, params, tier, status, priority, estimated cost, progress, attempts, checkpoint location, output video
  • Video — job, storage key, renditions, thumbnail, safety labels, visibility
  • Worker — GPU node(s), model version loaded, current job, health
  • Quota / credits — user, plan, balance, concurrent-job limit

API

POST /v1/videos/generations
{ "prompt": "a paper boat sailing down a rainy street, cinematic", "seconds": 10, "resolution": "1080p", "aspect": "16:9" }
→ 202 { "jobId": "gen_31", "status": "queued", "position": 128, "etaSec": 240 }

GET    /v1/videos/generations/{jobId}   → { "status": "running", "progress": 0.42 }
DELETE /v1/videos/generations/{jobId}   → cancel (credits refunded if not started)
GET    /v1/videos/{videoId}             → { "url": "https://cdn.../gen_31.mp4", "thumbnail": "..." }

Progress reaches the client via polling, or SSE/WebSocket notifications on status change.

High-Level Design

user = client "App" @0,1
api = service "Generation API" @1,1
mod = service "Prompt Moderation" @1,0
jobs = store "Jobs DB" @2,0
q = queue "Tiered Queues\n(per model / tier)" @2,1
sched = service "GPU Scheduler" @3,1
gpu = service "GPU Workers\n(diffusion + decode)" @4,1
ckpt = blob "Checkpoints" @4,0
post = service "Post-processing\n(safety, encode, watermark)" @4,2
store = blob "Video Storage + CDN" @3,2
user -> api
api -> mod : check prompt
api -> jobs
api -> q
sched -> q
sched -> gpu : assign
gpu -> ckpt : periodic
gpu -> post : raw frames
post -> store
post -> jobs : done
user -> store : stream video
  • Submit: the API authenticates, checks quota, runs prompt moderation, estimates cost, reserves credits and enqueues the job.
  • Schedule: the GPU scheduler picks the next job by tier and fairness and assigns it to a free worker that has the right model loaded and enough GPUs for the requested size.
  • Generate: the worker runs the denoising steps, reporting progress and saving checkpoints periodically.
  • Post-process: decoded frames go through output safety classifiers, are encoded (H.264/AV1, multiple renditions), watermarked with provenance metadata, and uploaded to object storage behind a CDN.
  • Notify: the job is marked done; credits are finalized (or refunded on failure).

Deep Dives

1. Queueing, tiers and fairness

  • Separate queues per tier (enterprise, paid, free) and per model/resolution class (a 4K 20-second job needs a different GPU slice than a 480p 5-second one).
  • The scheduler drains them with weighted fair queuing (e.g. 60/30/10), so free users still progress during sustained load.
  • Per-user concurrency limits (e.g. 2 running jobs on free, 5 on paid) stop one user flooding the queue.
  • Admission control: when the free queue's estimated wait exceeds a threshold, stop accepting free jobs ("high demand, try later") rather than letting the queue grow unbounded.

2. Preemption

When enterprise demand spikes, should a free job 80% done be killed? Preempt only recently started low-priority jobs, and only at a checkpoint, so they resume later without losing work. Jobs near completion run to the end. Preempted jobs keep their place at the front of their tier.

3. Failures and checkpoints

GPUs fail (ECC errors, hung kernels, node loss) — at thousands of GPUs, failures are routine.

  • Workers heartbeat; the scheduler holds a lease per job. A missed lease → the job is re-queued with attempt + 1.
  • Workers checkpoint the latent state every N denoising steps to object storage; a retry resumes from the last checkpoint instead of starting over.
  • After K attempts the job fails with a clear error, and credits are refunded automatically.
  • Bad requests that repeatedly crash workers are quarantined so they can't knock out more GPUs.

[!WARNING] Idempotent completion A worker that was presumed dead may still finish and upload. Use the attempt number as a fencing token: only the current attempt can mark the job done, and the output key includes the attempt, so a zombie can't overwrite the real video.

4. Efficient GPU use

  • Keep models warm: loading video-model weights takes minutes, so workers stay pinned to a model and the scheduler routes jobs to them (see Design ML Model Distribution).
  • Bin-packing by size: multi-GPU jobs (high resolution) need several GPUs on one node with fast interconnect; schedule them so small jobs don't fragment nodes.
  • Batch compatible jobs (same model, resolution, step count) on one worker where the model supports it.
  • Split the pipeline: diffusion on high-end GPUs, decoding/encoding and safety classifiers on cheaper GPUs or CPUs.

5. Safety and provenance

Moderation runs before generation (prompt, input images — cheap, saves GPU time) and after (frame-sampled output classifiers — catches what the prompt didn't reveal). Outputs carry C2PA metadata and a visible or invisible watermark, and blocked generations don't consume credits.

6. Delivering videos

Videos are immutable: store in object storage with content-addressed keys, serve via CDN with long cache lifetimes, and generate a thumbnail and a short preview for library browsing. Larger files use HLS streaming renditions (see Design YouTube / Netflix).

What's expected at each level

  • Mid-level: an async job API with status polling, a queue, GPU workers, output stored in object storage behind a CDN.
  • Senior: tiered queues with weighted fairness and per-user limits, leases and retries, checkpointing, refunds on failure, moderation before and after generation.
  • Staff: preemption policy, admission control under sustained overload, bin-packing multi-GPU jobs, model warmth and placement, fencing against zombie workers, and capacity planning for a permanently supply-constrained fleet.