Courses › System Design › Question Breakdowns

Design a Batch Inference System

Chunked batches on spare GPU capacity, continuous batching, exactly one result per request, deadlines.

Design a batch inference service like OpenAI's or Anthropic's Batch API: customers upload a file of thousands to millions of model requests, and the system runs them on a GPU fleet within a deadline (e.g. 24 hours) at a discount, returning a results file. It's a question about packing work onto scarce GPUs, batching requests efficiently, routing every result back to the right request, failures mid-batch, and sharing capacity with real-time traffic.

Requirements

Functional

  • Customers submit a batch: a file (JSONL) of requests, each with a custom_id, model and parameters
  • Customers can check status (validating, in progress, completed, failed, cancelled) with progress counts, and cancel
  • When done (or at the deadline), customers download a results file mapping each custom_id to its output or error
  • Batches complete within a completion window (24h)

Out of scope: the model serving engine internals (assume servers accept a batch of prompts and return completions), fine-tuning, the real-time API itself.

Non-functional

  • Throughput and cost over latency: maximize tokens per GPU-hour; individual requests can wait hours
  • Scale: ~10k batches/day, up to ~1M requests per batch, ~1B requests/day
  • Every request accounted for: each request ends with exactly one result (success or error) — none lost, none duplicated in the output
  • Fairness: one huge batch can't starve other customers
  • Don't hurt real-time traffic sharing the same GPUs

[!TIP] The interviewer's core scenario A common framing: "one GPU accepts a batch of at most N inputs at a time; design intake, batching, and routing results back to the right user." Start there — queue → batcher → GPU → result router — and then scale out to many GPUs, failures and fairness.

Capacity estimates

Core Entities

  • Batch — customer, model, input file, status, counts (total, completed, failed), deadline
  • Request — batch, custom_id, payload, status, attempts, result location
  • Work chunk — a slice of a batch's requests (e.g. 1,000) — the unit of scheduling
  • GPU worker — model loaded, capacity, health

API

POST /v1/files   (multipart JSONL upload) → { "fileId": "file_1" }
POST /v1/batches   { "inputFileId": "file_1", "endpoint": "/v1/messages", "completionWindow": "24h" }
→ 201 { "batchId": "batch_8", "status": "validating" }

GET  /v1/batches/{batchId}   → { "status": "in_progress", "counts": { "total": 500000, "completed": 120334, "failed": 12 } }
POST /v1/batches/{batchId}/cancel
GET  /v1/files/{outputFileId}/content   → JSONL of { custom_id, response | error }

High-Level Design

user = client "Customer" @0,0
api = service "Batch API" @1,0
s3 = blob "Input / Output Files\n(object storage)" @1,1
meta = store "Batch & Chunk\nMetadata DB" @2,0
split = service "Validator /\nChunker" @2,1
sched = service "Scheduler\n(fair queue per model)" @3,0
gpu = service "GPU Workers\n(continuous batching)" @4,0
rt = client "Real-time Traffic" @4,1
agg = service "Result Writer" @3,1
user -> api : upload, create batch
api -> s3
api -> meta
split -> s3 : read input
split -> meta : chunks
sched -> meta : pending chunks
sched -> gpu : assign chunk
rt -> gpu : priority
gpu -> agg : results by custom_id
agg -> s3 : partial outputs
agg -> meta : progress
  • Intake: the input file goes to object storage; the batch record starts as validating.
  • Validate and chunk: a validator checks every line (schema, model, token limits), writes invalid lines straight to the error output, and splits the rest into chunks of ~1,000 requests.
  • Schedule: the scheduler assigns chunks to GPU workers serving the right model, using spare capacity not needed by real-time traffic.
  • Run: each worker feeds the chunk's requests into its inference engine, which continuously batches them (up to N concurrent sequences on the GPU).
  • Collect: results, tagged with custom_id, are written per chunk to object storage; when every chunk is done, the result writer concatenates them into the output file and marks the batch complete.

Deep Dives

1. Batching on the GPU

GPUs are efficient only when many sequences run together. Static batching (wait for N requests, run them, wait for the slowest to finish) wastes the GPU on short requests waiting for long ones. Continuous batching (vLLM-style) adds new sequences as soon as others finish, keeping the batch full.

For offline work you also control what goes into a batch: group requests with similar prompt lengths and shared prefixes (the same long system prompt) so the KV cache can be reused — a big saving for batch jobs where thousands of requests share instructions.

2. Routing results back

Every request carries (batchId, chunkId, custom_id) through the pipeline; the worker returns results tagged with them. Results are written per chunk (batch_8/chunk_0042.jsonl) so a chunk is the unit of completion: the chunk is marked done only after its output file is durably written.

3. Failures mid-batch

  • Worker/GPU dies mid-chunk: chunks are assigned with a lease; if the worker stops heartbeating, the lease expires and the chunk goes back to pending. Requests already completed can be kept if partial results were checkpointed, or the whole chunk re-runs.
  • Duplicates: a re-run chunk may produce results for requests that already succeeded. Since output is written per chunk and committed atomically (write a new file, then flip the chunk's pointer), the final file contains exactly one result per custom_id.
  • Bad requests: a request that crashes the engine (e.g. pathological input) is retried a few times alone, then recorded as an error, so it can't poison its chunk forever.

[!WARNING] Exactly one result per request Customers rely on matching custom_ids. Never append results from a chunk attempt to the output directly; commit chunk outputs atomically and assemble the final file only from committed chunks.

4. Scheduling: deadlines, fairness and real-time traffic

  • Real-time first: the online API has priority on shared GPUs. Batch work fills the troughs (nights, weekends, between spikes) — that's why it's discounted. Workers can preempt batch chunks when real-time load rises, returning them to the queue.
  • Deadlines: order pending chunks by slack (time until deadline minus estimated remaining work). Batches at risk of missing the 24h window get priority and can trigger dedicated capacity.
  • Fairness: a per-customer weighted fair queue, so a 1M-request batch doesn't block a 1k-request batch submitted later; customers also have enqueued-token quotas.

5. Model placement

Each worker has one model loaded; loading another takes minutes (see Design ML Model Distribution). The scheduler prefers workers that already have the model, and only re-assigns GPUs between models when a model's backlog justifies the switch cost.

6. Progress, cancellation and expiry

  • Progress counters are updated per chunk (not per request) to keep metadata writes low.
  • Cancel: mark the batch cancelling; the scheduler stops dispatching its chunks, in-flight chunks finish or are abandoned, and completed results are still returned.
  • Expiry: at the deadline, unfinished requests are returned as expired and are not billed.

What's expected at each level

  • Mid-level: file upload → queue → GPU workers → results file, with status tracking and custom_id mapping.
  • Senior: chunking as the unit of work, continuous batching, leases and retries, atomic chunk outputs, per-customer fairness.
  • Staff: co-scheduling with real-time traffic (preemption, troughs), deadline-aware scheduling by slack, prefix sharing, model placement costs and capacity planning.