Courses › System Design › Question Breakdowns

Design a Chat LLM Serving Platform

Continuous batching, KV-cache affinity, token-based admission control, streaming, cancellation and GPU autoscaling.

A chat LLM serving platform is the backend behind ChatGPT or Claude: users send messages in a conversation, and the model's response streams back token by token. Most of the system looks like any chat app, but the core is unusual: the expensive resource is GPU time, requests last seconds to minutes, and cost and latency depend on tokens, not requests. It's a question about GPU scheduling and batching, streaming, KV-cache reuse, and fair capacity allocation when demand exceeds supply. For the product-level design of a playground UI, see Design ChatGPT Playground.

Requirements

Functional

  • Users can send a message in a conversation and receive the model's reply streamed token by token
  • Users can continue, regenerate or stop a response
  • Users can view conversation history
  • The platform supports several models of different sizes and tiers (free, paid, API)

Out of scope: model training, tool use / browsing, file uploads, safety classifier internals.

Non-functional

  • Time to first token (TTFT): < 1 s at p95 for paid users
  • Streaming speed: ≥ 30 tokens/s per response
  • Scale: ~50M daily users, ~15k new requests/s at peak, ~200k responses streaming concurrently
  • Cost efficiency: GPUs are the dominant cost; maximize tokens per GPU-second
  • Fairness and protection: spiky traffic and abusive users must not starve paying customers

[!TIP] Think in tokens, not requests One request may be 50 input tokens or 100,000; one response may be 10 tokens or 4,000. Capacity, rate limits, routing and pricing should all be expressed in tokens per second. Saying this early frames every later decision.

Capacity estimates

Core Entities

  • Conversation — id, user, title, model, created time
  • Message — conversation, role (user / assistant), content, token counts, status (streaming / complete / stopped / failed)
  • Inference request — message, model, prompt tokens, max tokens, priority tier, assigned replica
  • Quota — user or org, tier, tokens-per-minute limit, usage

API

POST /v1/conversations/{id}/messages     { content, model }
→ 200 text/event-stream
  data: {"type":"token","text":"Hel"}
  data: {"type":"token","text":"lo"}
  data: {"type":"done","messageId":"m_9","usage":{"input":812,"output":245}}

POST /v1/messages/{id}/stop
GET  /v1/conversations/{id}/messages?cursor=

High-Level Design

user = client "Clients" @0,1
gw = service "API Gateway\n(auth, token rate limits)" @1,1
chat = service "Chat Service" @2,1
convo = store "Conversation Store" @2,2
router = service "Inference Router\n(queue, priority, KV affinity)" @3,1
pool1 = service "Model A replicas\n(GPU, batched)" @4,0.5
pool2 = service "Model B replicas" @4,1.5
q = queue "Usage Events" @3,2
user <-> gw : SSE stream
gw -> chat
chat -> convo
chat -> router
router -> pool1, pool2
chat --> q : token usage
  • Accept: the gateway authenticates, checks the user's token-based rate limit, and opens an SSE stream.
  • Build the prompt: the chat service loads conversation history (truncated or summarized to fit the context window) and adds the system prompt.
  • Route: the inference router picks a replica of the requested model with capacity — preferring one that already has this conversation's prefix cached.
  • Generate: the replica adds the request to its running batch and streams tokens back through the chat service to the client.
  • Persist and meter: the finished message is saved, and token usage is emitted for quotas and billing.

Deep Dives

1. Batching on the GPU

A single request barely uses a GPU: generating a token is limited by memory bandwidth, and that cost is nearly the same for 1 sequence or 64. Batching multiplies throughput.

Engines like vLLM, TensorRT-LLM and SGLang implement continuous batching. Two more phases matter:

  • Prefill (processing the prompt) is compute-heavy and fast per token; decode (generating) is memory-bound. Long prompts in prefill can stall decoding for everyone, so engines chunk prefill, and large deployments run prefill and decode on separate GPU pools.
  • The batch size is limited by KV-cache memory — each in-flight token of context uses GPU memory. PagedAttention-style allocation avoids fragmentation.

See LLM Inference.

2. Reusing the KV cache across turns

Each turn of a conversation re-sends the whole history. Recomputing it is wasteful.

  • Prefix caching: the replica keeps the attention KV cache for recent prompts; if a new request shares a prefix (same system prompt + same history), only the new tokens are prefilled.
  • The router uses affinity: hash the conversation ID (or prefix hash) to prefer the replica that served the previous turn, falling back to the least-loaded replica if it's busy.
  • This cuts TTFT and GPU cost dramatically for long conversations.

3. Admission control, queueing and priority

When demand exceeds GPU capacity, something must wait or be rejected.

  • The router keeps per-model queues by tier (enterprise API, paid, free). Replicas pull work as batch slots free up.
  • Token-based rate limits per user and org (tokens per minute, estimated from prompt size + max tokens, reconciled after completion). See Throttling with Rate Limits.
  • If the queue wait would exceed the TTFT target, shed load: reject free-tier requests with a "high demand" message, or route them to a smaller model.

[!WARNING] Round-robin load balancing doesn't work here Requests vary 1000× in cost. Route on actual replica load — queued tokens, KV-cache usage, active sequences — reported by each replica, not request counts.

4. Streaming and stop

  • SSE from client to gateway; gRPC streams internally. See Real-time Updates.
  • The chat service accumulates tokens and writes the message periodically (every ~1 s) so a refresh mid-stream shows partial text.
  • Stop sends a cancel to the replica, which removes the sequence from the batch immediately — freeing GPU capacity matters more than the network close.
  • If a client disconnects, keep generating to completion (the user will see it on refresh) or cancel to save GPU time — a product decision worth stating.

5. Failures mid-response

  • A GPU or replica failure kills every sequence in its batch. The router detects it and retries the request on another replica; the client receives a "retrying" event, or the response restarts.
  • Health checks and outlier detection remove slow or erroring replicas.
  • Persisted partial messages are marked failed with a regenerate option if retries are exhausted.

6. Autoscaling and capacity

  • GPUs take minutes to provision and models take minutes to load (tens to hundreds of GB of weights), so reactive autoscaling is slow. Scale on forecasts plus a buffer, with fast weight loading from local NVMe or peer-to-peer distribution. See Design ML Model Distribution.
  • Shift capacity between models based on demand, and use idle off-peak capacity for batch inference jobs.
  • Serve multiple regions and route by GPU availability, not just user proximity — a few hundred milliseconds of network latency is small compared to queueing.

7. Conversation history and context limits

  • Store conversations sharded by user ID; recent ones are cached.
  • When the history exceeds the model's context window, truncate older turns or summarize them, and keep the system prompt pinned.

What's expected at each level

  • Mid-level: a streaming chat API, conversation storage, a queue in front of GPU workers, and basic rate limiting.
  • Senior: continuous batching and why it matters, token-based rate limits and priority tiers, load-aware routing, cancellation and failure handling.
  • Staff: prefill/decode separation, KV-cache affinity and prefix caching, admission control under GPU scarcity, slow autoscaling and model loading, multi-region capacity management, and cost per token as a design metric.