LLM Inference

Prefill vs decode, the KV cache, continuous batching, and what it means for latency, cost and serving.

More and more interview questions involve an AI feature: a chat assistant, a summarizer, a coding agent. You don't need to know how to train a model, but you should know what happens when one answers a request, because that shapes latency, cost and capacity in ways that are very different from a normal web service.

Tokens in, tokens out

A model doesn't see text; it sees tokens, chunks of roughly ¾ of an English word. A request is a prompt of N input tokens, and the model produces output one token at a time: each new token depends on all the tokens before it. That one fact explains most of what follows.

Two phases: prefill and decode

That's why responses stream: the first token takes a moment (prefill), then tokens arrive at a steady rate (decode). It's also why output tokens are priced several times higher than input tokens.

The KV cache

To produce each new token, attention needs the keys and values of every previous token. Recomputing them every step would be hugely wasteful, so they're stored in the KV cache on the GPU.

The KV cache grows with context length × concurrent requests, and it's often the real limit on how many requests a GPU can serve — not the model weights.
Prefix caching keeps the KV cache for prompt prefixes that repeat across requests, such as a long system prompt or a shared document. A request that reuses a cached prefix skips most of its prefill: faster and cheaper. This is what "prompt caching" in LLM APIs is.

Batching: how one GPU serves many users

Decode is memory-bound: reading the weights once costs about the same whether you generate a token for one request or for fifty. So inference servers batch many requests together.

Continuous batching: rather than waiting for a whole batch to finish, the server adds and removes requests at every decode step. Throughput rises several-fold.
PagedAttention (vLLM): stores the KV cache in fixed-size blocks, like virtual memory pages, so memory isn't wasted on fragmentation and more requests fit at once.

Bigger batches mean more throughput but slower tokens for each user. Every LLM serving system is tuned somewhere on that throughput vs latency curve.

Making it faster

Quantization: store weights in 8 or 4 bits instead of 16. Less memory and bandwidth per token, with a small quality cost.
Speculative decoding: a small draft model guesses several tokens, and the big model checks them all in one pass. Same output, fewer slow steps.
Parallelism: a model too large for one GPU is split across several (tensor or pipeline parallelism), connected by fast interconnects.
Disaggregated serving: run prefill and decode on separate GPU pools, since they stress different resources, and move the KV cache between them.

Serving architecture

user = client "Client" @0,1
gw = service "API Gateway\n(auth, token quotas)" @1,1
router = service "Router\n(model, prefix affinity)" @2,1
p1 = service "Inference Server\n(vLLM / TensorRT-LLM)" @3,0.5
p2 = service "Inference Server\n(GPU pool)" @3,1.5
weights = blob "Model Weights" @4,1
user -> gw : prompt
gw -> router
router -> p1, p2
p1 --> user : stream tokens (SSE)
weights --> p1, p2 : load at startup

Things that differ from a typical web service:

Stream responses over SSE or WebSockets so users see tokens immediately (Networking Essentials).
Route with affinity: send requests that share a prefix (the same conversation) to the same server so its KV cache is reused.
Rate limit by tokens, not requests: one request can cost a hundred times another (Rate Limiters).
Autoscaling is slow: a new GPU server has to load gigabytes of weights before it can serve. Keep headroom, and scale on queue depth and KV-cache use rather than CPU.
Offline work goes to batch: summarizing a million documents doesn't need low latency, so run it as a batch job at higher throughput and lower cost.

Rough numbers

[!INTERVIEW] Talk in TTFT, tokens per second, and cost per token "The chat service streams over SSE. Conversations are routed to the same inference server so the KV cache for the history is reused, which keeps time to first token low. Users get a daily token quota enforced at the gateway, and we scale the GPU pool on queue depth with headroom, since a new node takes minutes to load the weights."