Courses › System Design › Question Breakdowns

Design ChatGPT Playground

Prompt versions and threads, streaming tokens over SSE, side-by-side runs, quotas and metering.

Design a developer playground for LLMs, like the OpenAI Playground: a web app where developers write a system prompt and messages, tweak model parameters, stream responses, compare models side by side and save prompts to share. It's a product-shaped question: interviewers expect an API contract, a data model for threads and prompt versions, token streaming and usage metering, on top of an inference backend you treat as a dependency.

Requirements

Functional

  • Developers can compose a system prompt + messages, choose a model and parameters (temperature, max tokens, top-p, tools/JSON mode), and run it
  • Responses stream token by token, and can be stopped mid-generation
  • Developers can save prompts with versions, reopen past threads, and share a prompt with teammates
  • Developers can compare the same prompt across 2–3 models or parameter sets side by side
  • Every run shows token usage and cost, and counts against the organization's quota

Out of scope: training or hosting the models themselves (an inference gateway is given), fine-tuning, evals at scale.

Non-functional

  • Low time-to-first-token: streaming starts in < 1s (dominated by the model)
  • Scale: ~1M developers, ~100k concurrent streaming generations at peak
  • Durable history: a completed run is never lost, even if the browser closes mid-stream
  • Accurate metering: usage must match what billing charges
  • Isolation: one organization's prompts and keys are invisible to others

[!TIP] Don't redesign GPU serving The playground sits in front of the model API. Draw the inference gateway as a box, say what you need from it (streaming, cancellation, usage counts) and spend your time on the product: threads, prompt versions, streaming to the browser, quotas. If the interviewer wants GPUs, that's Batch Inference territory.

Capacity estimates

Core Entities

  • Organization / Project — quotas, API keys, members
  • Prompt — owner, name, sharing settings; has many prompt versions (system prompt, messages template, model, parameters — immutable)
  • Thread — a conversation in the playground; has ordered messages (role, content, tool calls)
  • Run — one generation: thread, model, parameters, status, token counts, latency, cost

API

POST /v1/threads/{threadId}/runs
{ "model": "gpt-x", "params": { "temperature": 0.7, "maxTokens": 800 }, "promptVersionId": "pv_12" }
→ 200 text/event-stream
   event: token   data: {"text":"Hello"}
   event: usage   data: {"input":412,"output":96}
   event: done    data: {"runId":"run_77","finishReason":"stop"}

POST /v1/runs/{runId}/cancel
POST /v1/prompts/{promptId}/versions   { systemPrompt, messages, model, params }  → { versionId }
GET  /v1/threads/{threadId}/messages?cursor=...
POST /v1/compare  { promptVersionId, variants: [{model, params}, ...] }  → { runIds: [...] }

High-Level Design

dev = client "Browser\n(playground UI)" @0,1
api = service "Playground API" @1,1
prompts = store "Prompts & Threads DB" @1,0
runs = service "Run Service\n(streaming)" @2,1
gw = service "Inference Gateway" @3,1
models = service "Model Servers" @4,1
quota = cache "Quota / Rate Limits\n(Redis)" @2,0
meter = queue "Usage Events" @2,2
bill = service "Billing" @3,2
dev -> api
api -> prompts
dev <-> runs : SSE stream
runs -> quota : check
runs -> gw : stream request
gw -> models
runs -> meter : tokens used
meter -> bill
  • The Playground API handles CRUD for prompts, versions, threads and sharing.
  • A run goes to the Run Service, which checks the org's quota, opens a streaming request to the inference gateway and relays tokens to the browser over SSE.
  • When the run ends (done, error or cancel), the Run Service writes the assistant message and run record, and emits a usage event for billing.

Deep Dives

1. Streaming tokens to the browser

SSE is the natural fit: one direction (server → client), plain HTTP, automatic reconnect. WebSockets work too but add little here.

Long-lived streams mean the Run Service is effectively stateful for the duration of a run. Keep it simple: the server that opened the stream holds the upstream connection, and the load balancer must allow long idle timeouts (minutes).

[!WARNING] The browser closing isn't the run ending If a developer closes the tab mid-stream, decide explicitly: keep generating and save the result (useful for long runs, and you're paying for the tokens anyway) or cancel upstream to save GPU time. Most playgrounds cancel on explicit Stop, and finish-and-save on disconnect so the thread is complete when they come back.

Resuming a stream: buffer tokens per run in Redis (a stream keyed by run ID with a short TTL). A reconnecting client sends Last-Event-ID and gets the missed tokens, then the live tail.

2. Saving runs durably without slowing the stream

Writing to the database on every token is wasteful. Accumulate the response in memory (and in the Redis buffer), then write one message row at the end. If the Run Service crashes mid-run, the Redis buffer lets a recovery job store the partial response marked interrupted.

3. Prompt versions and sharing

Treat each saved version as immutable: editing creates a new version, and threads and runs reference the exact version they used. That makes "what prompt produced this output?" always answerable and makes sharing safe (a teammate's edits don't change your history).

  • Sharing is an ACL on the prompt: private, project, or link (anyone in the org with the link).
  • Store prompts and threads in Postgres (relational, transactional, modest size per org); put large message bodies in a separate table or object storage if they grow beyond a few hundred KB.

4. Side-by-side comparison

POST /compare creates one run per variant and returns their IDs; the browser opens one SSE stream per run (or one multiplexed stream with a runId on each event). Variants run in parallel against the gateway, and each is metered separately.

5. Quotas, rate limits and metering

  • Before a run: check the org's requests-per-minute and tokens-per-minute in Redis (token bucket — see Design a Rate Limiter). Reserve the maximum possible tokens (input + maxTokens), then release the unused part when the run finishes.
  • After a run: emit a usage event (runId, org, model, inputTokens, outputTokens) to Kafka. Billing consumes it idempotently by runId, so retries can't double-bill.
  • Token counts come from the gateway, not the browser, so they match what was actually generated.

6. Secrets and isolation

Developers may paste API keys or customer data into prompts. Encrypt message content at rest, scope every query by org_id, and let orgs set retention (e.g. delete threads after 30 days). Never log prompt text in plain application logs.

What's expected at each level

  • Mid-level: a clear API, prompt/thread/message tables, SSE streaming from the inference API to the browser.
  • Senior: immutable prompt versions, durable run records, cancellation and reconnect behavior, per-org quotas with token reservation.
  • Staff: metering that billing can trust, stream resumption, multi-model comparison, data retention and isolation, and a crisp boundary between the product layer and GPU serving.