Design a Chat LLM Serving Platform
Hard · System Design · about 45 minutes
Asked at Anthropic, Databricks, Microsoft, NVIDIA, OpenAI, Scale AI, Perplexity, xAI, Cohere.
Design the backend for a ChatGPT-style product: users send messages in a conversation, and model responses stream back token by token. GPUs are scarce and expensive; traffic is spiky.