Design a Batch Inference System
Hard · System Design · about 45 minutes
Asked at Anthropic, OpenAI, Databricks, NVIDIA, Scale AI, Perplexity, Cohere, Mistral AI.
Design a batch inference API: customers upload a file of up to 1M model requests, each with a custom ID, and get a results file within 24 hours at a discount. Requests run on a shared GPU fleet that also serves real-time traffic, each GPU accepts at most N concurrent inputs, and GPUs can fail mid-batch. Every request must end with exactly one result.