Design a Web Crawler
Medium · System Design · about 45 minutes
Asked at Google, OpenAI, Anthropic, Microsoft, xAI, Mistral AI.
Design a crawler that fetches 10 billion web pages within 5 days and stores their text for training or indexing. It must respect robots.txt and per-domain politeness, and survive crashes without re-crawling everything.