Courses › System Design › Question Breakdowns

Design a Web Crawler

Frontier queues, politeness, dedup, crawler traps and checkpointed, fault-tolerant pipelines.

Design a web crawler that fetches billions of pages — for example, to collect training data for an LLM or to build a search index. It's a data pipeline question: throughput, politeness, deduplication and fault tolerance.

Requirements

Functional

  • Crawl the web starting from a set of seed URLs
  • Extract text from each page and store it
  • Discover new URLs from links and crawl them

Out of scope: processing of non-HTML content, ranking or indexing the data.

Ask what the crawl is for, because it changes the design. A crawl for LLM training data cares about text quality, near-duplicate removal and honoring site owners' opt-outs (including AI-specific robots.txt rules such as a GPTBot user-agent block). A crawl for a search index cares more about freshness: re-visiting pages that change often.

Non-functional

  • Scale: 10B pages in 5 days
  • Politeness: obey robots.txt and avoid overloading any single domain
  • Fault tolerance: crashes must not lose progress or require restarting the crawl
  • Efficiency: don't crawl the same URL or identical content twice

[!TIP] Throughput estimate 10B pages ÷ (5 × 86,400s) ≈ 23k pages/second. At ~100 KB of HTML per page that's ~2.3 GB/s, about 20 Gbps of sustained download, spread across millions of different hosts: clearly a distributed fleet of fetchers.

Storage: HTML compresses ~4×, so a 100 KB page is ~25 KB stored, and its extracted text another ~10 KB. 10B pages × ~35 KB is ~350 TB, cheap in object storage. The scarce resources are bandwidth and politeness: you can't fetch a site faster than its owner allows, however many machines you have.

Data Flow

  • Take a URL from the frontier (the queue of URLs to crawl)
  • Resolve DNS, check robots.txt and per-domain rate limits
  • Fetch the HTML and store it raw
  • Parse: extract text and links
  • Store the text; normalize and dedup new links; add them to the frontier

High-Level Design

front = queue "URL Frontier" @0,1
fetch = service "Fetchers" @1,1
dns = cache "DNS Cache" @1,0
meta = store "Domain Metadata\n(robots.txt, limits)" @0,0
raw = blob "Raw HTML" @2,1
pq = queue "Parse Queue" @2,2
parse = service "Parsers" @1,2
text = blob "Extracted Text" @1,3
dedup = service "URL Filter\n+ Dedup" @0,2
front -> fetch
fetch -> dns
fetch -> meta
fetch -> raw
raw -> pq
pq -> parse
parse -> text
parse -> dedup : new links
dedup -> front

Splitting fetching and parsing into separate stages connected by queues lets each scale independently and retry independently.

Inside a fetcher

A fetcher is an async event loop holding thousands of requests in flight. For each URL it:

  • Checks the cached robots.txt rules for the host.
  • Sends the request with timeouts (connect, first byte, total) and a size cap, and stops early if the Content-Type header isn't one it wants.
  • Follows at most a few redirects, recording the final URL so the redirect target isn't queued twice.
  • On a re-crawl, sends If-None-Match / If-Modified-Since; a 304 Not Modified costs almost nothing.
  • Writes the compressed body to object storage and a parse task to the queue, then acks the frontier message.

What to crawl next isn't plain breadth-first. Score each URL (link-based importance, the site's quality, how often the page has changed before) and let the frontier hand out high scores first. Hubs that change often (news front pages) get re-crawled every few minutes; deep archive pages, monthly.

Deep Dives

1. Fault tolerance

  • The frontier and parse queues are durable (Kafka or SQS), so work survives crashes.
  • Workers acknowledge a message only after their output is stored — at-least-once processing.
  • Stages are idempotent: re-fetching a URL or re-parsing a page overwrites the same output keys.
  • Track per-URL state (queued, fetched, parsed, failed, retry count) in a metadata store; failed fetches retry with exponential backoff and give up after N attempts.

2. Politeness

  • Cache robots.txt per domain (refresh daily) and skip disallowed paths.
  • Enforce a per-domain rate limit (e.g. 1 request/second, or the site's crawl-delay). One approach is to route URLs to per-domain queues so a single worker owns each domain's pacing. Another is a shared rate-limit check (Redis) before fetching, putting the URL back with a delay if the domain is busy.
  • Cache DNS results — DNS lookups otherwise become a bottleneck.

A frontier design that makes politeness automatic: keep a queue per host, plus one min-heap of hosts ordered by the time each is next allowed to be fetched. A fetcher pops the host whose time has come, takes one URL from that host's queue, and pushes the host back with now + crawl_delay. No host is ever fetched faster than its delay, and no fetcher sits idle while some other host is ready.

With billions of URLs waiting, host queues live in a disk-backed store (a key-value store, or Kafka partitions keyed by host), and only the heap and the heads of active queues are kept in memory.

3. Deduplication

  • URL dedup: normalize URLs (lowercase host, strip fragments and tracking params), then check a seen set. A Bloom filter gives a compact, fast "definitely not seen" check, backed by a key-value store for confirmation.
  • Content dedup: hash the extracted text; skip storing pages whose hash has already been seen (mirrors, syndicated content).

Sizes: 10B URLs as 8-byte hashes is ~80 GB, while a Bloom filter at a 1% false-positive rate needs ~10 bits per URL, ~12 GB, small enough to keep in memory on every node.

  • Near-duplicates matter more than exact ones, especially for LLM data: the same article with different ads or timestamps hashes differently. Compute a SimHash or MinHash signature of the text and treat pages within a small distance as duplicates.

[!WARNING] A Bloom filter's mistakes are silent When a Bloom filter wrongly says "seen", that URL is never crawled and nothing records the miss. At 1% that's 100M lost pages out of 10B. Size it for a lower rate, or confirm "seen" answers for high-priority URLs against the exact store.

4. Crawler traps and priority

  • Cap URL depth and the number of pages per domain; detect patterns like infinitely generated calendar pages.
  • Crawler traps are URL spaces that never end: a calendar whose "next month" link goes on forever, faceted shop filters that combine into millions of URLs (?color=red&size=m&sort=price…), or session IDs that make every visit look new. Defenses: a per-host page budget, a cap on URL length and query parameters, detecting repeated path segments (/a/b/a/b/a/b), and honoring rel=canonical.
  • Use prioritized frontiers: important or frequently changing pages (high PageRank, news sites) get crawled first and re-crawled more often.

5. Scaling the fetchers

Fetching is network-bound, so each fetcher uses async I/O with thousands of concurrent requests. Scale horizontally, and deploy fetchers in multiple regions to be closer to the sites being crawled.

6. Partitioning, checkpoints and failover

Key everything by host: frontier partitions, per-host politeness state and the per-host share of the seen set. Then one fetcher owns a host at a time, and its checks never leave the machine.

Keep that state in the durable stores (Kafka partitions, a key-value store), not on the fetchers. When a fetcher dies, the consumer group reassigns its partitions to the others, which resume from the last committed offsets. There's nothing to checkpoint or copy, because no fetcher ever held the only copy of anything.

What's expected at each level

  • Mid-level: a pipeline with a frontier, fetchers, parsers and storage; basic dedup and politeness.
  • Senior: throughput math, stage separation with durable queues, retries, Bloom filters, per-domain rate limiting.
  • Staff: prioritization and re-crawl strategy, near-duplicate detection, trap detection, DNS bottlenecks, stateless fetchers over durable host-partitioned state, and cost at this scale.