Crawling & Large Data Pipelines

Staged pipelines, the frontier, politeness, dedup with Bloom filters and fault tolerance.

Crawling billions of pages, ingesting partner data feeds, scraping prices, building a search index or a training dataset: these are large multi-stage pipelines where work is plentiful, failures are routine and nothing should ever be done twice or lost. The crawler is the classic interview version (see Design a Web Crawler), but the same ideas power any big data pipeline.

Break the work into stages

Don't build one giant worker that fetches, parses, deduplicates and stores. Split it into stages connected by durable queues, so each stage scales on its own and a failure in one doesn't lose work from another.

(Frontier) -> [Fetchers] -> (Raw pages: S3)
(Raw pages: S3) -> {Parse queue} -> [Parsers] -> {Discovered URLs} -> [Dedup + Filter] -> (Frontier)
[Parsers] -> {Extracted data} -> [Indexers / Loaders]

Fetching is network-bound — run many concurrent requests per machine.
Parsing is CPU-bound — scale it separately.
Store raw output (the fetched HTML) in object storage before processing it. If the parser has a bug, you fix it and reprocess from S3 instead of re-crawling the web.

The frontier: what to fetch next

The frontier is the queue of URLs waiting to be fetched, and it does more than first-in-first-out:

Prioritize by importance (page rank, freshness needs) — news homepages hourly, obscure pages monthly.
Group by host so politeness can be enforced: one queue per domain, and each domain is fetched by one worker at a time.
Persist it. A billion-URL frontier lives on disk or in a database, with an in-memory slice of what's next.

Politeness and robots.txt

A crawler that fires thousands of requests at one site is indistinguishable from an attack.

Fetch and cache each site's robots.txt, and obey its disallowed paths and crawl delay.
Limit to roughly one request per second per domain (or what the site allows), using a per-domain next-allowed-time.
Identify your crawler in the User-Agent, and back off on 429 or 503 responses.

Deduplication

The web is full of duplicates — the same page under many URLs, mirrors, tracking parameters.

URL dedup: normalize URLs (lowercase host, drop fragments and tracking parameters, sort query parameters), then check a set of seen URLs. At billions of URLs, a Bloom filter answers "probably seen?" in a few bits per URL, with a small false-positive rate (a few pages skipped).
Content dedup: hash the page body to catch exact copies; use SimHash or MinHash to catch near-duplicates (same article, different ads).

Traps and limits

Crawler traps: infinite calendars, session IDs in URLs, endlessly generated pages. Cap URL length and path depth, limit pages per domain, and watch for domains producing suspiciously many new URLs.
Timeouts and size limits on every fetch, so a slow or enormous response can't tie up a worker.
DNS caching — resolving each host every request is slow and can overload resolvers.

Fault tolerance

At this scale, something is always failing.

At-least-once processing with idempotent stages. Queues redeliver work after a crash; writes are keyed by URL (or content hash), so processing a page twice just overwrites the same record.
Checkpoint progress. Long jobs record which partitions or offsets are complete so a restart resumes instead of starting over.
Retries with backoff for transient failures; a dead-letter queue for permanent ones, with a limit on retries per URL.
Heartbeats and leases on claimed work, so items held by a dead worker go back to the queue.

Scaling the numbers

1B pages per month ÷ 2.6M seconds ≈ 400 pages/sec
average page 100 KB → 40 MB/s ≈ 320 Mbps of download
raw storage: 1B × 100 KB = 100 TB per crawl (compressed ~25 TB)

Work is partitioned by host (hash the domain to a worker), which keeps a domain's politeness state in one place and spreads load evenly. Fetchers should run close to their targets where possible, and you should plan bandwidth and storage explicitly — interviewers expect this back-of-the-envelope.

Recrawling and freshness

Pages change at very different rates. Track how often each page actually changed on past fetches and schedule recrawls accordingly; use conditional requests (If-Modified-Since, ETag) so unchanged pages cost a 304 response instead of a full download.

General data pipelines

The same toolkit applies to ETL and ingestion pipelines:

Batch (daily partner files) with Spark or warehouse SQL, orchestrated by a DAG scheduler like Airflow — stages, dependencies, retries and backfills.
Streaming ingestion through Kafka when freshness matters (see Aggregating Live Event Streams).
Keep raw data immutable and derive everything else from it, so any downstream table can be rebuilt.
Validate data at each stage (schema, row counts, null rates) and quarantine bad batches rather than loading them.

[!INTERVIEW] What a strong answer sounds like "The crawl is a pipeline: a frontier partitioned by host feeds fetchers, which enforce one request per second per domain, obey robots.txt and write raw HTML to S3. Parsing reads from S3 through a queue, so a parser bug means reprocessing, not re-crawling. New URLs are normalized and checked against a Bloom filter before entering the frontier, and page bodies are SimHashed to skip near-duplicates. Every stage is at-least-once and keyed by URL, so retries are harmless, and we cap depth and pages per domain to avoid traps."