Courses › System Design › Question Breakdowns

Design Google News

Adaptive ingestion, clustering articles into stories, coverage-based ranking and cached fronts with personalization.

A news aggregator like Google News collects articles from thousands of publishers, groups articles about the same story together, ranks the stories, and shows each user a fresh, personalized front page. It's a question about continuous ingestion, clustering near-duplicate content, freshness-aware ranking, and serving a read-heavy, mostly cacheable front page with a personalized layer on top.

Requirements

Functional

  • The system ingests articles from tens of thousands of publishers continuously
  • Articles about the same event are grouped into stories
  • Users see a front page of top stories, by region and category (World, Tech, Sports)
  • Users see a personalized "For you" section based on interests and reading history
  • Clicking an article sends the user to the publisher's site

Out of scope: hosting full article content, comments, publisher paywalls, ads.

Non-functional

  • Freshness: a breaking story appears within ~5 minutes of publication
  • Scale: ~50k publishers, ~2M new articles/day; ~100M daily readers
  • Low latency: front page loads in < 200 ms
  • Availability: stale front pages are acceptable during incidents; empty ones are not

[!TIP] The insight that frames the design Split the system into an offline/near-line pipeline (ingest → extract → cluster → rank) that continuously produces ranked story lists, and an online serving layer that just reads those lists, mostly from cache, and blends in a small personalized part. The front page is a precomputed artifact, not a query.

Capacity estimates

Core Entities

  • Publisher — id, name, feeds, crawl interval, authority/quality score
  • Article — id, publisher, URL, title, snippet, published time, language, entities, embedding, story id
  • Story (cluster) — id, representative article, articles, topic/category, first seen, score
  • User profile — followed topics and sources, reading history, region, language

API

GET /v1/frontpage?region=us&lang=en              → sections[] of stories
GET /v1/stories?category=tech&region=us&cursor=  → stories[]
GET /v1/stories/{id}                             → story with articles from many publishers
GET /v1/foryou?cursor=                           → personalized stories
POST /v1/events   { type: "click", storyId, articleId }

High-Level Design

pubs = client "Publisher Feeds\n(RSS, sitemaps)" @0,0
crawl = service "Fetchers" @1,0
q = queue "Article Stream" @2,0
proc = service "Extract + Embed" @3,0
clus = service "Clusterer" @4,0
stories = store "Stories + Articles" @4,1
rank = service "Ranker\n(per region, category)" @3,1
lists = cache "Ranked Lists" @2,1
api = service "Frontpage API" @1,1
user = client "Readers" @0,1
pubs -> crawl
crawl -> q
q -> proc
proc -> clus
clus -> stories
stories -> rank
rank -> lists
api -> lists
user -> api
  • Ingest: fetchers poll publisher feeds and sitemaps (and accept push via WebSub), dedupe URLs, and fetch new articles.
  • Extract: pull title, text, publish time, language and named entities; compute a text embedding.
  • Cluster: assign each article to an existing story or start a new one.
  • Rank: every minute or so, score stories per region and category and write ranked lists to a cache.
  • Serve: the front page API reads the ranked lists, adds the user's personalized section, and returns.

Deep Dives

1. Ingestion and freshness

  • Poll each feed on an adaptive interval: major publishers that post every few minutes are polled every minute; small blogs hourly. Back off when nothing changes, and use conditional requests (ETag, If-Modified-Since).
  • Accept push (WebSub, publisher APIs) where available for the fastest updates.
  • Dedupe by canonical URL before fetching, and respect robots.txt and rate limits per domain. See Design a Web Crawler.

2. Grouping articles into stories

Twenty publishers cover the same event with different headlines. They must appear as one story.

  • Compute an embedding per article; query a vector index of recent story centroids (last ~48 hours).
  • If the best match is above a similarity threshold and time and entities (people, places) agree, join that story; otherwise start a new one.
  • Periodically re-cluster recent stories offline to merge stories that split and split ones that grew too broad.
  • MinHash catches syndicated copies (the same AP story on 300 sites) cheaply before the embedding step. See Probabilistic Data Structures.

3. Ranking stories

Score each story with signals such as:

  • Coverage: how many distinct, authoritative publishers are covering it (strongest signal of importance)
  • Velocity: how fast new articles and clicks are arriving
  • Freshness: exponential time decay
  • Publisher quality and diversity of sources
  • Regional relevance: entities and publishers associated with the region

Recompute scores every minute per (region, language, category) and store top-N lists in Redis. Since there are only thousands of distinct lists, this is cheap. Breaking-news detection (a sudden spike in coverage) can trigger an immediate recompute.

4. Personalization

  • The "For you" section ranks candidate stories by similarity to the user's interests: followed topics, entities and publishers from their reading history, stored as a profile (an embedding plus topic weights).
  • Candidates come from the regional ranked lists plus topic lists the user follows, so personalization is a re-ranking of a few hundred candidates, not a search over everything.
  • Click events flow through Kafka to update profiles near-real-time and to feed ranking signals. See Aggregating Live Event Streams.

5. Serving at scale

  • The non-personalized front page for each region and category is cached at the CDN for ~30–60 s — most traffic never reaches the backend.
  • The personalized section is fetched separately (a second request or edge-side composition) so the cacheable part stays cacheable.
  • If the pipeline stalls, keep serving the last good ranked lists. See Scaling Reads.

6. Quality and diversity

  • Show several publishers per story and avoid letting one source dominate the page.
  • Down-rank low-quality or misleading sources using publisher-level quality scores; demote clickbait signals.
  • Ensure category diversity on the front page rather than ten stories about the same topic.

What's expected at each level

  • Mid-level: feed ingestion, article storage, a ranked list of recent articles per category, and caching.
  • Senior: clustering articles into stories, freshness-aware ranking with coverage and velocity, precomputed ranked lists, and CDN caching with a separate personalized layer.
  • Staff: near-duplicate detection plus embedding clustering with re-clustering, breaking-news detection, personalization as candidate re-ranking, quality and diversity controls, and degradation strategies.