Courses › System Design › Key Technologies

Apache Spark

Large-scale batch processing, shuffles, data skew and where batch fits next to streaming.

Apache Spark is the standard engine for large-scale batch processing: it runs the same computation in parallel over terabytes or petabytes of data spread across a cluster. In interviews, it's what you reach for when the requirement says "daily," "hourly," "reprocess everything" or "train on all historical data."

The mental model

(S3: raw events, Parquet) -> [Spark: read] -> [filter / join / groupBy] -> [Spark: write] -> (Warehouse / S3 tables)
  • A driver program plans the job; executors on many machines do the work.
  • Data is split into partitions; each task processes one partition. Parallelism comes from having many partitions.
  • You write high-level code with DataFrames or Spark SQL; Spark's optimizer turns it into an execution plan.
  • Lazy evaluation — transformations (filter, join) only build the plan; an action (write, count) runs it.

Narrow vs wide transformations

The shuffle is where Spark jobs spend most of their time and where they fail. Data skew — one key with far more rows than the rest — makes one task run for hours while the others finish in minutes. Common fixes are salting the hot key, broadcast joins (sending a small table to every executor instead of shuffling the big one), and filtering early.

Fault tolerance

  • Spark records the lineage of each dataset — the steps that produced it. If an executor dies, Spark recomputes only the lost partitions from their inputs.
  • Intermediate data can be cached in memory for iterative work (machine learning, graph algorithms), which is a big part of why Spark replaced MapReduce.

Where it fits

  • Storage — Spark doesn't store data. It reads and writes columnar files (Parquet, ORC) in blob storage, often managed as tables with Delta Lake or Iceberg.
  • Batch ETL — clean, join and aggregate raw logs into reporting tables.
  • Backfills and reprocessing — recompute history after a bug or a new feature.
  • Machine learning — feature generation and training-data preparation at scale.
  • Structured Streaming — Spark can also process streams in micro-batches (seconds of latency). For true low-latency, stateful streaming, Flink is the more common answer.

When not to use it

  • Low-latency, per-request work — Spark jobs take seconds to start and minutes to run
  • Small data that fits in one machine's memory — a single process or SQL query is simpler and faster
  • Serving queries to users — load the results into a database or OLAP store first

[!TIP] Batch plus streaming Many interview designs pair a fast streaming path (Flink) with a slower, exact batch path (Spark over the raw log). The stream gives fresh approximate numbers; the batch job reconciles them. Mentioning this shows you've thought about correctness, not just latency.

[!INTERVIEW] Recommendations pipeline "Every night a Spark job reads the last 30 days of watch events from S3, joins them with the catalog, computes per-user features and candidate items, and writes the results to a key-value store keyed by user_id. The serving path just does a lookup."