Courses › System Design › Question Breakdowns

Design a Job Scheduler

Finding due work with a two-layer poll, idempotent claims, leases, retries and the top-of-hour spike.

Design a service that runs other teams' jobs at the right time: once at a given moment, or on a cron schedule ("every day at 02:00 UTC"). It's a question about finding due work efficiently, running each execution once and recovering when workers die.

Requirements

Functional

  • Users can schedule a job to run once at a time, or on a recurring cron schedule
  • Users can see the status of executions (queued, running, succeeded, failed) and their history
  • Failed executions are retried according to a policy
  • Users can cancel or pause a job

Out of scope: job dependencies / DAGs (discuss as an extension), the code that jobs run (we invoke a container image or an HTTP endpoint).

Non-functional

  • Scale: 10k executions/second at peak (think midnight, when many cron jobs fire)
  • Timeliness: an execution starts within ~2 seconds of its scheduled time
  • At-least-once delivery with idempotent execution, so that from the user's perspective it runs exactly once
  • Durable: a scheduled job is never forgotten, even across scheduler crashes

[!TIP] Separate the job from its executions A cron job is one job definition but thousands of executions over time. Modeling them separately is what makes "find everything due in the next minute" a cheap index lookup rather than a cron-expression evaluation over every job.

Capacity estimates

Core Entities

  • Job — job ID, owner, schedule (cron or timestamp), target (image / URL), retry policy, status
  • Execution — job ID, scheduled time, status, attempt, worker, lease expiry, result

API

POST /v1/jobs
{ "schedule": "0 2 * * *", "target": { "image": "reports:1.4" }, "retry": { "max": 3, "backoff": "exp" } }
→ 201 { "jobId": "job_81" }

GET    /v1/jobs/{jobId}/executions?status=failed&limit=50
DELETE /v1/jobs/{jobId}

High-Level Design

user = client "Client" @0,0
api = service "Job API" @1,0
jobs = store "Jobs DB" @2,0
exec = store "Executions\n(by scheduled time)" @2,1
sched = service "Scheduler\n(watcher)" @1,1
q = queue "Ready Queue" @1,2
w = service "Workers" @2,2
user -> api : create job
api -> jobs
api -> exec : next execution
sched -> exec : due in next minute?
sched -> q : enqueue at time
q -> w
w -> exec : status, heartbeat
  • Create: the API stores the job and inserts its next execution (computed from the cron expression) into the executions table, indexed by scheduled_time.
  • Find due work: a scheduler polls for executions due in the next minute or so and pushes them onto a queue that releases each at its scheduled time.
  • Run: workers pull executions, run the target, heartbeat while running and record the result.
  • Recur: when an execution of a cron job is created or finishes, the next one is inserted, so there's always one future execution per active job.

Deep Dives

1. Finding "jobs due now"

Scanning every job and evaluating its cron expression each second doesn't scale. The executions table with an index on (scheduled_time, status) turns it into a range query:

SELECT id FROM executions
WHERE scheduled_time <= now() + interval '1 minute' AND status = 'pending'
ORDER BY scheduled_time LIMIT 1000;

The two-layer pattern keeps the database query cheap and timing precise:

  • Layer 1 (durable, coarse): every ~30s the scheduler pulls executions due in the next couple of minutes.
  • Layer 2 (precise): it puts them in a delay queue that releases each at the exact time — SQS with delivery delay, a Redis sorted set scored by timestamp, or Kafka with timer topics.

Jobs created for a time less than a minute away skip layer 1 and go straight into the delay queue.

2. Exactly once from the user's point of view

True exactly-once delivery isn't possible across failures, so combine at-least-once delivery with idempotent claiming:

UPDATE executions
SET status = 'running', worker = :w, lease_until = now() + interval '60 seconds'
WHERE id = :id AND status = 'pending';

If two workers receive the same message, only one update matches a row; the other gets 0 rows and drops it. The job's own side effects should also be idempotent (pass the execution ID to the target as an idempotency key).

3. Workers that die mid-job

A worker that crashes can't report failure. Use leases: the worker heartbeats every ~15s to extend lease_until. A reaper finds running executions whose lease expired and sets them back to pending (attempt + 1) or failed when retries run out. Retries use exponential backoff with jitter so a broken downstream isn't hammered.

[!WARNING] Long jobs and stale workers A worker that was only paused (long GC, network partition) may wake up after its lease was handed to someone else. Give each claim a fencing token (the attempt number) and have the worker's final status write check it, so a zombie can't overwrite the newer attempt's result.

4. The midnight spike

Cron favors round numbers: a huge share of jobs fire at :00. Options:

  • Pre-load executions into the delay queue a few minutes early, so the database isn't queried in the spike.
  • Partition the executions table and the scheduler by hash(jobId); each scheduler instance owns a set of partitions (assigned through ZooKeeper/etcd or a lease table), so they don't compete for the same rows.
  • Autoscale workers on queue depth, and let users opt into jitter ("any time in the first 5 minutes").

5. Storage choice

Executions are write-heavy and queried by time range and job ID. Postgres partitioned by day works well for moderate scale; at 1B rows/day, a wide-column store like Cassandra/DynamoDB with scheduled_time bucketed into the partition key (e.g. one partition per minute per shard) avoids hot partitions. Move history older than ~30 days to cheap storage.

6. Fairness between tenants

One team scheduling 1M jobs at midnight shouldn't starve everyone else. Give each tenant a quota and use per-tenant queues drained round-robin (or weighted fair queuing) by the dispatcher.

What's expected at each level

  • Mid-level: jobs and executions tables, a poller that finds due work, workers that run it and record status.
  • Senior: the two-layer poll + delay queue, conditional claims for idempotency, leases and retries with backoff.
  • Staff: fencing tokens for zombie workers, partitioning schedulers to avoid contention, handling the top-of-hour spike, tenant fairness and DAG extensions.