Scheduling Future Work

Finding due jobs efficiently, leases and retries, recurring schedules and time zones.

"Send this reminder at 9 a.m. tomorrow", "charge every subscriber on the 1st", "expire unpaid reservations after 10 minutes", "rebuild the search index nightly": a lot of work has to happen at a specific time, not in response to a request. Job scheduling is the pattern for storing that future work durably and running each job on time, once, even as machines fail.

Two kinds of jobs

One-off delayed jobs — run at a given time: a reminder, a retry in 5 minutes, a hold that expires. Often millions of them.
Recurring jobs — run on a schedule (a cron expression like 0 2 * * *, every day at 02:00). Usually far fewer, but each can be heavy.

A recurring job is easiest to handle as a generator of one-off executions: when one run is scheduled, compute and store the next.

Why not just cron?

Cron on one machine works until that machine dies (jobs silently don't run), you need two machines (jobs run twice), or you have millions of jobs (cron isn't a database). A real scheduler separates storing jobs, finding due jobs and running them.

api = service "Scheduling API" @0,1
db = store "Jobs DB\n(indexed by run_at)" @1,1
sch = service "Scheduler\n(finds due jobs)" @2,1
q = queue "Ready Queue" @3,1
w = service "Workers" @4,1
api -> db : create job
sch -> db : poll due jobs
sch -> q : enqueue
q -> w
w -> db : status, next run

Finding due jobs efficiently

The core query is "which jobs are due now?" — it must be cheap even with 100M scheduled jobs.

1. Database with an index on run_at. Pollers repeatedly claim a batch of due jobs. In Postgres, FOR UPDATE SKIP LOCKED lets many pollers work in parallel without blocking each other or claiming the same job:

UPDATE jobs SET status = 'claimed', claimed_by = :worker, claimed_at = now()
WHERE id IN (
  SELECT id FROM jobs
  WHERE status = 'scheduled' AND run_at <= now()
  ORDER BY run_at
  LIMIT 100
  FOR UPDATE SKIP LOCKED
)
RETURNING id, payload;

2. Time-bucketed partitions. Store jobs under keys like jobs:2026-10-12T09:05 (one bucket per minute). A scheduler reads exactly one bucket per minute. Works well in Cassandra or DynamoDB at very large scale.

3. Redis sorted set with the run time as the score: ZRANGEBYSCORE jobs -inf <now> returns due jobs. Very fast, but keep the durable copy in a database.

4. A two-tier design for precision at scale: the database holds all jobs; every few minutes a loader moves jobs due in the next window into a fast in-memory structure (a sorted set or timer wheel) that fires them on the second.

[!TIP] Queues with delays cover the simple case For "run this in 10 minutes" and nothing more, a queue with delayed delivery (SQS delay up to 15 minutes, RabbitMQ delayed messages, Redis-based job libraries) may be all you need. Reach for a full scheduler when delays are long, jobs must be queryable or cancellable, or schedules recur.

Running each job once

Exactly-once execution isn't possible; aim for at-least-once execution of idempotent jobs:

Leases, not permanent claims. A claimed job carries claimed_at; if the worker doesn't finish or heartbeat within a timeout, another worker can reclaim it. This is how jobs survive worker crashes.
Idempotent handlers. A job may run twice (a worker finished but crashed before marking it done). Use the job ID + scheduled time as an idempotency key for side effects (see Safe Retries & Idempotency).
Fencing for long jobs. If a slow worker loses its lease and another takes over, the old worker must not overwrite newer results: stamp writes with the lease's version.

Failures and retries

Retry with exponential backoff and a maximum attempt count, then move the job to a dead-letter state for a human to inspect.
Distinguish error types: a timeout is retried; "user deleted" is not.
Missed runs after an outage: decide per job whether to run once to catch up, run every missed occurrence, or skip to the next. A nightly report should run once; billing runs must not be skipped.

Time is harder than it looks

Store times in UTC. For user-facing schedules ("9 a.m. my time, every day"), store the time zone and compute each next run from it, so daylight saving changes keep it at 9 a.m.
Avoid the top-of-the-hour stampede. Millions of jobs at exactly 00:00 overload workers and downstream services. Add jitter where exact time doesn't matter, and rate-limit execution.
Precision: most systems promise "within a few seconds", not to the millisecond. Say what you're promising.

Scaling

Partition jobs (by job ID hash or by tenant) so each scheduler instance polls its own partitions — no contention between pollers.
Scale workers independently of schedulers; the queue between them absorbs bursts.
Separate queues by priority or job type, so a flood of low-priority batch jobs can't delay time-critical ones.

For jobs that take minutes or hours once started, see Managing Long-Running Tasks.

[!INTERVIEW] What a strong answer sounds like "Jobs live in Postgres with an index on (status, run_at), partitioned by job ID into 64 partitions. Each scheduler instance owns a set of partitions and claims due jobs with SKIP LOCKED, pushing them to SQS. Workers execute with a 5-minute lease and heartbeats; an expired lease makes the job claimable again, so handlers are idempotent on job ID plus run time. Recurring jobs store a cron expression and time zone, and the worker inserts the next execution when it finishes. Failures retry with backoff up to 5 times, then go to a dead-letter state."