Courses › System Design › Patterns

Managing Long-Running Tasks

Async job queues, workers, retries, idempotency and workflow engines.

Some work takes too long to do inside an HTTP request: transcoding a video, generating a report, sending 10,000 emails, running an ML job. Making a user wait — or worse, timing out halfway — is a poor design. The standard answer is the async worker pattern.

The async worker pattern

Client -> [API Server] -> (Jobs DB)
[API Server] -> {Job Queue} -> [Workers] -> (Jobs DB)
Client -> [API Server]
  • The API validates the request, creates a job record (status = 'queued'), enqueues a message, and immediately returns 202 Accepted with a job ID.
  • Workers pull jobs from the queue, do the work, and update the job's status and result.
  • The client polls GET /jobs/{id} — or gets notified via SSE, WebSocket, webhook or push notification — when the job completes.

This decouples request handling from processing, absorbs spikes (the queue grows; workers catch up), and lets you scale workers independently.

Choosing a queue

  • SQS / RabbitMQ / Redis-based queues — per-message acknowledgement, visibility timeouts, delayed messages, dead-letter queues. Ideal for task queues.
  • Kafka — better for high-volume event streams with replay; per-message retry semantics need more work (retry topics).

Reliability essentials

  • At-least-once processing — a worker can crash after doing the work but before acknowledging. The message will be redelivered, so jobs must be idempotent (check the job status before acting, use idempotency keys with external services).
  • Visibility timeouts / leases — a claimed job becomes available again if the worker doesn't finish in time. Long jobs should heartbeat to extend their lease.
  • Retries with backoff — for transient failures.
  • Dead-letter queue — after N failures, park the job for inspection instead of retrying forever.
  • Priorities — separate queues for urgent and bulk work so a batch job doesn't delay a password reset.

Multi-step workflows

When a job has many steps (upload → transcode to 5 resolutions → generate thumbnails → publish), you need to track progress, retry individual steps, and handle partial failure.

  • Simple chains — each step enqueues the next on success; state lives in the database.
  • Workflow engines (Temporal, AWS Step Functions) — define workflows as code; the engine persists state after every step, retries failures, handles timeouts, and resumes after crashes. Mention them for complex, long-lived or human-in-the-loop processes.

Scheduled and delayed jobs

For "send this at 9am" or "expire this reservation in 10 minutes": use delayed messages, a sorted set of due times polled by a scheduler, or a workflow engine's timers. See the job scheduler question.

[!INTERVIEW] Recognize the trigger Whenever a requirement involves something slow (seconds or more), unreliable (third-party APIs) or bursty, say: "This shouldn't happen in the request path — I'll make it an async job." Then cover idempotency and retries.