Courses › System Design › Key Technologies

SQS & RabbitMQ

Task queues vs logs, visibility timeouts, exchanges, DLQs and idempotent workers.

Amazon SQS and RabbitMQ are task queues: a producer hands off a unit of work, one worker picks it up, does it, and acknowledges it. They're simpler than Kafka for that job, and they're often the better interview answer for background work like sending emails, transcoding videos or processing payments asynchronously. For the general concepts, see Message Queues.

Queue vs log

Amazon SQS

  • Fully managed, with effectively unlimited throughput and no capacity planning.
  • Visibility timeout — when a worker receives a message, it becomes invisible to other workers for a set time (say 5 minutes). If the worker deletes it before then, it's done. If the worker crashes, the message reappears and another worker retries it.
  • Standard queues — nearly unlimited throughput, at-least-once delivery, best-effort ordering.
  • FIFO queues — strict ordering within a message group ID and deduplication within a 5-minute window, at lower throughput.
  • Delay queues — postpone delivery by up to 15 minutes.
  • Dead-letter queue (DLQ) — after N failed receives, the message moves aside for inspection instead of retrying forever.
  • Often paired with SNS for fan-out: one SNS topic publishes to several SQS queues, one per consuming service.
[Order service] -> {SNS: OrderPlaced} -> (SQS: email queue) -> [Email workers]
{SNS: OrderPlaced} -> (SQS: inventory queue) -> [Inventory workers]

RabbitMQ

  • An open-source broker you run yourself (or use as a managed service), implementing AMQP.
  • Producers publish to an exchange, which routes messages to queues by rules: direct (exact routing key), topic (wildcard patterns like orders.*.eu), or fanout (every bound queue).
  • Consumers ack each message; unacked messages are redelivered if the consumer disconnects. Prefetch limits how many unacked messages each worker holds, which balances load.
  • Supports priority queues, per-message TTLs and dead-letter exchanges.
  • Push-based with low latency, but throughput per queue is lower than Kafka's, and a large backlog stresses the broker because queues are meant to stay short.

Designing with task queues

  • Make workers idempotent. Both systems deliver at least once, so a crash after doing the work but before acknowledging means the job runs again. Use an idempotency key — see Safe Retries & Idempotency.
  • Set the visibility timeout above the worst-case job time, or have long jobs extend it with a heartbeat. Otherwise a slow job gets picked up by a second worker.
  • Always configure a DLQ and alert on it.
  • Autoscale workers on queue depth — the backlog is the clearest signal that you need more capacity.

[!INTERVIEW] Video transcoding "When an upload completes, the API puts a job on SQS with the video ID. Transcoding workers poll the queue, extend the visibility timeout while they work, write outputs to S3 and mark the video ready. Workers autoscale on queue depth, the job is idempotent because outputs are keyed by video ID and resolution, and failures land in a DLQ after three attempts." See Managing Long-Running Tasks.