Message Queues

Queues vs logs, delivery guarantees, acks, retries, dead-letter queues and backpressure.

A message queue lets one service hand work to another without waiting for it to finish. The producer writes a message and moves on; a consumer picks it up when it's ready. That one idea solves three common problems: absorbing traffic spikes, doing slow work off the request path, and decoupling services so one failing doesn't take down the other.

When to add a queue

Slow work: sending email, transcoding video, generating reports. Return 202 Accepted right away and process in the background (Managing Long-Running Tasks).
Bursty traffic: a flash sale produces 50× normal orders for a few minutes. The queue buffers them, and workers drain it at a steady rate.
Decoupling: the order service doesn't need to know that billing, shipping and analytics all react to an order.
Retrying flaky work: a failed message goes back on the queue instead of failing the user's request.

api = service "API Servers" @0,1
q = queue "Queue" @1,1
w1 = service "Worker" @2,0.5
w2 = service "Worker" @2,1.5
db = store "Database" @3,1
dlq = queue "Dead-Letter\nQueue" @1,2
api -> q : enqueue, return 202
q -> w1, w2 : pull
w1 -> db
w2 -> db
q --> dlq : after N failures

Queues vs logs

Two designs go by "message queue", and they behave differently:

See Kafka for how partitions and consumer groups work, and Broadcasting with Pub/Sub for delivering one message to many subscribers.

Delivery guarantees

At most once: a message may be lost, never duplicated. Acceptable for metrics you can afford to drop.
At least once: never lost, but may be delivered twice — for example, when a worker finishes the job and crashes before acknowledging it. This is what almost every queue gives you.
Exactly once: only achievable end to end by making the consumer idempotent (or using transactional processing inside one system like Kafka).

[!WARNING] Assume duplicates With at-least-once delivery, design every consumer so that processing a message twice is harmless: use a unique message ID, a database unique constraint, or an upsert. See Safe Retries & Idempotency.

Acknowledgements, retries and dead letters

A consumer acknowledges a message only after it has finished the work. If it doesn't acknowledge within a *visibility timeout*, the queue hands the message to another consumer. A message that keeps failing — a "poison message" — goes to a dead-letter queue after a few attempts, so it stops blocking the rest and someone can inspect it.

Retry with exponential backoff, so a struggling downstream service gets time to recover.

Backpressure and scaling

Watch queue depth and message age. If the queue keeps growing, consumers can't keep up: add workers (autoscale on depth), make processing faster, or push back on producers. A queue hides a capacity problem for a while; it doesn't fix one.

For ordering, remember that more consumers means less ordering. Kafka keeps order within a partition, so pick a partition key (such as user_id) for the things whose order matters, and accept that there's no order across keys.

[!INTERVIEW] Name the guarantee and the failure handling "Uploads publish a job to SQS; transcoding workers pull, process, and ack. Delivery is at-least-once, so the worker skips any video already marked done. After three failures the job goes to a DLQ and we alert" covers the queue, the guarantee, idempotency and poison messages.