Courses › System Design › Patterns

Safe Retries & Idempotency

Idempotency keys, dedup in consumers, Kafka transactions and the transactional outbox.

Networks fail in the worst possible place: after the server did the work but before the client heard back. The client can't tell "it never happened" from "it happened and the reply was lost", so it retries — and without care, the user is charged twice, the email goes out twice, the order ships twice. Idempotency means doing an operation once or many times has the same effect. It's how you make retries safe.

Delivery guarantees, honestly

The trick everyone actually uses: **at-least-once delivery + idempotent processing = exactly-once *effect***. You accept duplicates in transit and make sure processing them twice changes nothing.

Naturally idempotent operations

Some operations are already safe to repeat:

  • SET balance = 100, DELETE item 7, PUT /users/42 {…} — the end state is the same however many times they run.
  • INCREMENT balance BY 10, POST /orders, "send email" — each repetition adds another effect. These are the ones to fix.

Often you can rewrite the second kind into the first: "mark order 991 as paid" instead of "add a payment"; "set the read cursor to message 500" instead of "mark the next message read".

Idempotency keys

For operations that create something, the client generates a unique key per logical operation and sends it with every attempt:

POST /payments
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7
{ "amount": 4999, "currency": "usd", "card": "card_123" }

The server:

  • Looks up the key. Seen and finished? Return the stored response without doing the work again.
  • Seen and still in progress? Return a 409 (or wait) — never run it twice in parallel.
  • New? Record the key as in progress, do the work, store the response with the key.

This is how payment APIs like Stripe work: the first response for a key — success or failure — is saved and replayed for retries, a key reused with *different* parameters is rejected, and keys expire after about a day.

[!WARNING] Record the key and the effect together If you write the key in one place and the order in another, a crash between them breaks the guarantee. Put both in one database transaction — or make the key a unique constraint on the table you're writing to (UNIQUE(idempotency_key)), so a duplicate insert simply fails.

INSERT INTO payments (idempotency_key, user_id, amount_cents, status)
VALUES (:key, :user, 4999, 'pending')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;
-- no row returned → this is a retry; fetch and return the existing payment

Idempotent consumers

Message queues like Kafka redeliver after a consumer crashes, so every consumer must tolerate duplicates:

  • Dedup table: store processed message IDs (with a TTL longer than any redelivery window) and skip ones you've seen — in the same transaction as the side effect.
  • Version checks: apply an update only if its version is newer than what's stored (WHERE version < :incoming). This also handles out-of-order delivery.
  • Upserts keyed by a business ID instead of blind inserts.

Exactly-once inside Kafka

Kafka offers exactly-once semantics, with a precise scope:

  • An idempotent producer attaches a producer ID and per-partition sequence numbers, so the broker discards duplicate sends from retries.
  • Transactions let a read-process-write job atomically write its output *and* commit its consumer offsets; consumers with isolation.level=read_committed never see aborted writes.

That covers Kafka-to-Kafka pipelines (e.g. Kafka Streams, Flink). It does not cover side effects outside Kafka — a database write, an HTTP call, an email — which still need idempotency keys or dedup.

The dual-write problem

"Save the order, then publish OrderCreated" is two writes to two systems; either can fail after the other succeeded. The transactional outbox fixes it:

[Order Service] -> (DB: orders + outbox, one transaction)
(DB: orders + outbox, one transaction) -> [Relay / CDC] -> {Kafka} -> [Consumers]

The service writes the order and an outbox row in one transaction. A relay (a poller, or CDC with Debezium) publishes outbox rows to the broker and marks them sent. Publishing can repeat, so consumers dedupe by event ID — at-least-once plus idempotency again.

Calling external systems

When you call a third party (a payment processor, an email provider), pass your idempotency key to them if they support it, and record the attempt before calling. After a timeout, don't blindly retry with a new request: query the provider for the status of the original, or retry with the same key. For financial flows, see Keeping Money Correct.

[!INTERVIEW] What a strong answer sounds like "The client creates an idempotency key when the user taps Pay and reuses it on every retry. The payment service inserts the payment row with that key under a unique constraint in the same transaction as an outbox event, so a retry either finds the existing payment and returns its result, or fails the insert. We pass the same key to Stripe. Downstream consumers are at-least-once, so they dedupe on the event ID."