Courses › System Design › Patterns

Keeping Money Correct

Payment state machines, idempotency, double-entry ledgers, webhooks and reconciliation.

Payments, wallets, payouts, refunds, account balances: when money moves, "eventually roughly right" isn't acceptable. Every cent must be accounted for, nothing may be charged twice, and you must be able to prove later exactly what happened. Throughput is usually modest — the interview is about correctness under failure, especially when external payment providers are involved.

The non-negotiables

  • Never move money twice for one intent — retries must be safe.
  • Never lose track of money — every movement is recorded, and records always balance.
  • Never rewrite history — corrections are new records, not edits.
  • Always reconcile — your records are checked against the bank's and the provider's.

Payments as a state machine

A payment isn't a single action; it moves through states, and every transition is recorded:

created → authorized → captured → settled
     ↘ failed     ↘ voided       ↘ refunded (partially / fully)
  • Only valid transitions are allowed (captured → authorized is impossible), enforced with a conditional update: UPDATE payments SET status = 'captured' WHERE id = :id AND status = 'authorized'.
  • Authorize then capture: holding funds at checkout and capturing on shipment gives you a cheap cancellation path (void) before money actually moves.

Idempotency end to end

Every payment request carries an idempotency key created by the client for that intent (see Safe Retries & Idempotency):

  • The payment row is created with the key under a unique constraint, so a retried request finds the existing payment instead of creating another.
  • The same key is passed to the provider (Stripe, Adyen), so even a retry to them can't double-charge.
  • Record the attempt before calling out. If the call times out, the payment is in an *unknown* state: don't create a new charge — query the provider by that key, or retry with the same key, until you know the outcome.

The double-entry ledger

Store money movements as a ledger, the way accountants have for centuries. Each transaction has two or more entries that sum to zero: money leaves one account and enters another.

txn 8812  buyer_card_clearing    −50.00
txn 8812  seller_payable         +48.50
txn 8812  platform_fee_revenue    +1.50
                          sum =    0.00   ✓
  • Entries are immutable. A mistake or refund is a new, reversing transaction — never an UPDATE. This gives a complete audit trail.
  • Balances are derived by summing entries. For speed, keep a materialized balance per account, updated in the same database transaction as the entries it reflects.
  • Store amounts as integers in the smallest unit (cents) with a currency code. Never use floating point for money.
  • Enforce the invariants in the database where you can: a check that each transaction sums to zero, and conditional updates that stop a balance going negative (WHERE balance >= :amount).
[Checkout] -> [Payment Service] -> (Payments + Ledger DB)
[Payment Service] -> [Provider: Stripe/Adyen]
[Provider: Stripe/Adyen] -> [Webhook Handler] -> (Payments + Ledger DB)
(Payments + Ledger DB) -> [Outbox] -> {Kafka: payment events}

Strong consistency where it matters

Use a relational database with ACID transactions (PostgreSQL is the usual answer) for the ledger and payment states. The ledger is rarely the throughput bottleneck. When it grows, shard by account; a transfer between two accounts on different shards then becomes a small saga with a clearing account in between, rather than a distributed transaction (see Coordinating Workflows with Sagas).

Everything downstream — emails, analytics, seller dashboards — can be eventually consistent, fed from payment events through an outbox, so an event is published if and only if the transaction committed.

Talking to external providers

  • Webhooks are the source of updates from providers (captured, failed, refunded, disputed). They can arrive late, more than once, or out of order: verify the signature, dedupe by event ID, and apply only valid state transitions.
  • Don't rely on webhooks alone. A periodic job polls the provider for payments stuck in a pending or unknown state.
  • Never trust the client about payment success — confirm with the provider server-side before fulfilling.

Reconciliation

Even with careful code, systems drift: a webhook was missed, a provider reversed a charge, a bug double-posted an entry. Reconciliation catches it:

  • Every day, fetch settlement reports from each provider and bank.
  • Match each external transaction against your ledger (by provider ID and amount).
  • Flag mismatches — missing on either side, amount differences — into a queue for automated fixes or human review.

Interviewers love hearing this: it shows you assume failure rather than hoping it won't happen.

Security and compliance, briefly

Don't store raw card numbers: use the provider's tokenization so card data never touches your servers, which keeps you out of most PCI DSS scope. Log every state change with who and why, and restrict who can create manual ledger adjustments.

[!INTERVIEW] What a strong answer sounds like "The client sends an idempotency key with each checkout. The payment service inserts the payment with that key under a unique constraint, writes the ledger entries and an outbox event in the same Postgres transaction, then calls Stripe with the same key. Payment status moves through a state machine with conditional updates, driven by signature-checked, deduplicated webhooks plus a poller for anything stuck. The ledger is double-entry and append-only — refunds are reversing entries — and a nightly job reconciles it against Stripe's settlement report."