Coordinating Workflows with Sagas

Compensating actions, orchestration vs choreography, pivot steps and workflow engines.

Booking a trip means reserving a flight, a hotel and a car, then charging a card — four services, four databases. There's no single transaction that spans them, and two-phase commit across services is slow and brittle. A saga breaks the process into a sequence of local transactions, each with a compensating action that undoes it if a later step fails.

How a saga works

1. Reserve flight     ✓
2. Reserve hotel      ✓
3. Reserve car        ✗  (no cars available)
   → compensate 2: cancel hotel reservation
   → compensate 1: cancel flight reservation

Each step commits on its own. If a step fails, the saga runs the compensations for the completed steps in reverse order. The system is briefly inconsistent (the flight is held while the hotel is being reserved) but always ends in a consistent state: everything done, or everything undone.

[!NOTE] Compensation is not rollback You can't un-send an email or un-charge a card. A compensation is a new business action with the opposite effect: a refund, a cancellation notice, a release of a hold. Design steps so they're compensatable — "reserve, then confirm" is easier to undo than "book".

Orchestration vs choreography

Orchestration: a central orchestrator tells each service what to do and decides what's next.

[Trip Orchestrator] -> [Flight Service]
[Trip Orchestrator] -> [Hotel Service]
[Trip Orchestrator] -> [Payment Service]
[Trip Orchestrator] -> (Saga state)

Choreography: no coordinator. Each service listens for events and reacts: FlightReserved → hotel service reserves → HotelReserved → payment service charges → and so on.

Interviewers usually like hearing orchestration for anything non-trivial, because the state of each saga is visible and failure handling lives in one place.

Making it reliable

The orchestrator itself can crash mid-saga, so it must be durable:

Persist saga state (current step, results so far) before and after each step, so a restarted orchestrator resumes where it left off.
Every step and every compensation must be idempotent — on resume, the orchestrator may resend a command that already succeeded. Pass a saga ID + step as the idempotency key (see Safe Retries & Idempotency).
Use the outbox pattern when a service updates its database and emits the "done" event, so the event can't be lost.
Timeouts: a step that never answers is treated as failed (or retried, then failed), and compensation begins.
Compensations can fail too. Retry them with backoff — they must eventually succeed — and alert a human when one is stuck.

The pivot step

Some step is the point of no return — typically charging the card or shipping. Order the saga so that:

Steps before the pivot are easily compensatable (holds, reservations).
The pivot runs.
Steps after the pivot are ones that can always be retried to completion (sending a confirmation, updating loyalty points). They're never compensated; they're retried until they succeed.

reserve inventory → reserve delivery slot → [charge card: pivot] → confirm order → send email
   compensatable          compensatable                              retry until done

Isolation problems

Sagas don't have isolation: other requests can see the half-finished state. Common mitigations:

Semantic locks — mark records as PENDING so other flows know not to rely on them (a held seat isn't sellable, but isn't sold either).
Reorder steps so the riskiest, most likely to fail step comes first.
Re-read and re-check values at the step that commits, rather than trusting what an earlier step saw.

Workflow engines

Writing durable orchestrators by hand — state tables, retry timers, timeouts — is a lot of code. Workflow engines (Temporal, AWS Step Functions, Cadence) do it for you: you write the saga as ordinary code, and the engine records each step's result in an event history and replays it after a crash, so the function resumes exactly where it stopped. The rule is that workflow code must be deterministic; side effects go in *activities* that the engine retries. Mention one by name for complex flows; it signals you know the operational reality. See also Managing Long-Running Tasks.

[!INTERVIEW] What a strong answer sounds like "Checkout is an orchestrated saga: reserve inventory, reserve a delivery slot, charge the card, then confirm. Reservations are holds with a 15-minute TTL, so they're cheap to compensate. Charging is the pivot; after it, confirming and emailing are retried until they succeed. The orchestrator runs on Temporal, so each step's result is persisted and a crash resumes the workflow; every step takes the order ID as its idempotency key."