Courses › System Design › Key Technologies

Temporal & Workflow Engines

Durable workflows, activities, retries, timers and orchestrated sagas.

Temporal is a durable workflow engine: you write a multi-step business process as ordinary code, and Temporal guarantees it runs to completion despite crashes, deploys and outages — resuming exactly where it left off, even days later. It's the open-source successor to Uber's Cadence; AWS Step Functions and Netflix Conductor solve the same problem. In interviews, it's a strong answer for sagas, long-running jobs, and any flow with retries, timeouts and human steps.

The problem it solves

An order flow might reserve inventory, charge a card, book shipping and email the customer. Done by hand, you need a state table, a queue, retry logic, timeout timers, and compensation when step 3 fails after steps 1 and 2 succeeded. A workflow engine provides all of that.

Core concepts

• Workflow — deterministic code that orchestrates the process: call activities, wait for signals, sleep for days, branch on results.
• Activity — a single side-effecting step (call the payment API, write to the database). Activities can fail and are retried according to a retry policy with backoff and timeouts.
• Worker — your service process that polls Temporal for workflow and activity tasks and runs your code.
• Event history — Temporal persists every step's result. If a worker dies, another worker replays the history to rebuild the workflow's state and continues from the next step.
• Signals and queries — send events into a running workflow (e.g. "user approved") and read its current state.
• Durable timers — sleep(7 days) costs nothing while waiting and survives restarts.

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order):
        await workflow.execute_activity(reserve_inventory, order, start_to_close_timeout=timedelta(seconds=30))
        try:
            await workflow.execute_activity(charge_card, order, start_to_close_timeout=timedelta(seconds=30))
        except ActivityError:
            await workflow.execute_activity(release_inventory, order, start_to_close_timeout=timedelta(seconds=30))
            raise
        await workflow.execute_activity(create_shipment, order, start_to_close_timeout=timedelta(minutes=5))

[!WARNING] Determinism and idempotency Workflow code is replayed, so it must be deterministic: no direct random numbers, clock reads or network calls — those go in activities. Activities are retried, so they must be idempotent: pass an idempotency key to the payment provider. See Safe Retries & Idempotency.

Orchestration vs choreography

See Coordinating Workflows with Sagas for the general pattern.

When to use it

• Sagas across services with compensation (orders, bookings, money movement)
• Long-running processes: onboarding, KYC reviews, subscription lifecycles, video pipelines
• Anything with "retry until it succeeds," "wait for approval," or "do X if Y hasn't happened within 24 hours"

When not to

• Simple fire-and-forget background jobs — a task queue such as SQS is lighter
• Very high-volume, low-latency event processing — use a stream processor like Flink

[!INTERVIEW] Hotel booking "Each booking is a Temporal workflow: hold the room, charge the deposit, confirm with the hotel's API, then send the confirmation. If the hotel doesn't confirm within 15 minutes, the workflow refunds the deposit and releases the hold. Every step is an idempotent activity with retries, and support staff can query any booking's workflow to see exactly where it is." See Managing Long-Running Tasks.