Skip to content

Saga Pattern#

The saga pattern handles distributed transactions across microservices without two-phase commit. A business operation is split into a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails.

sequenceDiagram
  participant C as Client
  participant O as Order Service
  participant P as Payment Service
  participant I as Inventory Service
  participant S as Shipping Service

  C->>O: place order
  O->>P: charge card
  P-->>O: charged
  O->>I: reserve stock
  I-->>O: reserved
  O->>S: schedule shipment
  S-->>O: scheduled
  O-->>C: order confirmed
  Note over O,S: any failure triggers compensations in reverse order

A traditional ACID transaction would lock rows in every service and commit them atomically. Microservices cannot afford that: they own separate databases, and global locks would destroy availability. The saga gives up atomicity and instead guarantees eventual consistency: either every step succeeds, or every successful step is reversed by its compensation.

Two coordination styles exist. Choreography has each service listen for events and emit its own; control flow is implicit in the event graph. Orchestration uses a central coordinator that calls each step explicitly and triggers compensations on failure. Choreography couples services through events, orchestration centralizes the workflow but adds a stateful coordinator. Production sagas almost always need idempotent steps, durable state, and a way to retry partial failures.

Problem statement (interviewer prompt)

Design an order checkout that spans payment, inventory, and shipping services, each with its own database, so the operation either fully completes or fully unwinds even when individual services fail or restart mid-flow.

Why sagas exist#

Two-phase commit (2PC) works inside one database vendor but falls apart across microservices: it requires every participant to hold locks until the coordinator decides, and a coordinator crash blocks every service. Most cloud datastores do not support 2PC at all. The saga pattern, first described by Garcia-Molina and Salem in 1987, trades atomicity for availability: each step commits its own local transaction immediately and publishes its outcome, and any later failure triggers explicit compensating transactions that semantically undo earlier work.

flowchart LR
  T1[Local Txn 1]
  T2[Local Txn 2]
  T3[Local Txn 3]
  C1[Compensate 1]
  C2[Compensate 2]

  T1 --> T2 --> T3
  T3 -. fail .-> C2
  C2 --> C1

    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class T1,T2,T3 compute;
    class C1,C2 datastore;

Choreography vs orchestration#

Choreography has no central brain. Each service subscribes to events, performs its work, and emits the next event. The system flow is implicit in the event topology.

sequenceDiagram
  participant O as Order Svc
  participant P as Payment Svc
  participant I as Inventory Svc
  participant Bus as Event Bus

  O->>Bus: OrderCreated
  Bus->>P: OrderCreated
  P->>Bus: PaymentCharged
  Bus->>I: PaymentCharged
  I->>Bus: StockReserved
  Bus->>O: StockReserved

Orchestration uses a coordinator (a workflow engine or a saga service) that owns the state machine, calls each participant, and decides on compensation.

sequenceDiagram
  participant Coord as Saga Orchestrator
  participant P as Payment
  participant I as Inventory
  participant S as Shipping

  Coord->>P: charge
  P-->>Coord: charged
  Coord->>I: reserve
  I-->>Coord: out of stock
  Coord->>P: refund (compensate)
  P-->>Coord: refunded
Aspect Choreography Orchestration
Coupling Event-shaped, implicit Coordinator-shaped, explicit
Visibility Hard, distributed trace Easy, one state machine
Adding a step Touch one publisher and one subscriber Touch the workflow
Coordinator outage None to fail Single point if not HA
Best for 2-3 step flows, autonomous teams 4+ step flows, complex compensations

Common orchestrators: AWS Step Functions, Temporal, Camunda Zeebe, Netflix Conductor, Azure Durable Functions.

Compensating actions#

A compensation is not a rollback. The local transaction has already committed and side effects may be visible. The compensation must semantically reverse the effect:

Forward action Compensation
Charge card $100 Refund $100 to the same card
Reserve seat 14A Release seat 14A
Send confirmation email Send cancellation email
Decrement inventory by 3 Increment inventory by 3

Some actions are not naturally reversible (sent an email, called an external API). Mitigations include pivot transactions (the last point at which the saga can still abort) and retriable transactions after the pivot, plus business-level apologies for actions that truly cannot be undone.

Idempotency and durable state#

Network retries are unavoidable, so every step must be idempotent: receiving the same command twice must not produce two charges or two reservations. Standard implementations:

  • Idempotency keys (UUID per saga step) stored in the participant's local DB
  • Outbox table on each service to publish events exactly once (see the outbox pattern)
  • Saga log persisted by the orchestrator before each transition
def charge_payment(saga_id: str, amount: int) -> ChargeResult:
    with db.transaction():
        row = db.upsert_idempotency(saga_id, "charge")
        if row.status == "done":
            return row.result
        result = payment_gateway.charge(saga_id, amount)
        row.status = "done"
        row.result = result
        db.save(row)
        outbox.publish("PaymentCharged", saga_id, result)
    return result

Failure modes and recovery#

Mode Effect Mitigation
Step times out Status unknown Query participant, then retry or compensate
Compensation fails Stuck saga Retry with backoff, alert humans after N
Out-of-order events Wrong state transition Version the saga state, drop stale events
Coordinator crash All in-flight sagas blocked Persist state, run coordinator HA
Participant data divergence Inventory says reserved, order says canceled Periodic reconciliation jobs

A saga is never truly isolated. Other transactions can observe partial state. Add semantic locks (status fields like RESERVED) or commutative updates (counters) to keep concurrent business logic correct.

Production gotchas#

  • Compensations are first-class code, not afterthoughts. Test them with chaos drills.
  • Time bounds matter: a saga that waits forever for a participant blocks resources. Use TTLs and dead-letter to humans.
  • Trace IDs propagate through every event so distributed tracing reconstructs the flow.
  • Versioning the saga definition is mandatory; in-flight sagas must finish on their original version even if new ones use v2.
  • Avoid long-running sagas that hold business-visible state in PENDING for hours. Split them into smaller sagas.
  • External calls in compensations (refunds, emails) can themselves fail. Build a retry plus dead-letter playbook.

Quick reference#

Core idea#

  • Replace 2PC with a sequence of local transactions
  • Each step has a compensating action that reverses it
  • All-or-nothing becomes eventually-consistent

Coordination styles#

  • Choreography: services react to events, no central brain
  • Orchestration: coordinator drives the workflow explicitly
  • Choreography fits 2-3 steps, orchestration fits 4 plus

Compensations#

  • Not a rollback, a semantic inverse
  • Charge becomes refund, reserve becomes release
  • Some side effects (sent email) are unreversible: use pivot + retriable phases

Must-have properties#

  • Idempotency keys on every step
  • Durable saga state (DB row or workflow engine)
  • Outbox for reliable event publication
  • Trace id on every message

Failure modes#

  • Step timeout with unknown outcome, query then decide
  • Compensation that itself fails, retry with backoff then alert
  • Coordinator crash, run HA and persist state
  • Out-of-order events, version saga state
  • Stuck saga in PENDING for hours, add TTL and DLQ

Tools#

  • AWS Step Functions, Temporal, Camunda Zeebe
  • Netflix Conductor, Azure Durable Functions
  • Hand-rolled state machine in a service plus outbox

When to use#

  • Long-running business processes across services
  • No single database can host the transaction
  • Eventual consistency is acceptable

When to skip#

  • All steps live in one DB: use a local transaction
  • Real-time consistency is required: redesign or accept 2PC cost

Refs#

  • Garcia-Molina, Salem: "Sagas" (1987)
  • Chris Richardson: "Microservices Patterns" (2018)
  • Temporal docs: "Workflows and activities" (2022)

FAQ#

What problem does the saga pattern solve?#

It coordinates a business operation that spans multiple services without distributed two-phase commit. Each step commits locally and any failure triggers compensating actions to undo earlier steps.

What is the difference between choreography and orchestration sagas?#

Choreography lets each service react to events from peers with no central coordinator. Orchestration uses one orchestrator that calls each service and decides the next step or rollback.

When should I avoid the saga pattern?#

Avoid it for single-database operations where a local transaction suffices, and for flows requiring strict isolation, since intermediate saga states are visible between steps.

How do compensating transactions work?#

Each forward step has an inverse step. If step three fails, the orchestrator runs the compensations for steps two and one in reverse to bring the system back to a safe state.

Why must saga steps be idempotent?#

Retries, replays, and message redelivery can cause the same step to execute more than once. Idempotency keys or natural deduplication prevent double charges and duplicate orders.

Video walkthrough

Saga Pattern - Mastering Distributed Transactions : via ByteByteGo