How to Implement the Saga Pattern in Python With Compensating Transactions

Picture a trip-booking app that reserves a flight, reserves a hotel room, and then charges a credit card, each of those three things living in its own service with its own database. Reserving the flight works. Reserving the hotel works. Then the card gets declined. Now what? There is no single database transaction you can roll back, because there was never one transaction to begin with: three separate services each committed their own local write, and two of them already succeeded. If nothing else happens, that customer is left holding a seat and a hotel room they never paid for, forever, until someone notices.

The Saga pattern is the standard answer to that problem. Instead of one big transaction spanning three services (which is not actually available to you here, more on that below), you write the business operation as a sequence of small local transactions, and for every one of them you also write its opposite: a compensating transaction that undoes it. If step three fails, you do not try to roll anything back in the database sense. You run the compensations for steps one and two, in reverse order, and you accept that the failed step never happened at all.

In this tutorial you will build a real Saga orchestrator in Python from scratch: three independent SQLite-backed services standing in for three real microservices, a small engine that runs steps and compensations, a genuine bug where a compensation running twice quietly corrupts your data, the fix for that bug, and a crash-recovery routine that can pick a saga back up after the orchestrator process itself dies mid-flight. Every script below was actually run in this sandbox, and every number you see in an output block came from that run, not from a docstring.

What a Saga Is, and Why 2PC Is Not the Answer Here

If you have used a relational database, you already know what a local transaction gives you: run BEGIN, do some writes, run COMMIT, and either every one of those writes lands or none of them do, even across a crash. That guarantee stops at the edge of one database. The moment your flight reservation, your hotel reservation, and your card charge live in three separate databases owned by three separate services (the normal shape of a microservice architecture), no single COMMIT can cover all three.

You might reach for two-phase commit here, the classic protocol for getting several databases to agree on one all-or-nothing outcome. sxz.io already has a from-scratch tutorial on building 2PC in Python, and it is worth reading for contrast, because the canonical description of the Saga pattern on Chris Richardson’s microservices.io lists exactly one force driving you toward sagas instead: "2PC is not an option." In practice, 2PC needs every participant to hold locks and stay reachable for the whole duration of the transaction, which does not survive contact with independent services, mobile clients, human approval steps, or anything that might be slow or temporarily offline. Sagas trade that tight coordination away on purpose.

Here is the definition worth memorizing, quoted directly from microservices.io: "A saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule then the saga executes a series of compensating transactions that undo the changes that were made by the preceding local transactions."

A few terms that will come up constantly:

• Local transaction: an ordinary, single-database transaction, the kind that has always worked the normal way. Reserving a flight in the flights database is a local transaction.
• Compensating transaction: a second local transaction, written by you, whose entire job is to semantically undo an earlier local transaction. Cancelling a flight reservation is the compensating transaction for reserving it.
• Orchestration: one coordinator object knows the full list of steps and tells each participant what to do next, then waits for the answer. This is what you will build below.
• Choreography: there is no central coordinator; each service reacts to events published by the others. AWS Prescriptive Guidance’s Saga patterns page describes it this way: "The saga choreography pattern depends on the events published by the microservices. The saga participants (microservices) subscribe to the events and act based on the event triggers." You will see this pattern in prose near the end, without code, and there is a reason for that choice.

The name itself is older than microservices by decades. Garcia-Molina and Salem introduced the term in a paper simply titled "Sagas," presented at SIGMOD in 1987 (the archival record is still up at sigmodrecord.org), to describe long-lived transactions built from smaller steps with compensations for the ones that fail. Chris Richardson later revived the term for microservices, and the underlying idea has not changed: give up on one giant transaction, and instead build a sequence you can walk backward.

Prerequisites

• Python 3.10 or newer (this tutorial was built and run on Python 3.13.14), using only the standard library's sqlite3 module for every service. No packages to install for the core pattern.
• pytest for the verification suite at the end (pip install pytest).
• Basic SQL: you should be comfortable reading a SELECT and an UPDATE.
• No admin rights, no cloud account, and no message broker needed. Every "service" below is a plain Python module backed by its own local SQLite file, which is enough to demonstrate every real problem a saga has to solve.

Step 1: Book a Trip Across Three Services, the Naive Way

Start with three tiny services, each owning its own SQLite database, exactly like three real microservices would each own their own database. A flights service that can reserve and cancel a seat:

import sqlite3
from pathlib import Path

DB_PATH = Path(__file__).parent / "flights.db"


def _connect():
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS seat_inventory ("
        "  flight_no TEXT PRIMARY KEY,"
        "  seats_available INTEGER NOT NULL"
        ")"
    )
    conn.execute(
        "CREATE TABLE IF NOT EXISTS reservations ("
        "  trip_id TEXT PRIMARY KEY,"
        "  flight_no TEXT NOT NULL,"
        "  status TEXT NOT NULL"
        ")"
    )
    conn.commit()
    return conn


def seed(flight_no, seats):
    conn = _connect()
    conn.execute(
        "INSERT OR REPLACE INTO seat_inventory (flight_no, seats_available) VALUES (?, ?)",
        (flight_no, seats),
    )
    conn.commit()
    conn.close()


def reserve_flight(trip_id, flight_no):
    conn = _connect()
    try:
        cur = conn.execute(
            "SELECT seats_available FROM seat_inventory WHERE flight_no = ?",
            (flight_no,),
        )
        row = cur.fetchone()
        if row is None or row[0] <= 0:
            raise RuntimeError(f"No seats available on {flight_no}")
        conn.execute(
            "UPDATE seat_inventory SET seats_available = seats_available - 1 WHERE flight_no = ?",
            (flight_no,),
        )
        conn.execute(
            "INSERT INTO reservations (trip_id, flight_no, status) VALUES (?, ?, 'RESERVED')",
            (trip_id, flight_no),
        )
        conn.commit()
    finally:
        conn.close()


def cancel_flight(trip_id):
    conn = _connect()
    try:
        cur = conn.execute(
            "SELECT flight_no FROM reservations WHERE trip_id = ?", (trip_id,)
        )
        row = cur.fetchone()
        if row is None:
            return
        flight_no = row[0]
        conn.execute(
            "UPDATE seat_inventory SET seats_available = seats_available + 1 WHERE flight_no = ?",
            (flight_no,),
        )
        conn.commit()
    finally:
        conn.close()


def seats_available(flight_no):
    conn = _connect()
    try:
        cur = conn.execute(
            "SELECT seats_available FROM seat_inventory WHERE flight_no = ?",
            (flight_no,),
        )
        row = cur.fetchone()
        return row[0] if row else 0
    finally:
        conn.close()


def reservation_status(trip_id):
    conn = _connect()
    try:
        cur = conn.execute(
            "SELECT flight_no, status FROM reservations WHERE trip_id = ?", (trip_id,)
        )
        row = cur.fetchone()
        return {"flight_no": row[0], "status": row[1]} if row else None
    finally:
        conn.close()

Save that as services/flights.py. Build services/hotels.py the same way, swapping flight_no and seats_available for hotel_id and rooms_available, and services/cards.py with a charge_card(trip_id, card_id, amount_cents) function that raises if the card is in a small DECLINED_CARDS set, plus a matching refund_card(trip_id). Each of the three keeps its own SQLite file, so nothing here is faked.

Now book two trips with a naive script that just calls all three services one after another, with no safety net at all:

from services import flights, hotels, cards


def book_trip_naively(trip_id, card_id):
    print(f"[{trip_id}] reserving flight FL-100 ...")
    flights.reserve_flight(trip_id, "FL-100")
    print(f"[{trip_id}] flight reserved.")

    print(f"[{trip_id}] reserving hotel HTL-9 ...")
    hotels.reserve_hotel(trip_id, "HTL-9")
    print(f"[{trip_id}] hotel reserved.")

    print(f"[{trip_id}] charging card {card_id} ...")
    cards.charge_card(trip_id, card_id, amount_cents=45000)
    print(f"[{trip_id}] card charged. Trip booked!")


if __name__ == "__main__":
    flights.seed("FL-100", seats=5)
    hotels.seed("HTL-9", rooms=5)

    book_trip_naively("trip-naive-1", "card-good")

    try:
        book_trip_naively("trip-naive-2", "card-declined")
    except RuntimeError as exc:
        print(f"[trip-naive-2] booking FAILED: {exc}")

    print("flight reservation:", flights.reservation_status("trip-naive-2"))
    print("hotel reservation: ", hotels.reservation_status("trip-naive-2"))
    print("charge:            ", cards.charge_status("trip-naive-2"))
    print("seats left on FL-100:", flights.seats_available("FL-100"))
    print("rooms left at HTL-9: ", hotels.rooms_available("HTL-9"))

Running python step1_naive_booking.py produces this, exactly as captured:

[trip-naive-2] reserving flight FL-100 ...
[trip-naive-2] flight reserved.
[trip-naive-2] reserving hotel HTL-9 ...
[trip-naive-2] hotel reserved.
[trip-naive-2] charging card card-declined ...
[trip-naive-2] booking FAILED: Card card-declined was declined

flight reservation: {'flight_no': 'FL-100', 'status': 'RESERVED'}
hotel reservation:  {'hotel_id': 'HTL-9', 'status': 'RESERVED'}
charge:             None
seats left on FL-100 (started at 5, only 1 trip should have used one): 3
rooms left at HTL-9 (started at 5, only 1 trip should have used one): 3

Read that carefully. The card decline is handled fine as an exception, but nothing ever tells the flight or hotel services that the trip failed. Both reservations sit there marked RESERVED forever, and inventory for both services is down by two seats and two rooms when only one trip actually completed. That seat and that room are gone until a human notices and fixes the data by hand. This is the exact failure a saga exists to close.

Step 2: Wrap the Booking in a Saga

The fix is to stop calling the three services directly, and instead describe the booking as a list of steps, where every step carries its own compensation right next to it:

import sqlite3
from dataclasses import dataclass
from pathlib import Path
from typing import Callable

DB_PATH = Path(__file__).parent / "saga_log.db"


@dataclass
class SagaStep:
    name: str
    action: Callable[[], None]
    compensation: Callable[[], None]


class SagaFailed(Exception):
    def __init__(self, failed_step, cause):
        super().__init__(f"saga failed at step '{failed_step}': {cause}")
        self.failed_step = failed_step
        self.cause = cause


def _connect():
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS saga_log ("
        "  saga_id TEXT NOT NULL,"
        "  step_index INTEGER NOT NULL,"
        "  step_name TEXT NOT NULL,"
        "  status TEXT NOT NULL,"
        "  PRIMARY KEY (saga_id, step_index)"
        ")"
    )
    conn.commit()
    return conn


def _log(saga_id, step_index, step_name, status):
    conn = _connect()
    try:
        conn.execute(
            "INSERT INTO saga_log (saga_id, step_index, step_name, status) "
            "VALUES (?, ?, ?, ?) "
            "ON CONFLICT(saga_id, step_index) DO UPDATE SET status = excluded.status",
            (saga_id, step_index, step_name, status),
        )
        conn.commit()
    finally:
        conn.close()


def _compensate_indexes(saga_id, steps, indexes_desc):
    for index in indexes_desc:
        step = steps[index]
        _log(saga_id, index, step.name, "COMPENSATING")
        step.compensation()
        _log(saga_id, index, step.name, "COMPENSATED")


class SagaOrchestrator:
    def __init__(self, saga_id, steps):
        self.saga_id = saga_id
        self.steps = steps

    def run(self):
        completed = []
        for index, step in enumerate(self.steps):
            _log(self.saga_id, index, step.name, "PENDING")
            try:
                step.action()
            except Exception as exc:
                _log(self.saga_id, index, step.name, "FAILED")
                _compensate_indexes(self.saga_id, self.steps, list(reversed(completed)))
                raise SagaFailed(step.name, exc) from exc
            else:
                _log(self.saga_id, index, step.name, "DONE")
                completed.append(index)

That is the whole engine for now. run() walks the steps in order. The moment one of them raises, it compensates everything that already succeeded, strictly in reverse order, using the completed list it has been building as it goes. Every transition also gets written to a small saga_log table before and after each call. That log looks unnecessary right now; Step 4 is the reason it exists.

Save that as saga.py, then describe the actual trip-booking saga as data instead of as a hardcoded sequence of calls:

from services import flights, hotels, cards
from saga import SagaOrchestrator, SagaStep, SagaFailed


def build_trip_saga(trip_id, card_id):
    steps = [
        SagaStep(
            name="reserve_flight",
            action=lambda: flights.reserve_flight(trip_id, "FL-100"),
            compensation=lambda: flights.cancel_flight(trip_id),
        ),
        SagaStep(
            name="reserve_hotel",
            action=lambda: hotels.reserve_hotel(trip_id, "HTL-9"),
            compensation=lambda: hotels.cancel_hotel(trip_id),
        ),
        SagaStep(
            name="charge_card",
            action=lambda: cards.charge_card(trip_id, card_id, amount_cents=45000),
            compensation=lambda: cards.refund_card(trip_id),
        ),
    ]
    return SagaOrchestrator(saga_id=trip_id, steps=steps)

Run the same declined-card scenario through the orchestrator instead of calling the services directly:

build_trip_saga("trip-saga-1", "card-good").run()

try:
    build_trip_saga("trip-saga-2", "card-declined").run()
except SagaFailed as exc:
    print(f"saga failed as expected: {exc}")

The captured output:

saga failed as expected: saga failed at step 'charge_card': Card card-declined was declined

flight reservation: {'flight_no': 'FL-100', 'status': 'RESERVED'}
hotel reservation:  {'hotel_id': 'HTL-9', 'status': 'RESERVED'}
charge:             None
seats left on FL-100 (should be back to 4, only trip-saga-1 used one): 4
rooms left at HTL-9 (should be back to 4, only trip-saga-1 used one): 4

The inventory counts are correct this time: only trip-saga-1's real seat and room are gone, the declined trip left nothing behind. Notice that the reservation status still says RESERVED, though, even though the seat and room really were given back. That is not a typo. The first cut of cancel_flight and cancel_hotel gives the resource back but never touches the reservation row itself, and that shortcut is about to bite.

Step 3: The Idempotency Trap

A real orchestrator does not get the luxury of assuming every call it makes lands exactly once. A network call to the hotel service can time out with the cancellation already applied on the far side; the orchestrator, not knowing that, retries. AWS's own pattern library keeps retry with backoff right next to its saga pattern entry, and its own guidance on that page is direct: "Operations should be idempotent when you use the retry with backoff pattern. Otherwise, partial updates might corrupt the system state." (sxz.io has its own from-scratch tutorial on that pattern if you want the full backoff-and-jitter mechanics.) Whatever triggers it, imagine a retry loop that calls cancel_hotel(trip_id) a second and third time for the same trip, unsure whether the first attempt actually succeeded:

rooms after reserving 1 room out of 5: 4
rooms after the saga's own (first) compensation: 5
rooms after retry #1: 6
rooms after retry #2: 7

That is a real, captured bug, not a hypothetical one: the hotel now claims more rooms are available than it has beds, because the first version of cancel_hotel checks only whether a reservation row exists for the trip, credits the room back, and stops. It never records that the cancellation happened, so the exact same call run twice more finds the exact same row and credits the room back twice more. Every retry the orchestrator thinks is a harmless safety net quietly inflates inventory.

The fix is to make the compensation check and record its own completion, so a second call is a safe no-op:

def cancel_hotel(trip_id):
    """Idempotent: only the first call for a given trip_id actually
    credits the room back. Returns True if this call did the crediting,
    False if it was a no-op."""
    conn = _connect()
    try:
        cur = conn.execute(
            "SELECT hotel_id, status FROM reservations WHERE trip_id = ?",
            (trip_id,),
        )
        row = cur.fetchone()
        if row is None or row[1] != "RESERVED":
            return False
        hotel_id = row[0]
        conn.execute(
            "UPDATE room_inventory SET rooms_available = rooms_available + 1 WHERE hotel_id = ?",
            (hotel_id,),
        )
        conn.execute(
            "UPDATE reservations SET status = 'CANCELLED' WHERE trip_id = ?",
            (trip_id,),
        )
        conn.commit()
        return True
    finally:
        conn.close()

The change is one extra column check and one extra UPDATE: only credit the room, and only flip the status, if the row is still RESERVED. Apply the identical fix to cancel_flight and refund_card. Re-run the exact same over-eager retry scenario against the fixed version:

rooms at HTL-9 right after the saga's own compensation: 5

retry #1: did_credit=False, rooms now = 5
retry #2: did_credit=False, rooms now = 5

FIXED: both retries were safe no-ops. rooms_available never exceeds 5.

And re-running Step 2's scenario against the fixed services shows the earlier stale status is gone too: the reservation now correctly reports status: 'CANCELLED' instead of sitting at RESERVED forever. That was the same shortcut showing up twice: skipping the status update did not just enable double-crediting, it also lied about the trip's own state to anyone who asked.

Why This Matters More for Sagas Than for 2PC

Two-phase commit gets to hold a lock on every participant for the duration of the transaction, so a coordinator basically never has to guess whether a commit landed twice. A saga's compensations run against services that keep taking normal traffic the whole time, over an ordinary, at-least-once network call. If your compensations are not idempotent, retries (which you need, because networks fail) turn into corruption (which you did not want). Idempotent compensations are not a nice-to-have here; they are the thing that makes retrying safe at all.

Step 4: Surviving a Crash With a Persisted Saga Log

The orchestrator itself is just a process, and processes get killed: a deploy, an out-of-memory event, a hardware fault. If it dies mid-saga, something has to be able to pick the saga back up later, and it has to be able to tell the difference between "keep going forward" and "finish rolling back." That is exactly what the saga_log table from Step 2 is for. Extend saga.py with a recovery function that reads the log and decides:

def recover_saga(saga_id, steps):
    log = load_log(saga_id)
    if not log:
        return f"no saga log entries found for '{saga_id}'; nothing to recover"

    by_index = {entry["step_index"]: entry for entry in log}
    max_index = max(by_index)
    last_status = by_index[max_index]["status"]
    any_failed = any(entry["status"] == "FAILED" for entry in log)

    if not any_failed and last_status == "DONE":
        if max_index == len(steps) - 1:
            return f"saga '{saga_id}' had already completed successfully; nothing to recover"
        resume_from = max_index + 1
        completed = list(range(resume_from))
        print(f"  no failure was ever logged, and step {max_index} "
              f"('{steps[max_index].name}') finished cleanly.")
        print(f"  resuming forward from step {resume_from} ('{steps[resume_from].name}').")
        for index in range(resume_from, len(steps)):
            step = steps[index]
            _log(saga_id, index, step.name, "PENDING")
            try:
                step.action()
            except Exception as exc:
                _log(saga_id, index, step.name, "FAILED")
                _compensate_indexes(saga_id, steps, list(reversed(completed)))
                raise SagaFailed(step.name, exc) from exc
            else:
                _log(saga_id, index, step.name, "DONE")
                completed.append(index)
        return f"resumed '{saga_id}' forward and completed it"

    # Either a step definitely failed, or the last thing the log shows is
    # a step still marked PENDING or COMPENSATING: the crash happened
    # *during* that step, and we cannot tell whether its action (or
    # compensation) actually took effect before the process died. The
    # safe move in both cases is to (re-)run compensation for every step
    # that is DONE, PENDING, or COMPENSATING. Because every compensation
    # here is idempotent (Step 3), re-running one that already succeeded,
    # or one whose action never actually took effect, is always harmless.
    needs_compensation = sorted(
        (
            index
            for index, entry in by_index.items()
            if entry["status"] in ("DONE", "PENDING", "COMPENSATING")
        ),
        reverse=True,
    )
    print("  a failure was logged, or a step's outcome was left uncertain by the crash.")
    print(f"  finishing the rollback for step index(es) {needs_compensation}, high to low.")
    _compensate_indexes(saga_id, steps, needs_compensation)
    return f"finished rolling back '{saga_id}'"

Read the decision logic once more, slowly, because it is the whole point of this section. If nothing ever failed and the last step we know about finished cleanly, the crash happened in a quiet moment between two steps, and it is safe to just keep going forward from the next one. Anything else, a step that definitely failed, or a step whose own outcome is unknown because the crash landed in the middle of running it, gets treated the same way: compensate it and everything before it. That second branch only works because of the idempotency fix in Step 3. Re-running a compensation for a step that never actually completed its action is fine (there is nothing to undo, so it is a no-op); re-running one that did complete, but whose completion never made it into the log, is also fine (idempotent by design). Recovery gets to be simple precisely because it does not have to be certain.

You cannot easily kill a real process for a blog post, so simulate the crash instead: add an optional crash_after_step argument to SagaOrchestrator.run() that raises a plain SagaCrashed exception right after the chosen step is logged DONE, before the next one starts. A real crash would not raise a Python exception at all, the process would simply stop, but raising here lets the demo scripts show exactly what was and was not written to the log at the moment things went wrong:

class SagaCrashed(Exception):
    pass


class SagaOrchestrator:
    ...
    def run(self, crash_after_step=None, on_step_done=None):
        completed = []
        for index, step in enumerate(self.steps):
            _log(self.saga_id, index, step.name, "PENDING")
            try:
                step.action()
            except Exception as exc:
                _log(self.saga_id, index, step.name, "FAILED")
                _compensate_indexes(self.saga_id, self.steps, list(reversed(completed)))
                raise SagaFailed(step.name, exc) from exc
            else:
                _log(self.saga_id, index, step.name, "DONE")
                completed.append(index)
                if on_step_done is not None:
                    on_step_done(index)
                if crash_after_step == index:
                    raise SagaCrashed(
                        f"simulated crash right after step {index} ('{step.name}') committed"
                    )

The on_step_done callback is not for crash testing, Step 5 uses it to simulate a concurrent reader checking the trip's status right after each step commits. Scenario B below needs one more variant, a method that raises partway through the reverse compensation loop instead of the forward loop; it is a near-duplicate of run() above with the crash point moved into the compensation sweep, so it is not repeated line for line here.

Scenario A: Crash Right After a Step Commits

Simulate the orchestrator dying immediately after reserve_flight succeeds, before reserve_hotel even starts:

*** SIMULATED CRASH: simulated crash right after step 0 ('reserve_flight') committed ***

saga_log.db on disk right now:
  {'step_index': 0, 'step_name': 'reserve_flight', 'status': 'DONE'}

=== A fresh orchestrator process starts up and calls recover_saga() ===
  no failure was ever logged, and step 0 ('reserve_flight') finished cleanly.
  resuming forward from step 1 ('reserve_hotel').
recovery result: resumed 'trip-crash-fwd' forward and completed it

=== Final state ===
flight reservation: {'flight_no': 'FL-100', 'status': 'RESERVED'}
hotel reservation:  {'hotel_id': 'HTL-9', 'status': 'RESERVED'}
charge:             {'amount_cents': 45000, 'status': 'CHARGED'}

A brand-new call to recover_saga, with no other state than the log on disk, correctly figures out that step 0 is the only thing that happened, and finishes the saga forward from step 1. The trip books successfully, exactly as if nothing had gone wrong.

Scenario B: Crash in the Middle of a Rollback

This is the scenario that actually needs the idempotency fix. Book a trip with the declining card, so the saga fails and starts compensating, and simulate a crash right after cancel_hotel really runs, but before the log gets to record that it finished:

*** SIMULATED CRASH: simulated crash right after compensating 'reserve_hotel', before the COMPENSATED status was written to the log ***

saga_log.db on disk right now:
  {'step_index': 0, 'step_name': 'reserve_flight', 'status': 'DONE'}
  {'step_index': 1, 'step_name': 'reserve_hotel', 'status': 'COMPENSATING'}
  {'step_index': 2, 'step_name': 'charge_card', 'status': 'FAILED'}

real hotel room count already reflects the compensation that ran:
 rooms at HTL-9: 5 (out of 5 -- the cancel_hotel() call
  itself already succeeded, the crash only stopped the log write)
but the flight is still sitting reserved, since we never got that far:
 flight reservation: {'flight_no': 'FL-100', 'status': 'RESERVED'}

=== A fresh orchestrator process starts up and calls recover_saga() ===
  a failure was logged, or a step's outcome was left uncertain by the crash.
  finishing the rollback for step index(es) [1, 0], high to low.
recovery result: finished rolling back 'trip-crash-rollback'

=== Final state ===
flight reservation: {'flight_no': 'FL-100', 'status': 'CANCELLED'}
hotel reservation:  {'hotel_id': 'HTL-9', 'status': 'CANCELLED'}
charge:             None
seats left on FL-100 (should be back to 5): 5
rooms left at HTL-9 (should be back to 5): 5

Look at what recovery actually does here: it sees step 1 still marked COMPENSATING, and, having no way to know that cancel_hotel already truly succeeded, calls it again anyway. Because of Step 3's fix, that second call is a safe no-op, the room count stays at 5, and recovery moves on to compensate step 0 (the flight), which really had never been touched yet. The saga finishes in a fully consistent state either way. Without the idempotency fix, this exact recovery path would have pushed the room count to 6.

Step 5: What You Give Up: Isolation

Sagas buy you a way to fail safely, but they do not buy back the isolation a single ACID transaction gives you for free. microservices.io names this directly as a drawback of the whole pattern: sagas lack isolation in the ACID sense, and it says the concurrent execution of multiple sagas can produce data anomalies that a saga developer has to guard against with countermeasures, its own term for design techniques that implement isolation by hand. Prove it with your own reader function that queries all three services directly, standing in for anything else in your system that might look at a trip mid-flight, a customer-facing status page, an internal dashboard, anything:

def get_trip_status(trip_id):
    flight = flights.reservation_status(trip_id)
    hotel = hotels.reservation_status(trip_id)
    charge = cards.charge_status(trip_id)
    return {
        "flight": flight["status"] if flight else "not booked yet",
        "hotel": hotel["status"] if hotel else "not booked yet",
        "payment": charge["status"] if charge else "not charged yet",
    }

Book a trip with the declining card again, but this time call get_trip_status after every step commits, simulating a concurrent reader checking in along the way:

  [concurrent reader, right after step 0] {'flight': 'RESERVED', 'hotel': 'not booked yet', 'payment': 'not charged yet'}
  [concurrent reader, right after step 1] {'flight': 'RESERVED', 'hotel': 'RESERVED', 'payment': 'not charged yet'}
saga failed and rolled back: saga failed at step 'charge_card': Card card-declined was declined

=== What the saga's own final, fully-compensated state looks like ===
  {'flight': 'CANCELLED', 'hotel': 'CANCELLED', 'payment': 'not charged yet'}

Both of those intermediate snapshots were real and accurate at the moment they were taken. They were also completely wrong by the time the saga finished a few milliseconds later. If that reader had been a page rendered for a customer, or a confirmation email queued for delivery, it would have shown a booking that no longer exists. A single ACID transaction would never let a reader see that halfway state at all; a saga hands it to you on a plate. The usual countermeasures are things like showing the trip as a distinct, honest "PENDING" state rather than reusing the participants' own per-service statuses, or simply not reading (or acting on) any state until the saga as a whole reports done. Which countermeasure fits depends entirely on what the reader is going to do with the information, so this tutorial stops at demonstrating the problem rather than picking one for you.

Choreography vs Orchestration: Which Should You Use?

Everything above is orchestration: one object holds the list of steps and drives them. AWS Prescriptive Guidance's own comparison is a good, short way to decide between the two styles. On choreography: "The saga choreography pattern is suitable when there are only a few participants in the saga, and you need a simple implementation with no single point of failure. When more participants are added, it becomes harder to track the dependencies between the participants by using this pattern." On orchestration: "The saga orchestration pattern is suitable when there are many participants, and loose coupling is required between saga participants... However, the orchestrator can become a single point of failure because it controls the entire workflow."

For a three-step trip booking, either style would work. Orchestration was the better choice for this tutorial specifically because it gives you one obvious place to put the saga log, which is what makes Step 4's crash recovery possible with so little code. A choreography-based version of the same saga is entirely possible (each service publishes an event when it finishes, the next service subscribes and reacts), but recovering a choreography saga after a crash means reconstructing what happened from events scattered across every participant's own log instead of reading one table, which is a meaningfully harder problem to solve well. That tradeoff, not just style preference, is usually what should decide it for you.

Common Mistakes and Gotchas

• Non-idempotent compensations. This tutorial's whole Step 3 exists because it is the single easiest way to quietly corrupt data in a saga. If a compensation can be called more than once (and in a real distributed system, it can), it needs to check whether it has already run.
• Assuming a compensation always succeeds. A compensating transaction is still a call to a real service, and real services can be down, slow, or wrong. A production saga needs its own retry policy around compensations (see sxz.io's retry with exponential backoff and jitter tutorial), and a way to alert a human when a compensation keeps failing rather than retrying forever.
• No persisted log. Without something like saga_log, an orchestrator crash just loses the saga's state completely; there is nothing left to recover from. sxz.io's Transactional Outbox tutorial covers a closely related problem: reliably getting a database write and a message out together in the first place.
• Compensating in the wrong order. Compensations must run in the reverse of the order the actions ran in, not in the saga's original forward order and not in an arbitrary order. A payment refund that fires before an inventory release can violate assumptions the inventory service makes about payment state.
• Treating the isolation gap as someone else's problem. Step 5 is not a corner case; it is guaranteed to happen under enough load. Decide up front what any reader of in-flight saga state is allowed to assume.

How to Verify Everything Works

A short pytest suite locks in every behavior demonstrated above: idempotent compensations, correct reverse-order rollback on failure, forward recovery after a clean crash, and finishing a rollback after a crash mid-compensation.

@pytest.fixture(autouse=True)
def fresh_databases():
    for db_file in ("flights.db", "hotels.db", "cards.db"):
        (Path(__file__).parent / "services" / db_file).unlink(missing_ok=True)
    (Path(__file__).parent / "saga_log.db").unlink(missing_ok=True)
    seed_inventory()
    yield


def test_cancel_hotel_is_idempotent():
    hotels.reserve_hotel("t1", "HTL-9")
    first = hotels.cancel_hotel("t1")
    second = hotels.cancel_hotel("t1")
    third = hotels.cancel_hotel("t1")
    assert (first, second, third) == (True, False, False)
    assert hotels.rooms_available("HTL-9") == 5  # not 7


def test_recovery_finishes_rollback_after_a_crash_mid_compensation():
    trip_id = "t-recover-rollback"
    orchestrator = build_trip_saga(trip_id, "card-declined")
    with pytest.raises(SagaCrashed):
        orchestrator.run_and_crash_during_compensation(crash_after_compensating="reserve_hotel")

    result = recover_saga(trip_id, orchestrator.steps)
    assert "rolling back" in result
    assert flights.seats_available("FL-100") == 5
    assert hotels.rooms_available("HTL-9") == 5

Running pytest test_saga.py -v against the finished code:

test_saga.py::test_reserve_then_cancel_flight_frees_the_seat PASSED      [ 11%]
test_saga.py::test_cancel_flight_is_idempotent PASSED                    [ 22%]
test_saga.py::test_cancel_hotel_is_idempotent PASSED                     [ 33%]
test_saga.py::test_refund_card_is_idempotent PASSED                      [ 44%]
test_saga.py::test_saga_happy_path_books_everything PASSED               [ 55%]
test_saga.py::test_saga_failure_compensates_everything_and_leaves_no_orphans PASSED [ 66%]
test_saga.py::test_recovery_resumes_forward_after_a_clean_crash PASSED   [ 77%]
test_saga.py::test_recovery_finishes_rollback_after_a_crash_mid_compensation PASSED [ 88%]
test_saga.py::test_already_completed_saga_recovers_as_a_no_op PASSED     [100%]

9 passed in 0.94s

To confirm the whole thing end to end yourself: run step1_naive_booking.py first and see the orphaned reservations, then run step2_saga_booking.py and confirm inventory is clean after a failure, then edit the three service files to add the idempotency check from Step 3 and re-run step2_saga_booking.py again to see the reservation status correctly flip to CANCELLED. From there, step4a and step4b exercise the two crash-recovery paths, and the pytest suite pins every one of those behaviors down so a future change cannot silently break them.

Next Steps

The Saga pattern is one of several ways sxz.io has covered keeping data consistent when a single transaction is not available to you. If you built this tutorial, the natural next reads are the Two-Phase Commit tutorial for the case where you genuinely can hold locks across participants, the Transactional Outbox tutorial for reliably publishing an event alongside a single local write, the retry with backoff tutorial for the mechanics behind Step 3's retry scenario, and the circuit breaker tutorial for what to do when a dependency a saga step calls is not just failing once, but down entirely.