How to Build a Bulkhead in Python to Stop a Slow Dependency From Starving the Rest

If your application calls more than one downstream dependency, and almost every real application does, then all of those calls are competing for the same finite resources: thread pool workers, database connections, socket handles. Most of the time that is invisible, because most of the time every dependency responds quickly. The problem shows up the moment exactly one dependency gets slow, not down, just slow, and the resources every other dependency also needed are suddenly all tied up waiting on it. A checkout page that only needs a fast inventory lookup can end up hanging for seconds because a third-party shipping-rate API it does not even care about is degraded, and both calls happened to share the same pool of worker threads.

The fix is called the Bulkhead pattern, and the name is not a metaphor invented for software. It is borrowed directly from ship design. Microsoft’s Azure Architecture Center explains it this way: This pattern is named after the sectioned partitions (bulkheads) of a ship’s hull. If the hull of a ship is compromised, only the damaged section fills with water, which prevents the ship from sinking. A bulkhead in code does the same job: it partitions a shared resource (a thread pool, a connection pool, a semaphore) into isolated compartments, one per dependency, so that one compartment flooding with slow calls cannot flood the compartments next to it.

This tutorial builds that isolation from scratch in Python, no framework required beyond a small test server. You will reproduce a real case of one slow dependency starving a fast one, measure exactly how bad it gets, fix it two different ways (a dedicated thread pool per dependency, and a semaphore-based bulkhead that fails fast instead of queuing forever), and then carry the same idea into async code, where the standard-library building block changes and getting it wrong silently freezes your entire event loop. Every number in this post is real output from code that was actually run while writing it, including two genuine measurement bugs this tutorial’s own author walked into and had to fix.

What You Will Build

• A tiny FastAPI service simulating two downstream dependencies: a healthy, fast inventory lookup and a currently degraded, slow shipping-rate lookup
• A naive client that shares one thread pool across both dependencies, and a reproducible demonstration of the starvation this causes
• A bulkhead built as dedicated thread pools, one per dependency, and proof that it fixes the starvation
• A second bulkhead implementation, SemaphoreBulkhead, modeled on the real design used by the Java library resilience4j, including its fail-fast behavior when a dependency is overwhelmed well past what the bulkhead can absorb
• An asyncio version of the same idea, plus a live demonstration of what goes wrong if you reach for the wrong kind of semaphore in async code
• A pytest suite that verifies the bulkhead’s core guarantees without relying on real network calls

Prerequisites

• Python 3.10 or newer (this tutorial used Python 3.13.14 on Windows, but nothing here is platform-specific)
• Comfort with basic Python: functions, exceptions, and what a thread pool is in general terms
• Basic familiarity with what a semaphore does (a counter that lets a limited number of things happen at once). This tutorial explains the rest as it comes up
• No external services, accounts, or API keys. Everything in this tutorial runs on your own machine
• About 45 minutes

Step 1: Build Two Downstream Dependencies, One Healthy and One Slow

To see a starvation problem for real, you need two dependencies that behave differently: one that is fast and healthy, and one that is currently slow, the way a degraded third-party API often is. Create a project folder, an isolated virtual environment, and install what you need:

mkdir bulkhead-demo
cd bulkhead-demo
python -m venv .venv
.venv\Scripts\activate   # on Linux/macOS: source .venv/bin/activate
pip install fastapi uvicorn requests pytest

This tutorial was written against fastapi 0.141.1, uvicorn 0.52.4, requests 2.34.2, and pytest 9.1.1 on Python 3.13.14. Newer versions should behave the same way.

Create downstream.py:

"""Two fake downstream dependencies for the bulkhead demo.

InventoryService: healthy, always fast (~50ms).
ShippingRatesService: a third-party carrier-rate API that is currently
degraded and takes ~2 seconds per call. Nothing here is failing outright
(no 5xx), it is just slow, which is exactly the case a circuit breaker's
failure-threshold logic will not catch.
"""

import time

from fastapi import FastAPI

app = FastAPI()


@app.get("/inventory/{sku}")
def get_inventory(sku: str):
    time.sleep(0.05)
    return {"sku": sku, "in_stock": 42}


@app.get("/shipping-rate")
def get_shipping_rate(zip_code: str):
    time.sleep(2.0)
    return {"zip_code": zip_code, "rate_cents": 899}

Worth pausing on: ShippingRatesService is not broken. It answers every request correctly, eventually. A circuit breaker built to trip on a failure threshold would never fire here, because there are no failures to count, just a dependency that is unhelpfully slow. That is exactly the gap the Bulkhead pattern fills that a circuit breaker does not: it protects the rest of your application from a dependency that is technically working but too slow to share resources with anything else.

Start the service and leave it running for the rest of this tutorial:

uvicorn downstream:app --host 127.0.0.1 --port 8100

In a second terminal, confirm both endpoints work:

curl http://127.0.0.1:8100/inventory/ABC123
# {"sku":"ABC123","in_stock":42}

curl "http://127.0.0.1:8100/shipping-rate?zip_code=94103"
# (waits about 2 seconds)
# {"zip_code":"94103","rate_cents":899}

Step 2: Reproduce the Starvation Problem for Real

Most application code that calls multiple dependencies shares one general-purpose thread pool for all outbound calls, because that is what you get by default if you never think about it. Create timing.py first, a small helper you will reuse for the rest of this tutorial:

"""Shared helper for measuring caller-observed latency correctly.

The naive way to time a batch of futures is to loop over them in
submission order and call .result() on each in turn. That is wrong: a
future that finished almost instantly on its own worker thread will
still report a huge "latency" if the main thread was busy blocking on
an earlier, slower future before it got around to checking. The fix is
to record each future's completion time the moment it actually
completes, using add_done_callback, not when the main thread happens
to look at it.
"""

import time
from concurrent.futures import Future
from typing import List, Tuple


def submit_timed(pool, kind: str, results: List[Tuple[str, float]], fn, *args) -> Future:
    start = time.monotonic()

    def _on_done(future: Future) -> None:
        future.result()  # re-raise if the call failed
        results.append((kind, time.monotonic() - start))

    future = pool.submit(fn, *args)
    future.add_done_callback(_on_done)
    return future

That docstring is not theoretical. The first version of this tutorial’s demo started each timer inside the worker function, after the task had already begun running, and it reported a fast inventory call as taking exactly as long as the slow shipping call queued in front of it, because it never measured the wait. Python’s own documentation for Future.add_done_callback confirms the fix works as intended: Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. […] If the future has already completed or been cancelled, fn will be called immediately. Recording the timestamp the instant a future actually finishes, rather than whenever your code happens to check it, is the only way to get an honest number.

Now create step1_naive_shared_pool.py:

"""Step 1: a naive client that shares one thread pool across both
downstream dependencies, and what happens when one of them is slow.
"""

from concurrent.futures import ThreadPoolExecutor, wait

import requests

from timing import submit_timed

BASE = "http://127.0.0.1:8100"
session = requests.Session()

# One shared pool for every outbound call the app makes, regardless of
# which downstream service it is calling. 4 workers is a realistic size
# for a small service instance.
pool = ThreadPoolExecutor(max_workers=4)


def call_inventory(sku: str) -> None:
    session.get(f"{BASE}/inventory/{sku}", timeout=10)


def call_shipping_rate(zip_code: str) -> None:
    session.get(f"{BASE}/shipping-rate", params={"zip_code": zip_code}, timeout=10)


def main() -> None:
    results = []  # (kind, elapsed_seconds), appended the instant each future completes

    # Simulate a burst of real traffic: 8 checkout requests need a
    # shipping rate (the slow dependency), and, at the same moment,
    # 2 product-page requests just need a quick inventory check.
    futures = []
    for i in range(8):
        futures.append(submit_timed(pool, "shipping", results, call_shipping_rate, f"9410{i}"))
    for i in range(2):
        futures.append(submit_timed(pool, "inventory", results, call_inventory, f"SKU-{i}"))

    wait(futures)

    shipping = sorted(t for k, t in results if k == "shipping")
    inventory = sorted(t for k, t in results if k == "inventory")
    print(f"shipping-rate caller-observed latencies (s): {[round(t, 2) for t in shipping]}")
    print(f"inventory caller-observed latencies (s):     {[round(t, 2) for t in inventory]}")
    print(f"worst inventory latency: {round(max(inventory), 2)}s "
          f"(inventory itself takes ~0.05s when it actually runs)")


if __name__ == "__main__":
    main()

Run it while the downstream service from Step 1 is still up:

python step1_naive_shared_pool.py

Real captured output from this exact run:

shipping-rate caller-observed latencies (s): [2.01, 2.01, 2.01, 2.01, 4.02, 4.02, 4.02, 4.02]
inventory caller-observed latencies (s):     [4.08, 4.08]
worst inventory latency: 4.08s (inventory itself takes ~0.05s when it actually runs)

Walk through why: the pool has 4 workers. The first 4 shipping-rate calls start immediately and each take 2 seconds. The remaining 4 shipping-rate calls have to wait for a worker to free up, so they land around 4 seconds. The 2 inventory calls were submitted last, so in strict first-in-first-out order they queue up behind all eight shipping-rate calls, both batches of them, and only get a worker once the last shipping-rate call finishes. An inventory lookup that should take 50 milliseconds took 4.08 seconds instead, more than 80 times slower, and it has nothing wrong with it at all. It is only guilty by association with a shared pool.

Step 3: Build a Bulkhead With a Dedicated Pool Per Dependency

The most direct fix mirrors Azure’s own description of the pattern: A consumer can also partition resources to ensure that resources used to call one service don’t affect the resources used to call another service. For example, a consumer that calls multiple services might be assigned a connection pool for each service. If a service begins to fail, it only affects the connection pool assigned for that service. The consumer can continue to use other services. In Python, the equivalent of “a connection pool per service” is a ThreadPoolExecutor per service. Create step2_bulkhead_dedicated_pools.py:

"""Step 2: the Bulkhead pattern, thread-pool-per-dependency variant.

Each downstream dependency gets its own dedicated ThreadPoolExecutor.
A burst of slow shipping-rate calls can only ever occupy the shipping
pool's own workers; it cannot touch the workers reserved for inventory.
"""

from concurrent.futures import ThreadPoolExecutor, wait

import requests

from timing import submit_timed

BASE = "http://127.0.0.1:8100"
session = requests.Session()

# Two separate pools instead of one shared pool. Sizing still matters
# (see the Common Mistakes section), but the key change is that each
# dependency now has its own bulkhead: exhausting one cannot starve
# the other.
inventory_pool = ThreadPoolExecutor(max_workers=2, thread_name_prefix="inventory")
shipping_pool = ThreadPoolExecutor(max_workers=4, thread_name_prefix="shipping")


def call_inventory(sku: str) -> None:
    session.get(f"{BASE}/inventory/{sku}", timeout=10)


def call_shipping_rate(zip_code: str) -> None:
    session.get(f"{BASE}/shipping-rate", params={"zip_code": zip_code}, timeout=10)


def main() -> None:
    results = []

    # The exact same burst as step 1: 8 slow shipping-rate lookups and
    # 2 quick inventory checks arriving at the same moment.
    futures = []
    for i in range(8):
        futures.append(submit_timed(shipping_pool, "shipping", results, call_shipping_rate, f"9410{i}"))
    for i in range(2):
        futures.append(submit_timed(inventory_pool, "inventory", results, call_inventory, f"SKU-{i}"))

    wait(futures)

    shipping = sorted(t for k, t in results if k == "shipping")
    inventory = sorted(t for k, t in results if k == "inventory")
    print(f"shipping-rate caller-observed latencies (s): {[round(t, 2) for t in shipping]}")
    print(f"inventory caller-observed latencies (s):     {[round(t, 2) for t in inventory]}")
    print(f"worst inventory latency: {round(max(inventory), 2)}s "
          f"(inventory itself takes ~0.05s when it actually runs)")


if __name__ == "__main__":
    main()

Run it against the exact same downstream service, with no other changes:

python step2_bulkhead_dedicated_pools.py

Real captured output:

shipping-rate caller-observed latencies (s): [2.02, 2.02, 2.02, 2.01, 4.02, 4.02, 4.02, 4.02]
inventory caller-observed latencies (s):     [0.06, 0.06]
worst inventory latency: 0.06s (inventory itself takes ~0.05s when it actually runs)

The shipping-rate numbers are unchanged, that dependency is still slow, a bulkhead does not fix a slow dependency. What changed is inventory: 0.06 seconds instead of 4.08. The two inventory calls never had to wait on anything happening in the shipping pool, because they were never competing for the same workers in the first place. That is the entire pattern: not making a slow dependency faster, but making sure its slowness stays contained to itself.

Why This Counts as a Bulkhead, Not Just Two Thread Pools

It is worth being precise about what makes this a deliberate pattern rather than an accident of code structure. Azure’s guidance frames the goal as isolating consumers and services from cascading failures, and lists a specific benefit: Provides different quality of service levels for consuming applications. You can configure a high-priority consumer pool to use high-priority services. Notice that the shipping pool above is deliberately larger (4 workers) than the inventory pool (2 workers), which can look backwards until you think about why: each shipping-rate call occupies its worker for roughly 40 times as long as an inventory call does, so sustaining even a modest amount of concurrent shipping traffic needs more workers, not fewer, just to keep pace. Sizing each pool to its own dependency’s actual latency and expected volume, rather than splitting one undifferentiated number evenly, is the actual design decision. A bulkhead is a sizing decision made on purpose, once, per dependency, not a coincidence of how many thread pools happen to exist in your code.

Step 4: The Other Flavor: Semaphore Bulkheads That Fail Fast

Dedicated thread pools work, but they are not the only way to build a bulkhead, and they are not always the cheapest. The Java library resilience4j, which Azure’s own documentation names as a reference implementation, ships two different bulkhead designs, and the distinction matters. Resilience4j’s own documentation describes it directly: Resilience4j provides two implementations of a bulkhead pattern that can be used to limit the number of concurrent execution: a SemaphoreBulkhead which uses Semaphores [and] a FixedThreadPoolBulkhead which uses a bounded queue and a fixed thread pool. The SemaphoreBulkhead should work well across a variety of threading and I/O models. It is based on a semaphore, and unlike Hystrix, does not provide “shadow” thread pool option. It is up to the client to ensure correct thread pool sizing that will be consistent with bulkhead configuration.

In other words: a semaphore bulkhead does not own any threads at all. It sits on top of whatever pool you already have (or an async event loop, see Step 6) and simply caps how many calls to one specific dependency may be in flight at once, using a counter. Resilience4j configures this with two settings, and this tutorial’s implementation below uses the same two names: maxConcurrentCalls (default 25 in resilience4j) is, per its own docs, the Max amount of parallel executions allowed by the bulkhead, and maxWaitDuration (default 0) is the Max amount of time a thread should be blocked for when attempting to enter a saturated bulkhead. That second setting is the important one this tutorial builds next: what should happen when a dependency is so overwhelmed that even its own bulkhead is full.

Create bulkhead.py:

"""A small semaphore-based bulkhead, modeled on resilience4j's
SemaphoreBulkhead: it caps concurrent calls per dependency and fails
fast instead of queuing forever once a caller has waited too long for
a free slot.

This deliberately does NOT manage its own thread pool. Per resilience4j's
own docs, the semaphore only gates how many calls to one dependency may
be in flight at once; callers still need a big-enough shared pool (or
async runtime) to run work on.
"""

import threading


class BulkheadFullError(Exception):
    """Raised when a call could not get a bulkhead slot in time."""


class SemaphoreBulkhead:
    def __init__(self, name: str, max_concurrent_calls: int, max_wait_seconds: float):
        self.name = name
        self.max_concurrent_calls = max_concurrent_calls
        self.max_wait_seconds = max_wait_seconds
        self._semaphore = threading.Semaphore(max_concurrent_calls)

    def call(self, fn, *args, **kwargs):
        acquired = self._semaphore.acquire(timeout=self.max_wait_seconds)
        if not acquired:
            raise BulkheadFullError(
                f"bulkhead '{self.name}' full: {self.max_concurrent_calls} calls "
                f"already in flight, waited {self.max_wait_seconds}s for a slot"
            )
        try:
            return fn(*args, **kwargs)
        finally:
            self._semaphore.release()

The timeout behavior of self._semaphore.acquire(timeout=...) is exactly what makes the fail-fast behavior possible, and it is worth confirming precisely rather than assuming it. Python’s own documentation for threading.Semaphore.acquire states: When invoked with a timeout other than None, it will block for at most timeout seconds. If acquire does not complete successfully in that interval, return False. Return True otherwise. That is the whole mechanism: if not acquired checks for exactly that False, and raises instead of letting the caller wait indefinitely.

Overwhelm It on Purpose

Create step3_semaphore_bulkhead.py to see the fail-fast behavior for real, not just described:

"""Step 3: semaphore bulkheads on top of one shared thread pool, plus
what happens when a dependency is overwhelmed well past what its
bulkhead can absorb: it should fail fast, not queue forever.
"""

from concurrent.futures import ThreadPoolExecutor, wait

import requests

from bulkhead import BulkheadFullError, SemaphoreBulkhead
from timing import submit_timed

BASE = "http://127.0.0.1:8100"
session = requests.Session()

# One shared, generously-sized pool for all outbound calls...
shared_pool = ThreadPoolExecutor(max_workers=20)

# ...but each dependency gets its own semaphore bulkhead limiting how
# many of those calls may target it at once.
shipping_bulkhead = SemaphoreBulkhead("shipping-rate", max_concurrent_calls=3, max_wait_seconds=1.0)
inventory_bulkhead = SemaphoreBulkhead("inventory", max_concurrent_calls=5, max_wait_seconds=1.0)


def call_inventory(sku: str) -> str:
    return inventory_bulkhead.call(
        lambda: session.get(f"{BASE}/inventory/{sku}", timeout=10).json()["sku"]
    )


def call_shipping_rate(zip_code: str) -> str:
    return shipping_bulkhead.call(
        lambda: session.get(f"{BASE}/shipping-rate", params={"zip_code": zip_code}, timeout=10).json()["zip_code"]
    )


def main() -> None:
    results = []
    outcomes = {"shipping": {"ok": 0, "rejected": 0}, "inventory": {"ok": 0, "rejected": 0}}

    def run_and_record(kind: str, fn, *args):
        try:
            fn(*args)
            outcomes[kind]["ok"] += 1
        except BulkheadFullError:
            outcomes[kind]["rejected"] += 1

    # Flood shipping-rate with 10 concurrent checkouts, far more than its
    # 3-slot bulkhead plus 1-second patience can absorb (each real call
    # takes 2s), while 4 ordinary inventory checks arrive at the same time.
    futures = []
    for i in range(10):
        futures.append(submit_timed(shared_pool, "shipping", results,
                                     run_and_record, "shipping", call_shipping_rate, f"9410{i}"))
    for i in range(4):
        futures.append(submit_timed(shared_pool, "inventory", results,
                                     run_and_record, "inventory", call_inventory, f"SKU-{i}"))

    wait(futures)

    shipping = sorted(t for k, t in results if k == "shipping")
    inventory = sorted(t for k, t in results if k == "inventory")
    print(f"shipping-rate: {outcomes['shipping']['ok']} succeeded, "
          f"{outcomes['shipping']['rejected']} rejected fast, latencies (s): {[round(t, 2) for t in shipping]}")
    print(f"inventory:     {outcomes['inventory']['ok']} succeeded, "
          f"{outcomes['inventory']['rejected']} rejected fast, latencies (s): {[round(t, 2) for t in inventory]}")


if __name__ == "__main__":
    main()

Run it:

python step3_semaphore_bulkhead.py

Real captured output:

shipping-rate: 3 succeeded, 7 rejected fast, latencies (s): [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.01, 2.01, 2.01]
inventory:     4 succeeded, 0 rejected fast, latencies (s): [0.06, 0.06, 0.06, 0.06]

Ten checkout requests needed a shipping rate at the same moment. Three got a real slot and took the full 2.01 seconds the dependency actually needs. The other seven could not get a slot within 1 second, so they failed fast, in about a second, with a clear BulkheadFullError instead of piling into an unbounded queue and eventually timing out anyway, minutes later, having wasted a thread the entire time. A caller that gets BulkheadFullError quickly can retry, show the user a “try again in a moment” message, or fall back to a cached rate; a caller stuck in an invisible queue for two minutes cannot do any of that. Meanwhile, all four inventory checks succeeded in 0.06 seconds, completely unaffected, because they were never competing for the shipping-rate bulkhead’s three slots in the first place.

Step 5: Verify With a Test Suite

Real HTTP calls and real sleeps make good demonstrations, but a test suite should not depend on a live server or take multiple seconds to run. Create test_bulkhead.py, testing the SemaphoreBulkhead class directly with fast, synthetic functions:

import threading
import time

import pytest

from bulkhead import BulkheadFullError, SemaphoreBulkhead


def test_allows_calls_up_to_capacity():
    bulkhead = SemaphoreBulkhead("test", max_concurrent_calls=2, max_wait_seconds=1.0)
    assert bulkhead.call(lambda: 1 + 1) == 2


def test_rejects_once_capacity_and_wait_are_exhausted():
    bulkhead = SemaphoreBulkhead("test", max_concurrent_calls=1, max_wait_seconds=0.2)
    started = threading.Event()

    def hold_slot():
        started.set()
        time.sleep(1.0)

    holder = threading.Thread(target=lambda: bulkhead.call(hold_slot))
    holder.start()
    started.wait(timeout=1.0)

    with pytest.raises(BulkheadFullError):
        bulkhead.call(lambda: None)

    holder.join()


def test_releases_the_slot_even_if_the_call_raises():
    bulkhead = SemaphoreBulkhead("test", max_concurrent_calls=1, max_wait_seconds=0.5)

    def boom():
        raise ValueError("downstream exploded")

    with pytest.raises(ValueError):
        bulkhead.call(boom)

    # If the slot leaked, this would raise BulkheadFullError instead.
    assert bulkhead.call(lambda: "ok") == "ok"


def test_never_exceeds_max_concurrent_calls():
    bulkhead = SemaphoreBulkhead("test", max_concurrent_calls=3, max_wait_seconds=2.0)
    concurrent_count = 0
    peak_concurrent = 0
    lock = threading.Lock()

    def track():
        nonlocal concurrent_count, peak_concurrent
        with lock:
            concurrent_count += 1
            peak_concurrent = max(peak_concurrent, concurrent_count)
        time.sleep(0.2)
        with lock:
            concurrent_count -= 1

    threads = [threading.Thread(target=lambda: bulkhead.call(track)) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert peak_concurrent <= 3


def test_two_bulkheads_are_fully_independent():
    a = SemaphoreBulkhead("a", max_concurrent_calls=1, max_wait_seconds=0.2)
    b = SemaphoreBulkhead("b", max_concurrent_calls=1, max_wait_seconds=0.2)
    started = threading.Event()

    holder = threading.Thread(target=lambda: a.call(lambda: (started.set(), time.sleep(0.5))))
    holder.start()
    started.wait(timeout=1.0)

    # a's single slot is held, but b is a separate bulkhead and must be unaffected.
    assert b.call(lambda: "fine") == "fine"
    holder.join()

The most important test here is test_never_exceeds_max_concurrent_calls: it fires 10 threads at a bulkhead configured for 3 concurrent calls, and asserts the actual observed peak concurrency, tracked with a plain lock and a counter, never went above 3. This does not just check the happy path, it checks the guarantee the whole pattern exists to provide. Run the suite:

pytest test_bulkhead.py -v

Real captured output:

test_bulkhead.py::test_allows_calls_up_to_capacity PASSED                [ 20%]
test_bulkhead.py::test_rejects_once_capacity_and_wait_are_exhausted PASSED [ 40%]
test_bulkhead.py::test_releases_the_slot_even_if_the_call_raises PASSED  [ 60%]
test_bulkhead.py::test_never_exceeds_max_concurrent_calls PASSED         [ 80%]
test_bulkhead.py::test_two_bulkheads_are_fully_independent PASSED        [100%]

5 passed in 2.36s

Step 6: Bulkheads in Async Code, and a Mistake That Freezes Everything

Everything so far has used threading.Semaphore and ThreadPoolExecutor, both blocking, thread-based tools. If your application is built on asyncio instead, the same idea applies, but the tool has to change too, and this is the part where it is easy to get subtly wrong.

The correct version uses asyncio.Semaphore, an async with block, and offloads the actual blocking network call with asyncio.to_thread. Create step4_asyncio_bulkhead.py and start with the correct pattern:

"""Step 4: the same bulkhead idea in async code.

Two things to verify for real, not just assert:
1. An asyncio.Semaphore-based bulkhead correctly isolates a slow
   dependency from a fast one, same as the threaded version.
2. Reusing the threading.Semaphore-based SemaphoreBulkhead directly
   inside async code (instead of an asyncio.Semaphore) blocks the
   entire event loop while it waits, freezing every other coroutine,
   not just the one calling the slow dependency.
"""

import asyncio
import time

import requests

from bulkhead import SemaphoreBulkhead

BASE = "http://127.0.0.1:8100"
session = requests.Session()


# --- Part A: correct asyncio bulkhead -------------------------------

shipping_sem = asyncio.Semaphore(3)
inventory_sem = asyncio.Semaphore(5)


async def call_shipping_async(zip_code: str) -> float:
    start = time.monotonic()
    async with shipping_sem:
        await asyncio.to_thread(
            session.get, f"{BASE}/shipping-rate", params={"zip_code": zip_code}, timeout=10
        )
    return time.monotonic() - start


async def call_inventory_async(sku: str) -> float:
    start = time.monotonic()
    async with inventory_sem:
        await asyncio.to_thread(session.get, f"{BASE}/inventory/{sku}", timeout=10)
    return time.monotonic() - start


async def part_a() -> None:
    tasks = [asyncio.create_task(call_shipping_async(f"9410{i}")) for i in range(6)]
    tasks += [asyncio.create_task(call_inventory_async(f"SKU-{i}")) for i in range(2)]
    results = await asyncio.gather(*tasks)
    shipping_times = [round(t, 2) for t in results[:6]]
    inventory_times = [round(t, 2) for t in results[6:]]
    print(f"[asyncio.Semaphore] shipping latencies (s): {shipping_times}")
    print(f"[asyncio.Semaphore] inventory latencies (s): {inventory_times}")

Run just this part first to confirm it works the way you would expect from Steps 3 and 4:

python -c "import asyncio, step4_asyncio_bulkhead as m; asyncio.run(m.part_a())"

[asyncio.Semaphore] shipping latencies (s): [2.02, 2.02, 2.02, 4.03, 4.03, 4.03]
[asyncio.Semaphore] inventory latencies (s): [0.07, 0.07]

Same shape as before: shipping is gated to 3 at a time by its semaphore, and inventory sails through in 0.07 seconds regardless.

Now the mistake. Append this to the same file:

# --- Part B: the mistake, a blocking Semaphore inside async code ----

broken_shipping_bulkhead = SemaphoreBulkhead("shipping-rate", max_concurrent_calls=1, max_wait_seconds=5.0)


async def call_shipping_blocking_mistake(zip_code: str) -> None:
    # threading.Semaphore.acquire() is a blocking call. Awaiting nothing
    # here means it runs synchronously on the event loop's own thread.
    broken_shipping_bulkhead.call(
        lambda: session.get(f"{BASE}/shipping-rate", params={"zip_code": zip_code}, timeout=10)
    )


async def heartbeat(t_start: float, arrivals: list) -> None:
    # A trivial coroutine that should tick roughly every 0.1s if the
    # event loop stays responsive. Measuring "time since the previous
    # await" hides a block that happens before this coroutine gets its
    # first turn at all, so record arrival time against a shared
    # reference clock (t_start) instead of a per-iteration local one.
    for _ in range(20):
        await asyncio.sleep(0.1)
        arrivals.append(time.monotonic() - t_start)


async def part_b() -> None:
    arrivals: list = []
    t_start = time.monotonic()
    await asyncio.gather(
        call_shipping_blocking_mistake("94103"),
        call_shipping_blocking_mistake("94104"),  # will contend for the same 1 slot
        heartbeat(t_start, arrivals),
    )
    expected = [round(0.1 * (i + 1), 2) for i in range(len(arrivals))]
    worst_lag = max(actual - exp for actual, exp in zip(arrivals, expected))
    print(f"[blocking Semaphore in async code] first 5 heartbeat tick arrival times (s): "
          f"{[round(a, 2) for a in arrivals[:5]]} (expected ~{expected[:5]})")
    print(f"[blocking Semaphore in async code] worst lag behind schedule: {round(worst_lag, 2)}s")


if __name__ == "__main__":
    asyncio.run(part_a())
    asyncio.run(part_b())

The heartbeat coroutine's docstring comment describes a bug this tutorial actually walked into while writing this exact demo: the first version measured each tick's delay as "time since asyncio.sleep(0.1) was awaited," starting the clock fresh inside each loop iteration. That number cannot see a delay that happens before the coroutine gets its first turn to run at all, because the local clock has not started yet, the exact same category of mistake as the naive timing this tutorial fixed in Step 2, just showing up in a different place. The fix is the same idea too: measure against one shared reference clock (t_start, captured before anything starts) instead of a clock local to code that might not run immediately.

Run the whole file:

python step4_asyncio_bulkhead.py

Real captured output:

[asyncio.Semaphore] shipping latencies (s): [2.01, 2.02, 2.01, 4.02, 4.02, 4.02]
[asyncio.Semaphore] inventory latencies (s): [0.06, 0.06]
[blocking Semaphore in async code] first 5 heartbeat tick arrival times (s): [4.12, 4.23, 4.34, 4.46, 4.57] (expected ~[0.1, 0.2, 0.3, 0.4, 0.5])
[blocking Semaphore in async code] worst lag behind schedule: 4.26s

The heartbeat's first tick, which should land at 0.1 seconds, actually landed at 4.12 seconds. The entire event loop, including a coroutine that has nothing to do with shipping rates at all, froze for over four seconds. This is because call_shipping_blocking_mistake has no await anywhere in it: once the event loop starts running it, Python executes the whole function body, including the blocking threading.Semaphore.acquire() call and the blocking requests.get() call, synchronously on the event loop's single thread, and cannot switch to any other coroutine until it returns. A threading.Semaphore-based bulkhead built for thread-based code will still run without raising an error if you drop it into an async def function, which is exactly what makes this mistake easy to miss in a code review: nothing crashes, it just quietly stops your entire application from doing anything else while it waits.

Common Mistakes and Gotchas

• Using a blocking semaphore in async code. As Step 6 showed with real numbers, a threading.Semaphore (or this tutorial's SemaphoreBulkhead) called directly from an async def function without a thread offload will stall the entire event loop for as long as it waits. Use asyncio.Semaphore with async with in async code, and if you need to call a blocking library like requests from inside it, wrap the call in asyncio.to_thread.
• Sizing a bulkhead the same as your old shared pool. If you give every dependency's bulkhead the same capacity the old shared pool had, in aggregate they can still exhaust the machine's real resources (threads, sockets, memory) even though no single dependency can starve another. Size each bulkhead to what that specific dependency actually needs and how critical it is, the way Step 3 gave inventory 2 workers and shipping 4, not an arbitrary equal split.
• Forgetting a per-call timeout. A bulkhead limits how many calls to a dependency can run at once, but if an individual call has no timeout of its own, a single hung call can occupy its slot indefinitely, and the bulkhead's own capacity slowly bleeds down to zero. Azure's own guidance is explicit that this pattern is meant to be combined with others: To provide more sophisticated fault handling, consider combining bulkheads with retry, circuit breaker, and throttling patterns.
• Treating "bulkhead" and "rate limiter" as the same thing. A rate limiter throttles how many requests arrive at your service over time. A bulkhead partitions the resources your service uses to call other services. You can need either one without the other, and a system under heavy load often needs both.
• Assuming a bulkhead makes a slow dependency fast. It does not, and none of the numbers in this tutorial pretend otherwise: shipping-rate calls took exactly as long with a bulkhead as without one. What changed is that the slowness stopped spreading.

How to Confirm It All Works End to End

1. Start downstream.py with uvicorn and confirm both endpoints respond via curl, per Step 1.
2. Run step1_naive_shared_pool.py and confirm you see a large inventory latency (multiple seconds) despite inventory itself only taking ~0.05s. If you do not see starvation, check that both endpoints are hitting the same running server and that max_workers=4 was not changed.
3. Run step2_bulkhead_dedicated_pools.py against the same server and confirm inventory latency drops back to roughly 0.05 to 0.1 seconds while shipping-rate latency is unchanged.
4. Run step3_semaphore_bulkhead.py and confirm some shipping-rate calls succeed (up to max_concurrent_calls) and the rest fail fast with a rejected count greater than zero, landing around the configured max_wait_seconds, while every inventory call still succeeds.
5. Run step4_asyncio_bulkhead.py and confirm Part A shows the same isolation as Step 2/3, and Part B shows a large lag (multiple seconds) on the heartbeat coroutine, proving the blocking-semaphore mistake is real and not just theoretical.
6. Run pytest test_bulkhead.py -v and confirm all 5 tests pass, in particular test_never_exceeds_max_concurrent_calls, which is the guarantee the whole pattern exists to provide.

Next Steps

A bulkhead answers one specific question: how do I stop a slow or overloaded dependency from starving the resources other dependencies need? It is one piece of a larger resilience toolkit, and it is meant to be combined with the others, not to replace them. If you want to keep building on this:

• Read How to Build a Circuit Breaker in Python to Stop Cascading Failures for the pattern that stops calling a dependency once it is actually failing, not just slow.
• Read How to Retry Failed API Calls in Python With Exponential Backoff and Jitter for what to do with a call that failed rather than one that timed out on a full bulkhead.
• Read How to Build an API Rate Limiter in Python: Token Bucket and Sliding Window Algorithms for the inbound-side counterpart to this outbound-side pattern.
• Read How to Implement the Transactional Outbox Pattern in Python to Avoid Dual Writes for a resilience pattern aimed at a different failure mode entirely: losing an update between two systems, rather than one system starving another.
• Try wiring a real ThreadPoolBulkhead-style bounded queue on top of the semaphore bulkhead in this tutorial (resilience4j's second implementation), so an overloaded dependency queues a small, bounded number of calls before failing fast, instead of failing fast immediately at the moment its concurrency cap is hit.