Learning Hub

How to Stop Hung HTTP Calls in Python With Timeouts and Deadlines, and Catch the Slow-Drip Response That Slips Under requests, httpx, and urllib

Learn how to stop hung HTTP calls in Python: measure what requests, httpx, urllib, and aiohttp do by default, watch a slow-drip server slip under every read timeout, then add whole-call deadlines, a...

November 5, 2026 39 Min Read
14

Most code that calls another service looks safe until that service slows down. Your program sends a request and waits for the answer. If the answer never comes, the program keeps waiting. A thread that is waiting cannot do other work, so one stuck call ties up one thread. A few dozen stuck calls can tie up every thread a service has, and a service that was healthy a minute ago stops answering anyone, even though nothing is wrong with its own code.

Table Of Content

The surprise is how many popular ways of making an HTTP call in Python will wait as long as the server likes unless you say otherwise. In the first test below, three of the four clients I tried waited the full seven seconds that a slow server held back its answer, and the fourth gave up after five.

In this tutorial you will learn what a timeout actually limits (less than most people expect), measure what four Python HTTP clients do by default, set timeouts on each of them, and watch a server that sends one byte per second slip under every read timeout. Then you will add the limit that really bounds a call: a deadline for the whole operation. You will write a small helper for requests, use asyncio.timeout for async code, see why a timeout on the waiting side does not stop the work, share one time budget between several calls and their retries, and put it all behind a FastAPI service that answers 504 instead of hanging. A 13-test suite and six planted bugs show whether the helper deserves your trust.

Everything runs on your own machine against a deliberately misbehaving server on 127.0.0.1, so nobody else’s server is involved. The one exception is a connect test aimed at 192.0.2.1. RFC 5737 says the block 192.0.2.0/24 is “provided for use in documentation” and that these addresses “SHOULD NOT appear on the public Internet”, so nothing real is supposed to answer there.

By the end you will be able to:

  • say what connect, read, write and pool timeouts each limit, and what none of them limits;
  • set timeouts in requests, httpx, urllib and aiohttp, and catch the right exceptions, including the case where requests reports a read timeout as a ConnectionError;
  • recognise a slow-drip response and put a deadline on the whole download, in synchronous and in asynchronous code;
  • stream a body with requests without losing the ability to stop it, and know why iter_content(8192) cannot do that;
  • explain why Future.result(timeout=...) and asyncio.timeout do not always stop the work underneath them;
  • give a request one time budget that is shared by several calls and their retries;
  • return a 504 from both an async def and a plain def FastAPI endpoint without leaving a worker busy;
  • test all of it, and check that the tests catch planted bugs.

Everything below was run on Python 3.13.14 on Windows 11 on November 5, 2026, with the package versions printed in Step 1. Every output block is copied from those runs. The timings come from one machine, so expect your numbers to differ by a fraction of a second.

The terms used in this tutorial

  • Timeout. A limit on how long one step of a network call may take. It does not always limit the whole call, which is the main subject of this tutorial.
  • Connect timeout. The longest the client waits for the TCP connection to the server to be set up.
  • Read timeout. The longest the client waits for the server to send more data. In most libraries it restarts every time some data arrives.
  • Write and pool timeouts. httpx also limits how long sending data may take (write) and how long the client may wait for a free connection from its connection pool (pool).
  • Deadline. A moment in time after which the whole operation must be over, however many steps it contains. This tutorial also uses budget for a deadline object that several calls share.
  • Slow-drip response. An answer that arrives a few bytes at a time, with short gaps between the pieces. Each gap is small, but the total is long.
  • Blocking call. A call that does not return until it has a result. The thread that made it can do nothing else meanwhile.
  • Cancellation. In asyncio, the way a running task is told to stop. It takes effect when the task reaches an await.
  • Dependency. Any other service your code calls over the network.
  • 504 Gateway Timeout. The HTTP status a server uses when it is acting as a gateway or proxy and did not get a timely answer from the service behind it.

Prerequisites

  • Python 3.11 or newer. asyncio.timeout was added in 3.11. I used 3.13.14.
  • A terminal. The commands work in PowerShell, Bash and zsh. Only the way you activate a virtual environment differs.
  • Basic Python. Functions, exceptions, with blocks and threads at the level of knowing that a thread runs a function. Each new idea is explained when it first appears.
  • About 90 minutes. Each script finishes in well under a minute, and the whole lab runs in about two minutes.
  • A network that drops packets sent to 192.0.2.1. Mine did, so the connect test ends in a timeout. Some networks reject such packets at once. If yours does, you will see an immediate connection error in those lines instead, which is your network answering rather than your timeout failing.

Step 1: Set Up the Lab and See What Your Client Does by Default

Create the project

A virtual environment is a private copy of Python and its packages for one project. Create one, activate it (venv\Scripts\Activate.ps1 in Windows PowerShell, source venv/bin/activate on Linux and macOS), and install the packages with the exact versions used here.

mkdir timeouts-lab
cd timeouts-lab
python -m venv venv
python -m pip install requests==2.34.2 httpx==0.28.1 aiohttp==3.14.5 fastapi==0.162.0 uvicorn==0.54.0 pytest==9.2.0

Every file in this tutorial goes into the timeouts-lab folder. The first file is the server you will test against. Real servers rarely misbehave on cue, so this one does it on request.

Build a server that misbehaves on purpose

Save this as slowserver.py. It uses only the standard library. It has three routes: /fast answers at once, /stall?seconds=S thinks for S seconds before it sends anything, and /drip?n=N&gap=G sends its headers at once and then one byte every G seconds.

"""slowserver.py: a deliberately misbehaving HTTP server for timeout experiments.

Routes (all GET, all on 127.0.0.1):
  /fast                      answers at once with "ok"
  /stall?seconds=S           holds the whole answer (headers too) for S seconds
  /drip?n=N&gap=G            sends the headers at once, then one byte every G seconds (N bytes)
"""
import socket
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse

EVENTS = []  # (seconds since this process started, kind, details)
_LOCK = threading.Lock()
_T0 = time.monotonic()


def log(kind, **details):
    with _LOCK:
        EVENTS.append((round(time.monotonic() - _T0, 1), kind, details))


def events(kind=None):
    with _LOCK:
        return [e for e in EVENTS if kind is None or e[1] == kind]


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, *args):  # keep the output clean
        pass

    def _head(self, length):
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(length))
        self.send_header("Connection", "close")
        self.end_headers()
        self.wfile.flush()

    def do_GET(self):
        self.close_connection = True
        self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
        url = urlparse(self.path)
        q = {k: v[0] for k, v in parse_qs(url.query).items()}
        started = time.monotonic()
        try:
            if url.path == "/fast":
                self._head(2)
                self.wfile.write(b"ok")
            elif url.path == "/stall":
                time.sleep(float(q.get("seconds", "5")))
                self._head(4)
                self.wfile.write(b"done")
            elif url.path == "/drip":
                n, gap = int(q.get("n", "10")), float(q.get("gap", "1"))
                self._head(n)
                for _ in range(n):
                    time.sleep(gap)
                    self.wfile.write(b"x")
                    self.wfile.flush()
                log("drip_finished", seconds=round(time.monotonic() - started, 1))
            else:
                self.send_error(404)
        except OSError:
            log("client_gone", path=url.path, seconds=round(time.monotonic() - started, 1))


class Server(ThreadingHTTPServer):
    daemon_threads = True
    request_queue_size = 128  # many clients at once must not wait in a tiny accept queue


def serve_in_thread(host="127.0.0.1", port=0):
    httpd = Server((host, port), Handler)
    threading.Thread(target=httpd.serve_forever, daemon=True).start()
    return httpd, f"http://{host}:{httpd.server_address[1]}"

A few details are worth knowing. Connection: close keeps every request on its own connection, so no earlier request can affect a later one. TCP_NODELAY asks the operating system to send each small write immediately instead of holding it back to combine it with the next one, which would otherwise blur the one-byte-at-a-time timing. When a write fails because the client has gone away, the server records a client_gone event, which later steps use to show that a cancelled call really closes its connection. request_queue_size = 128 lets many clients connect at the same moment without waiting in a tiny accept queue.

Add a timing helper

Save this as timing.py. timed runs a function and returns how long it took and how it ended. parallel runs several jobs at the same time, so their waiting overlaps, and prints one line per job in order. Each job is still timed on its own.

"""timing.py: run jobs, print how long each took and how it ended."""
import time
from concurrent.futures import ThreadPoolExecutor


def timed(fn):
    t0 = time.perf_counter()
    try:
        outcome = "ok: " + str(fn())
    except BaseException as exc:  # we want to see every kind of ending
        module = type(exc).__module__
        name = type(exc).__qualname__
        outcome = name if module == "builtins" else f"{module}.{name}"
    return time.perf_counter() - t0, outcome


def parallel(jobs):
    """jobs: list of (label, callable). Runs them at the same time, prints in order."""
    with ThreadPoolExecutor(max_workers=len(jobs)) as pool:
        futures = [(label, pool.submit(timed, fn)) for label, fn in jobs]
        width = max(len(label) for label, _ in jobs)
        for label, fut in futures:
            seconds, outcome = fut.result()
            print(f"{label:<{width}}  {seconds:4.1f} s  {outcome}")

Check your versions

Save this as versions.py and run it with python versions.py. Your output should show the same versions, except that urllib3 is installed as a dependency of requests and may be newer by the time you read this.

# versions.py
import platform

import aiohttp
import fastapi
import httpx
import requests
import urllib3
import uvicorn

print("Python", platform.python_version(), "on", platform.system())
for module in (requests, urllib3, httpx, aiohttp, fastapi, uvicorn):
    print(f"{module.__name__:<9} {module.__version__}")
Python 3.13.14 on Windows
requests  2.34.2
urllib3   2.8.0
httpx     0.28.1
aiohttp   3.14.5
fastapi   0.162.0
uvicorn   0.54.0

Measure the defaults

Now ask four clients to call /stall?seconds=7 without setting any timeout. Save this as step01_defaults.py. The clients are the standard library’s urllib.request, requests, httpx and aiohttp. The script builds the httpx client before the clock starts, because constructing one is not free and would blur the timings.

"""step01_defaults.py: what each client does when you set no timeout at all."""
import asyncio
import socket
import urllib.request

import aiohttp
import httpx
import requests

from slowserver import serve_in_thread
from timing import parallel

httpd, base = serve_in_thread()
url = base + "/stall?seconds=7"  # the server thinks for 7 seconds before it answers
client = httpx.Client()  # created before the clock starts


def with_urllib():
    with urllib.request.urlopen(url) as resp:
        return resp.read().decode()


async def fetch_with_aiohttp():
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            return await resp.text()


print("the server holds every answer for 7 seconds")
parallel([
    ("urllib.request.urlopen(url)", with_urllib),
    ("requests.get(url)", lambda: requests.get(url).text),
    ("httpx client.get(url)", lambda: client.get(url).text),
    ("aiohttp session.get(url)", lambda: asyncio.run(fetch_with_aiohttp())),
])
print("socket.getdefaulttimeout():", socket.getdefaulttimeout())

Run it with python step01_defaults.py:

the server holds every answer for 7 seconds
urllib.request.urlopen(url)   7.0 s  ok: done
requests.get(url)             7.0 s  ok: done
httpx client.get(url)         5.0 s  httpx.ReadTimeout
aiohttp session.get(url)      7.0 s  ok: done
socket.getdefaulttimeout(): None

Three clients waited the full seven seconds. Two of them, urllib and requests, would have kept waiting for as long as the server stayed silent, and aiohttp for up to five minutes. httpx gave up at 5.0 seconds. This matches what each project documents:

Client What its documentation says Seen above
urllib.request.urlopen The timeout parameter is “(if not specified, the global default timeout setting will be used)”. The script prints socket.getdefaulttimeout(), and it is None. waited 7.0 s
requests “If no timeout is specified explicitly, requests do not time out.” waited 7.0 s
httpx “The default behavior is to raise a TimeoutException after 5 seconds of network inactivity.” ReadTimeout at 5.0 s
aiohttp “By default aiohttp uses a total 300 seconds (5min) timeout, it means that the whole operation should finish in 5 minutes.” waited 7.0 s

The requests documentation puts the risk plainly: “Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely”. Five minutes, the aiohttp default, is much longer than most users will wait for a page. Notice also that httpx’s five seconds is “network inactivity”, a phrase that matters in Step 3.

How to confirm Step 1 worked

You should see three ok: done lines at 7.0 seconds and one httpx.ReadTimeout at 5.0 seconds, plus socket.getdefaulttimeout(): None. If every line takes about 7.0 seconds, check that httpx is version 0.28.1. If a line fails at once with a connection error, check that no firewall or security tool blocks connections to 127.0.0.1.

Step 2: Add Connect and Read Timeouts, and Catch the Right Exceptions

What happens during one HTTP call

A call has several phases, and a timeout usually guards only some of them:

  1. Look up the name. Turn example.com into an IP address (DNS).
  2. Connect. Set up the TCP connection. The connect timeout guards this.
  3. Send the request. httpx guards this with its write timeout.
  4. Wait for the first byte of the answer. The read timeout guards this.
  5. Receive the rest. The read timeout guards each wait for more data, one at a time.

Phase 1 is easy to forget. The urllib3 documentation, which requests is built on, warns: “Python’s DNS resolver does not obey the timeout specified on the socket.” Amazon’s Builders’ Library makes the same point more broadly: “There are also implementations where the timeout doesn’t cover all remote calls, like DNS or TLS handshakes.” This tutorial does not test DNS or TLS handshakes, so treat a timeout as covering the connect and read phases only.

Set the timeouts and read the exceptions

Save this as step02_timeouts.py. It tries all three failure shapes with each client: no first byte (the /stall route), no connection (the documentation address) and silence in the middle of a body (a /drip with three seconds between bytes).

"""step02_timeouts.py: connect and read timeouts, and the exception each library raises."""
import urllib.request

import httpx
import requests

from slowserver import serve_in_thread
from timing import parallel

httpd, base = serve_in_thread()
stall = base + "/stall?seconds=7"
silent_body = base + "/drip?n=3&gap=3"  # headers at once, then 3 s of silence before each byte
BLACKHOLE = "http://192.0.2.1:81/"  # RFC 5737 documentation address: nothing answers
client = httpx.Client()


def urllib_get(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as resp:
        return len(resp.read())


parallel([
    ("requests  no first byte, timeout=2", lambda: requests.get(stall, timeout=2).text),
    ("requests  no connection, (1, 5)", lambda: requests.get(BLACKHOLE, timeout=(1, 5)).text),
    ("requests  silent mid-body, timeout=1", lambda: requests.get(silent_body, timeout=1).content),
    ("httpx     no first byte, timeout=2", lambda: client.get(stall, timeout=2).text),
    ("httpx     no connection, connect=1", lambda: client.get(BLACKHOLE, timeout=httpx.Timeout(5, connect=1)).text),
    ("httpx     silent mid-body, timeout=1", lambda: client.get(silent_body, timeout=1).content),
    ("urllib    no first byte, timeout=2", lambda: urllib_get(stall, 2)),
    ("urllib    no connection, timeout=1", lambda: urllib_get(BLACKHOLE, 1)),
    ("urllib    silent mid-body, timeout=1", lambda: urllib_get(silent_body, 1)),
])

print()
for cls in (requests.ConnectTimeout, requests.ReadTimeout):
    print(f"requests.{cls.__name__:<14} is a Timeout: {issubclass(cls, requests.Timeout)!s:<5}"
          f"  is a ConnectionError: {issubclass(cls, requests.ConnectionError)}")
for cls in (httpx.ConnectTimeout, httpx.ReadTimeout):
    print(f"httpx.{cls.__name__:<17} is a TimeoutException: {issubclass(cls, httpx.TimeoutException)!s:<5}"
          f"  is the built-in TimeoutError: {issubclass(cls, TimeoutError)}")

Run it with python step02_timeouts.py:

requests  no first byte, timeout=2     2.0 s  requests.exceptions.ReadTimeout
requests  no connection, (1, 5)        1.0 s  requests.exceptions.ConnectTimeout
requests  silent mid-body, timeout=1   1.0 s  requests.exceptions.ConnectionError
httpx     no first byte, timeout=2     2.0 s  httpx.ReadTimeout
httpx     no connection, connect=1     1.0 s  httpx.ConnectTimeout
httpx     silent mid-body, timeout=1   1.0 s  httpx.ReadTimeout
urllib    no first byte, timeout=2     2.0 s  TimeoutError
urllib    no connection, timeout=1     1.0 s  urllib.error.URLError
urllib    silent mid-body, timeout=1   1.0 s  TimeoutError

requests.ConnectTimeout is a Timeout: True   is a ConnectionError: True
requests.ReadTimeout    is a Timeout: True   is a ConnectionError: False
httpx.ConnectTimeout    is a TimeoutException: True   is the built-in TimeoutError: False
httpx.ReadTimeout       is a TimeoutException: True   is the built-in TimeoutError: False

Read the first block like a table. Every timeout fired on time, at 2.0 seconds for the read limits and at 1.0 second for the connect limits and the one-second silence limits. The names of the exceptions differ, and one of them is a trap:

What stalled requests httpx urllib.request
connecting ConnectTimeout ConnectTimeout URLError
waiting for the first byte ReadTimeout ReadTimeout TimeoutError
silence in the middle of the body ConnectionError ReadTimeout TimeoutError

The trap is the requests row for silence in the middle of the body. The read timeout did fire, after 1.0 second, but it reached your code as requests.exceptions.ConnectionError, not as a Timeout. The last four lines of the output show why: ReadTimeout is a Timeout but not a ConnectionError, so code that handles only requests.Timeout lets this one through. In httpx the timeouts inherit from TimeoutException, and the two tested here are not the built-in TimeoutError. In urllib, a failure to connect is wrapped in URLError while a timeout while reading is a bare TimeoutError.

Rules of thumb for the exceptions

  • With requests, catch requests.RequestException around a call, or the pair (requests.Timeout, requests.ConnectionError) when you only want to retry network trouble.
  • With httpx, catch httpx.TimeoutException for any timeout and httpx.TransportError for network trouble in general.
  • With urllib, catch both urllib.error.URLError and TimeoutError.
  • Never write a bare except: around a network call, because it also swallows KeyboardInterrupt.

Choose the numbers

The requests documentation gives one concrete tip: “It’s a good practice to set connect timeouts to slightly larger than a multiple of 3, which is the default TCP packet retransmission window.” Its own example uses timeout=(3.05, 27), which follows that advice. It also defines the read timeout carefully: “it’s the number of seconds that the client will wait between bytes sent from the server. In 99.9% of cases, this is the time before the server sends the first byte”.

For the read timeout itself, Amazon’s Builders’ Library describes a method that works for calls inside one region: “we choose an acceptable rate of false timeouts (such as 0.1%). Then, we look at the corresponding latency percentile on the downstream service (p99.9 in this example).” In other words, measure how long the dependency takes, decide how many healthy answers you are willing to cut off, and set the timeout there. The same page lists cases where this fails, such as clients with substantial network latency, so treat it as a starting point and not a law.

The rest of the tutorial uses small numbers (1 to 3 seconds) only so that the lab runs quickly. In a real service, take your own numbers from your own latency data.

How to confirm Step 2 worked

Your timings should be within a few tenths of a second of the ones above, and the exception names should match the table. If a connect line fails at once with a different exception, your network rejected the packet to 192.0.2.1 instead of dropping it. That is fine for the rest of the tutorial, which does not depend on it.

Step 3: Watch a Slow-Drip Server Slip Under Every Read Timeout

Why a read timeout is not a time limit

A read timeout measures silence, not duration. Every time some data arrives, the clock starts again. The requests documentation says so directly: “timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds).” The urllib3 documentation gives an example: “if a server streams one byte every fifteen seconds, a timeout of 20 seconds will not trigger, even though the request will take several minutes to complete.”

A server that is struggling, a proxy that is draining a slow upstream, or a hostile server can all send an answer in small pieces. Whatever the cause, your client sees a response that is alive but slow.

Reproduce it

Save this as step03_slow_drip.py. The /drip route sends ten bytes, one per second, so the whole answer takes about ten seconds. Every client below is told that two seconds of silence is too much, and the longest silence in this response is one second. The last two lines use aiohttp twice: once with a limit on silence (sock_read) and once with a limit on the whole operation (total).

"""step03_slow_drip.py: a server that sends one byte per second slips under every read timeout."""
import asyncio
import urllib.request

import aiohttp
import httpx
import requests

from slowserver import serve_in_thread
from timing import parallel

httpd, base = serve_in_thread()
url = base + "/drip?n=10&gap=1"  # headers at once, then one byte every second for 10 seconds
client = httpx.Client()


def with_urllib():
    with urllib.request.urlopen(url, timeout=2) as resp:
        return len(resp.read())


async def with_aiohttp(timeout):
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url) as resp:
            return len(await resp.read())


print("every client allows 2 seconds of silence; the longest silence here is 1 second")
parallel([
    ("requests  timeout=2", lambda: len(requests.get(url, timeout=2).content)),
    ("httpx     timeout=2", lambda: len(client.get(url, timeout=2).content)),
    ("urllib    timeout=2", with_urllib),
    ("aiohttp   sock_read=2", lambda: asyncio.run(with_aiohttp(aiohttp.ClientTimeout(sock_read=2)))),
    ("aiohttp   total=2", lambda: asyncio.run(with_aiohttp(aiohttp.ClientTimeout(total=2)))),
])

Run it with python step03_slow_drip.py:

every client allows 2 seconds of silence; the longest silence here is 1 second
requests  timeout=2    10.0 s  ok: 10
httpx     timeout=2    10.0 s  ok: 10
urllib    timeout=2    10.0 s  ok: 10
aiohttp   sock_read=2  10.0 s  ok: 10
aiohttp   total=2       2.0 s  TimeoutError

Four calls “succeeded” after 10.0 seconds, five times the limit that was supposed to protect you. Each of them saw at most one second of silence at a time, so each timer restarted before it could fire. Only aiohttp’s total setting stopped the call at 2.0 seconds. The aiohttp documentation explains the difference. total is “The maximal number of seconds for the whole operation including connection establishment, request sending and response reading.” sock_read is “The maximal number of seconds allowed for period between reading a new data portion from a peer.” None of the other three clients has a setting for the whole operation: requests has none, httpx documents four kinds of timeout (connect, read, write and pool), and urllib passes its number to the socket.

Ten bytes make this look harmless. At one byte per second, a one-megabyte body would take more than eleven days, and the call would hold its thread the whole time.

How to confirm Step 3 worked

You should see four lines at about 10.0 seconds ending in ok: 10, and one TimeoutError at about 2.0 seconds. If the first four fail early, your timeout is shorter than the one-second gap in this experiment, which is a different problem.

Step 4: Put a Deadline on the Whole Download With requests

The idea, and why the obvious loop fails

To bound the whole download with requests, you stream the body in chunks and check the clock between chunks. The documentation recommends iter_content for streaming: “When streaming a download, the above is the preferred and recommended way to retrieve the content.” For ordinary downloads that advice is right. For a deadline it has a catch, which the first half of the next experiment measures. The second half tests the helper you are about to write.

The catch is simple. If the loop is waiting inside the call that fetches the next chunk, it cannot check the clock. For a response with a known length, as in this experiment, iter_content(chunk_size=8192) does not hand over a chunk until it has 8,192 bytes or the body ends. With a slow drip, that can be most of the download.

Write the Budget class

A budget is a deadline object that you create once and pass around. Save the following as deadline.py. This is the first part of the file. Python’s documentation describes time.monotonic() as “a monotonic clock, i.e. a clock that cannot go backwards”, which is what you want here, because it does not jump when someone adjusts the system clock. The clock parameter exists only so that tests can replace the clock.

"""deadline.py: one time budget for a whole operation, and a requests helper that honours it."""
import time

import requests
import urllib3


class DeadlineExceeded(TimeoutError):
    """The whole operation ran out of time."""


class TooLarge(Exception):
    """The body is bigger than we are willing to read."""


class Budget:
    """A deadline that several calls can share. It counts on the monotonic clock."""

    def __init__(self, seconds, *, clock=time.monotonic):
        self.seconds = seconds
        self._clock = clock
        self._end = clock() + seconds

    def remaining(self):
        return max(0.0, self._end - self._clock())

    def expired(self):
        return self._clock() >= self._end

    def timeout(self, cap=None, floor=0.05):
        """Seconds to give the next network step: the cap, or what is left if that is less."""
        left = self.remaining()
        if left <= floor:
            raise DeadlineExceeded(f"the {self.seconds:g} s budget is used up")
        return left if cap is None else min(cap, left)

remaining() never returns less than zero. timeout(cap, floor) is the method the rest of the tutorial leans on. It answers the question of how long the next network step may take, with the smaller of two numbers: your usual limit for that step (the cap) and the time that is left. If almost nothing is left (less than the floor, 0.05 seconds), it raises DeadlineExceeded instead of starting a call that cannot finish. DeadlineExceeded inherits from the built-in TimeoutError, so code that handles timeouts in general also handles it.

Write the download helper

Now add the second part to deadline.py, directly below the first.

# deadline.py (continued)
def get_with_deadline(url, budget, *, connect=1.0, read=1.0, max_bytes=1_000_000):
    """GET url and return the body. Never runs past the budget by more than `read` seconds."""
    timeout = (budget.timeout(connect), budget.timeout(read))
    body = bytearray()
    try:
        with requests.get(url, stream=True, timeout=timeout) as resp:
            resp.raise_for_status()
            while True:
                # read1 returns as soon as ANY bytes arrive; iter_content(8192) would wait for 8192
                chunk = resp.raw.read1(16384, decode_content=True)
                if not chunk:
                    return bytes(body)
                body.extend(chunk)
                if len(body) > max_bytes:
                    raise TooLarge(f"more than {max_bytes} bytes")
                if budget.expired():
                    raise DeadlineExceeded(f"the {budget.seconds:g} s budget ran out after {len(body)} bytes")
    except urllib3.exceptions.ReadTimeoutError as exc:  # raw.read1 bypasses requests' wrapping
        raise requests.ReadTimeout(str(exc)) from exc
    except urllib3.exceptions.HTTPError as exc:
        raise requests.ConnectionError(str(exc)) from exc

Go through it from top to bottom:

  1. The connect and read timeouts passed to requests come from the budget, so neither of them can be longer than what is left of it.
  2. stream=True tells requests not to download the body inside requests.get. The with block closes the connection when the function ends, including when it ends with an exception.
  3. resp.raw.read1(16384, decode_content=True) is the key line. The urllib3 documentation describes read1 as “Similar to http.client.HTTPResponse.read1 and documented in io.BufferedReader.read1(), but with an additional parameter: decode_content.” In plain terms, it returns as soon as any data has arrived, up to 16,384 bytes, instead of waiting for a full chunk.
  4. After each piece, the loop checks the size cap (max_bytes) and the budget.
  5. Because raw.read1 bypasses the wrapping that requests does for iter_content, the urllib3 errors would leak out. The last two except clauses turn them into the requests exceptions your callers already handle.

Test the idea and the helper

Save this as step04_deadline.py. The first group of lines asks when each style of loop gets its first chunk from a six-byte, one-byte-per-second drip. The second group runs get_with_deadline in five situations. The httpx line is there to show that its streaming interface already behaves the way you want.

"""step04_deadline.py: a deadline for the WHOLE download, using requests."""
import httpx
import requests

from deadline import Budget, get_with_deadline
from slowserver import serve_in_thread
from timing import parallel

httpd, base = serve_in_thread()
client = httpx.Client()


# What the obvious loop does: iter_content(8192) waits for 8192 bytes or the end of the body.
def first_chunk(chunk_size):
    with requests.get(base + "/drip?n=6&gap=1", stream=True, timeout=(1, 3)) as resp:
        for chunk in resp.iter_content(chunk_size=chunk_size):
            return f"first chunk has {len(chunk)} byte(s)"


def first_read1():
    with requests.get(base + "/drip?n=6&gap=1", stream=True, timeout=(1, 3)) as resp:
        return f"first chunk has {len(resp.raw.read1(16384))} byte(s)"


def first_httpx_chunk():
    with client.stream("GET", base + "/drip?n=6&gap=1", timeout=3) as resp:
        for chunk in resp.iter_bytes():
            return f"first chunk has {len(chunk)} byte(s)"


print("6 bytes, one per second: when does the loop get its first chunk?")
parallel([
    ("iter_content(chunk_size=8192)", lambda: first_chunk(8192)),
    ("iter_content(chunk_size=None)", lambda: first_chunk(None)),
    ("iter_content(chunk_size=1)", lambda: first_chunk(1)),
    ("resp.raw.read1(16384)", first_read1),
    ("httpx stream, iter_bytes()", first_httpx_chunk),
])

print()
print("get_with_deadline(url, Budget(seconds), read=...):")


def go(path, seconds, read):
    return len(get_with_deadline(base + path, Budget(seconds), read=read))


parallel([
    ("fast answer, budget 3", lambda: go("/fast", 3, 1.0)),
    ("10 bytes 0.5 s apart, budget 3, read 1", lambda: go("/drip?n=10&gap=0.5", 3, 1.0)),
    ("10 bytes 0.5 s apart, budget 8, read 1", lambda: go("/drip?n=10&gap=0.5", 8, 1.0)),
    ("silent for 7 s, budget 3, read 1", lambda: go("/stall?seconds=7", 3, 1.0)),
    ("5 bytes 1.8 s apart, budget 3, read 2", lambda: go("/drip?n=5&gap=1.8", 3, 2.0)),
])

Run it with python step04_deadline.py:

6 bytes, one per second: when does the loop get its first chunk?
iter_content(chunk_size=8192)   6.0 s  ok: first chunk has 6 byte(s)
iter_content(chunk_size=None)   6.0 s  ok: first chunk has 6 byte(s)
iter_content(chunk_size=1)      1.0 s  ok: first chunk has 1 byte(s)
resp.raw.read1(16384)           1.0 s  ok: first chunk has 1 byte(s)
httpx stream, iter_bytes()      1.0 s  ok: first chunk has 1 byte(s)

get_with_deadline(url, Budget(seconds), read=...):
fast answer, budget 3                    0.0 s  ok: 2
10 bytes 0.5 s apart, budget 3, read 1   3.0 s  deadline.DeadlineExceeded
10 bytes 0.5 s apart, budget 8, read 1   5.0 s  ok: 10
silent for 7 s, budget 3, read 1         1.0 s  requests.exceptions.ReadTimeout
5 bytes 1.8 s apart, budget 3, read 2    3.6 s  deadline.DeadlineExceeded

The first group confirms the catch. iter_content(chunk_size=8192) and iter_content(chunk_size=None) returned their first chunk only after 6.0 seconds, with all six bytes in it, so a clock check between chunks would have come too late. With chunk_size=1 the first byte arrived after 1.0 second, but a loop that runs once per byte is no way to download a large file. raw.read1 and httpx’s iter_bytes() both returned after 1.0 second with one byte.

The second group shows the helper at work:

  • A fast answer came back in 0.0 seconds.
  • Ten bytes 0.5 seconds apart with a 3-second budget: the helper stopped at 3.0 seconds with DeadlineExceeded.
  • The same drip with an 8-second budget finished normally at 5.0 seconds with all ten bytes.
  • A server silent for 7 seconds with a 3-second budget and a 1-second read limit: ReadTimeout at 1.0 second. The read limit fired first, because the silence was longer than the read limit.
  • Five bytes 1.8 seconds apart with a 3-second budget and a 2-second read limit: DeadlineExceeded at 3.6 seconds. The bytes arrived at 1.8 and 3.6 seconds, and the budget check only runs when a byte arrives, so the call overran its budget by 0.6 seconds.

The last case shows the rule for this helper: it can run past the budget by less than one read timeout. The check happens between reads, and a single read can wait for at most the read timeout. So choose a read timeout that is small compared with the budget (for example a quarter of it) but larger than the longest normal pause in a healthy response. Do not count on the read timeout to cover DNS lookups, which the urllib3 documentation says are outside the socket timeout.

max_bytes is a separate safety net, a limit on size rather than time. It counts bytes after decoding. A compressed body that expands enormously needs more care than this; see the decompression bomb tutorial on this site.

How to confirm Step 4 worked

The two iter_content lines that wait for a full chunk must show about 6.0 seconds, and the other three about 1.0 second. In the second group, the 3-second budget must stop at 3.0 and 3.6 seconds, never at 5 or 10.

Step 5: Bound Async Calls With asyncio.timeout and Cancellation

The tools

In asynchronous code the standard library already has a whole-operation limit. asyncio.timeout(delay) is “an asynchronous context manager that can be used to limit the amount of time spent waiting on something.” When the time is up, it cancels the work inside the block: “the context manager will cancel the current task and handle the resulting asyncio.CancelledError internally, transforming it into a TimeoutError which can be caught and handled.” asyncio.timeout_at(when) does the same with an absolute time on the event loop’s clock, which is convenient when one deadline has to be passed through several functions. Both were added in Python 3.11.

One rule from the documentation matters for error handling: “the asyncio.timeout() context manager is what transforms the asyncio.CancelledError into a TimeoutError, which means the TimeoutError can only be caught outside of the context manager.”

Try it, including a case that fails

Save this as step05_async_deadline.py. It calls the ten-second drip five ways, then makes one more call that goes wrong on purpose: a blocking requests.get inside asyncio.timeout(1), a mistake you will meet in real code. The script ends by asking the server how many clients it saw leave.

"""step05_async_deadline.py: asyncio.timeout puts a hard limit on the whole call and cancels it."""
import asyncio
import time

import aiohttp
import httpx
import requests

from slowserver import events, serve_in_thread

httpd, base = serve_in_thread()
url = base + "/drip?n=10&gap=1"  # one byte per second for 10 seconds


async def show(label, make_call):
    t0 = time.perf_counter()
    try:
        outcome = "ok: " + str(await make_call())
    except BaseException as exc:
        module = type(exc).__module__
        outcome = type(exc).__qualname__ if module == "builtins" else f"{module}.{type(exc).__qualname__}"
    print(f"{label:<44}  {time.perf_counter() - t0:4.1f} s  {outcome}")


async def main():
    async with httpx.AsyncClient(timeout=httpx.Timeout(5.0)) as client:
        async def plain():
            return len((await client.get(url)).content)

        async def with_timeout():
            async with asyncio.timeout(3):
                return len((await client.get(url)).content)

        async def with_timeout_at():
            deadline = asyncio.get_running_loop().time() + 3
            async with asyncio.timeout_at(deadline):
                return len((await client.get(url)).content)

        async def caught():
            try:
                async with asyncio.timeout(3):
                    await client.get(url)
            except (TimeoutError, httpx.TimeoutException) as exc:
                return f"caught {type(exc).__name__} in one except clause"

        await show("httpx timeout=5, no deadline", plain)
        await show("asyncio.timeout(3) around the call", with_timeout)
        await show("asyncio.timeout_at(now + 3)", with_timeout_at)
        await show("except (TimeoutError, httpx.TimeoutException)", caught)

    async def aiohttp_total():
        async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=3)) as session:
            async with session.get(url) as resp:
                return len(await resp.read())

    await show("aiohttp ClientTimeout(total=3)", aiohttp_total)

    async def blocking_inside():
        async with asyncio.timeout(1):
            return requests.get(base + "/stall?seconds=4").text  # a blocking call, no await inside

    await show("asyncio.timeout(1) around a blocking call", blocking_inside)


asyncio.run(main())
time.sleep(1.5)  # give the server time to notice that the clients left
gone = sorted(d["seconds"] for _, _, d in events("client_gone"))
print(f"{len(gone)} clients left at the 3 second mark; the server noticed on its writes at these seconds: {gone}")

Run it with python step05_async_deadline.py. It takes about half a minute:

httpx timeout=5, no deadline                  10.0 s  ok: 10
asyncio.timeout(3) around the call             3.0 s  TimeoutError
asyncio.timeout_at(now + 3)                    3.0 s  TimeoutError
except (TimeoutError, httpx.TimeoutException)   3.0 s  ok: caught TimeoutError in one except clause
aiohttp ClientTimeout(total=3)                 3.0 s  TimeoutError
asyncio.timeout(1) around a blocking call      4.0 s  ok: done
4 clients left at the 3 second mark; the server noticed on its writes at these seconds: [4.0, 4.0, 4.0, 4.0]

Line by line:

  • With only a client timeout of 5 seconds, the call took 10.0 seconds, the same lesson as Step 3.
  • asyncio.timeout(3) and asyncio.timeout_at(now + 3) both ended the call at 3.0 seconds with TimeoutError. Use the loop’s own clock for timeout_at (loop.time()), not time.time().
  • The fourth line shows the exception question. asyncio.timeout raises the built-in TimeoutError, while httpx raises its own TimeoutException family, which is not that class. One except (TimeoutError, httpx.TimeoutException) clause covers both.
  • aiohttp with ClientTimeout(total=3) ended at 3.0 seconds without needing asyncio.timeout.
  • The last case is the trap. asyncio.timeout(1) wrapped around a blocking call took 4.0 seconds and no TimeoutError appeared. Cancellation only takes effect when the task reaches an await. A blocking call has none, so nothing could interrupt it, and the event loop could not run anything else for those four seconds. Inside async def, use an async client, or run the blocking client in a thread.
  • The final line shows what cancellation does to the other side. The four cancelled calls closed their connections at the 3-second mark, and the server noticed on a failed write one or two seconds later, because it writes once per second (4.0 seconds for all four in the output above, and 4.0 or 5.0 in another run). A timeout that cancels the call also lets the dependency find out that nobody is listening any more.

How to confirm Step 5 worked

Three lines must end at 3.0 seconds with TimeoutError (or the caught message), the plain call must take about 10 seconds, and the blocking case must take about 4.0 seconds with ok: done. If the blocking case ends in a TimeoutError for you, check that the call really is blocking and not an async client.

Step 6: A Timeout on the Waiter Is Not a Timeout on the Work

The problem

Many programs push a blocking call into a thread pool and wait for the result with a timeout, such as future.result(timeout=2). The documentation for that method says: “If the call hasn’t completed in timeout seconds, then a TimeoutError will be raised.” It does not say that the call stops, and it does not stop. The thread keeps running the work. Python has no supported way to stop a running thread from the outside.

See it happen

Save this as step06_threads.py. Case A puts the timeout only on the waiting side. Case B puts it inside the work, as a requests timeout.

"""step06_threads.py: a timeout on the waiter is not a timeout on the work."""
import time
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import TimeoutError as FutureTimeout

import requests

from slowserver import serve_in_thread

httpd, base = serve_in_thread()
url = base + "/stall?seconds=6"


def run(label, work, wait):
    t0 = time.perf_counter()
    with ThreadPoolExecutor(max_workers=1) as pool:
        future = pool.submit(work)
        try:
            future.result(timeout=wait)
        except FutureTimeout:
            print(f"{label}: the caller gave up after {time.perf_counter() - t0:.1f} s,"
                  f" worker still running: {future.running()}")
        except requests.Timeout as exc:
            print(f"{label}: the worker raised {type(exc).__name__} after {time.perf_counter() - t0:.1f} s")
    print(f"{label}: left the with block after {time.perf_counter() - t0:.1f} s")


run("A) waiter timeout only ", lambda: requests.get(url).text, wait=2)
run("B) timeout inside work ", lambda: requests.get(url, timeout=2).text, wait=3)

Run it with python step06_threads.py:

A) waiter timeout only : the caller gave up after 2.0 s, worker still running: True
A) waiter timeout only : left the with block after 6.0 s
B) timeout inside work : the worker raised ReadTimeout after 2.0 s
B) timeout inside work : left the with block after 2.0 s

In case A the caller printed its give-up message after 2.0 seconds, and the worker was still running. Then the program stayed at the end of the with block until 6.0 seconds, when the server finally answered. The reason is in the concurrent.futures documentation. Leaving a with block shuts the executor down “as if Executor.shutdown() were called with wait set to True”, and “If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed.” Even without the with block, “Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing.”

In case B the requests timeout lives inside the work, so the worker thread itself gave up after 2.0 seconds and the block exited at once. The lesson is where to put the limit: on the operation that blocks (the socket), not only on whoever waits for it. A timeout on the waiter is a good second line of defense, because it lets the caller move on and report an error, but it frees nothing.

How to confirm Step 6 worked

Case A must show the caller giving up at 2.0 seconds and leaving the block at 6.0 seconds. Case B must show 2.0 seconds twice.

Step 7: Share One Budget Across Several Calls and Their Retries

The problem

Per-call timeouts add up. A page that needs a profile, a list of orders and some recommendations makes three calls. If each call has a 2-second timeout and the dependency is slow, the page can take 6 seconds, even though each call obeyed its limit. Add retries and the worst case grows again. Your user sees only the total.

Add retries that respect the budget

Now add the third part to deadline.py, below the second.

# deadline.py (continued)
def call_with_retries(fn, budget, *, attempts=3, per_try=1.0, pause=0.2,
                      retry_on=(requests.Timeout, requests.ConnectionError)):
    """Call fn(timeout) up to `attempts` times, but only while the budget has time for another try."""
    last = None
    for attempt in range(1, attempts + 1):
        try:
            return fn(budget.timeout(per_try))
        except DeadlineExceeded:
            raise
        except retry_on as exc:
            last = exc
            if attempt == attempts or budget.remaining() <= pause:
                break
            time.sleep(pause)
    raise last

The function calls fn(timeout) up to attempts times. Each attempt gets budget.timeout(per_try), which is the per-try limit or whatever is left of the budget, whichever is smaller. The loop stops when no more than one pause is left in the budget, and it re-raises DeadlineExceeded at once instead of retrying it. Only the errors listed in retry_on are retried, by default network trouble; a ValueError from a bug passes straight through.

Two cautions. Retry only operations that are safe to repeat: reads, or writes protected by an idempotency key (see the idempotency keys tutorial). And this helper waits a fixed pause between tries; the retry tutorial adds exponential backoff and jitter.

Compare per-call timeouts with one shared budget

Save this as step07_budget.py. The dependency answers after 4 seconds, which is too slow for every scenario. Scenario A gives each of three calls its own 2-second timeout. Scenario B gives all three calls one 3-second Budget. Scenario C retries one call with a 2.5-second budget.

"""step07_budget.py: three calls and some retries must share one time budget."""
import time

import requests

from deadline import Budget, DeadlineExceeded, call_with_retries
from slowserver import serve_in_thread

httpd, base = serve_in_thread()
slow = base + "/stall?seconds=4"  # a dependency that answers after 4 s, which is too slow for us


def describe(exc):
    return type(exc).__qualname__


def page_without_budget():
    for name in ("profile", "orders", "recommendations"):
        try:
            requests.get(slow, timeout=2)
        except requests.Timeout as exc:
            print(f"   {name:<16} gave up with {describe(exc)}")


def page_with_budget():
    budget = Budget(3.0)
    for name in ("profile", "orders", "recommendations"):
        try:
            requests.get(slow, timeout=budget.timeout(cap=2))
        except (requests.Timeout, DeadlineExceeded) as exc:
            print(f"   {name:<16} gave up with {describe(exc)}   ({budget.remaining():.1f} s left)")


def timed(title, fn):
    t0 = time.perf_counter()
    print(title)
    fn()
    print(f"   -> the page took {time.perf_counter() - t0:.1f} s\n")


timed("A) three calls, each with timeout=2 and no shared budget", page_without_budget)
timed("B) the same three calls sharing one 3 second Budget", page_with_budget)

tries = []


def one_try(timeout):
    tries.append(round(timeout, 2))
    return requests.get(slow, timeout=timeout).text


t0 = time.perf_counter()
try:
    call_with_retries(one_try, Budget(2.5), attempts=4, per_try=1.0, pause=0.25)
except requests.Timeout as exc:
    print(f"C) 4 attempts allowed, 2.5 s budget: {len(tries)} attempts with timeouts {tries},"
          f" then {describe(exc)} after {time.perf_counter() - t0:.1f} s")

Run it with python step07_budget.py:

A) three calls, each with timeout=2 and no shared budget
   profile          gave up with ReadTimeout
   orders           gave up with ReadTimeout
   recommendations  gave up with ReadTimeout
   -> the page took 6.1 s

B) the same three calls sharing one 3 second Budget
   profile          gave up with ReadTimeout   (1.0 s left)
   orders           gave up with ReadTimeout   (0.0 s left)
   recommendations  gave up with DeadlineExceeded   (0.0 s left)
   -> the page took 3.0 s

C) 4 attempts allowed, 2.5 s budget: 2 attempts with timeouts [1.0, 1.0], then ReadTimeout after 2.3 s

Scenario A took 6.0 seconds: three calls, 2.0 seconds each, every one within its limit. In scenario B the first call used its 2.0 seconds and left 1.0 second. The second call was given only 1.0 second and used it. The third call never touched the network, because budget.timeout raised DeadlineExceeded before requests.get was called. The page took 3.0 seconds in total. In scenario C, four attempts were allowed but the budget paid for two. The loop stopped because less than one 0.25-second pause was left after the second attempt, and it raised the last ReadTimeout after 2.3 seconds, inside the 2.5-second budget.

How to confirm Step 7 worked

Scenario A must take about 6 seconds and scenario B about 3, with the third call reporting DeadlineExceeded. Scenario C must report two attempts, each with a 1.0 second timeout.

Step 8: Use the Pieces in a FastAPI Service That Answers 504

Why services need this most

A service is where slow dependencies hurt most, because every stuck call also holds a worker. FastAPI runs the two kinds of endpoint differently. Its documentation says: “When you declare a path operation function with normal def instead of async def, it is run in an external threadpool that is then awaited, instead of being called directly (as it would block the server).” So a plain def endpoint that makes a blocking call occupies one thread of that pool. The script below prints the pool size on this setup. If the dependency stalls, a pool of that size is used up by that many stuck calls.

When the dependency is too slow, the honest answer is a gateway error. RFC 9110 defines it: “The 504 (Gateway Timeout) status code indicates that the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request.”

Three endpoints, one slow dependency

Save this as step08_service.py. The dependency needs 8 seconds, sending one byte per second. The service has three endpoints. /async-deadline uses asyncio.timeout with httpx.AsyncClient. /sync-no-deadline is a plain def endpoint that calls requests.get(DRIP, timeout=(1, 2)), which looks careful. /sync-deadline is a plain def endpoint that uses the get_with_deadline helper from Step 4. The script starts the service on port 8766 in a background thread, calls all three endpoints at once, waits for the slow handler to finish, and then prints what each handler did and what the dependency saw.

"""step08_service.py: the same slow dependency behind a small FastAPI service."""
import asyncio
import threading
import time

import anyio
import httpx
import requests
import uvicorn
from anyio import to_thread
from fastapi import FastAPI
from fastapi.responses import JSONResponse

from deadline import Budget, DeadlineExceeded, get_with_deadline
from slowserver import events, serve_in_thread
from timing import parallel


async def pool_size():
    return to_thread.current_default_thread_limiter().total_tokens


print("worker threads FastAPI gives plain def endpoints by default:", anyio.run(pool_size))
_, dependency = serve_in_thread()
DRIP = dependency + "/drip?n=8&gap=1"  # this dependency needs 8 seconds, one byte per second
NOTES = []  # what each handler did, with the seconds since its request began
app = FastAPI()


def note(text, t0):
    NOTES.append((round(time.perf_counter() - t0, 1), text))


@app.get("/async-deadline")
async def async_deadline():
    t0 = time.perf_counter()
    try:
        async with asyncio.timeout(2.5):
            async with httpx.AsyncClient(timeout=5.0) as client:
                reply = await client.get(DRIP)
        return {"bytes": len(reply.content)}
    except TimeoutError:
        note("async handler: deadline hit, outgoing call cancelled", t0)
        return JSONResponse({"error": "dependency too slow"}, status_code=504)


@app.get("/sync-no-deadline")
def sync_no_deadline():
    t0 = time.perf_counter()
    reply = requests.get(DRIP, timeout=(1, 2))  # 2 s of silence allowed, bytes arrive every 1 s
    note("sync handler with no deadline: finished the download", t0)
    return {"bytes": len(reply.content)}


@app.get("/sync-deadline")
def sync_deadline():
    t0 = time.perf_counter()
    try:
        return {"bytes": len(get_with_deadline(DRIP, Budget(2.5), read=2.0))}
    except (DeadlineExceeded, requests.Timeout):
        note("sync handler with a deadline: gave up, thread freed", t0)
        return JSONResponse({"error": "dependency too slow"}, status_code=504)


server = uvicorn.Server(uvicorn.Config(app, host="127.0.0.1", port=8766, log_level="warning"))
threading.Thread(target=server.run, daemon=True).start()
while not server.started:
    time.sleep(0.05)
service = "http://127.0.0.1:8766"
caller = httpx.Client()


def call(path, patience):
    reply = caller.get(service + path, timeout=patience)
    return f"HTTP {reply.status_code} {reply.text}"


print("three callers hit three endpoints; the dependency needs 8 s")
parallel([
    ("GET /async-deadline (patience 5 s)", lambda: call("/async-deadline", 5)),
    ("GET /sync-deadline (patience 5 s)", lambda: call("/sync-deadline", 5)),
    ("GET /sync-no-deadline (patience 4 s)", lambda: call("/sync-no-deadline", 4)),
])
time.sleep(5)  # let the slow handler run to the end
print("\nwhat the handlers did:")
for seconds, text in sorted(NOTES):
    print(f"  {seconds:4.1f} s  {text}")
print("what the dependency saw:")
for _, kind, details in sorted(events(), key=lambda e: e[2].get("seconds", 0)):
    print(f"  {details['seconds']:4.1f} s  {kind}")
server.should_exit = True

Run it with python step08_service.py:

worker threads FastAPI gives plain def endpoints by default: 40
three callers hit three endpoints; the dependency needs 8 s
GET /async-deadline (patience 5 s)     2.5 s  ok: HTTP 504 {"error":"dependency too slow"}
GET /sync-deadline (patience 5 s)      3.2 s  ok: HTTP 504 {"error":"dependency too slow"}
GET /sync-no-deadline (patience 4 s)   4.0 s  httpx.ReadTimeout

what the handlers did:
   2.5 s  async handler: deadline hit, outgoing call cancelled
   3.0 s  sync handler with a deadline: gave up, thread freed
   8.0 s  sync handler with no deadline: finished the download
what the dependency saw:
   4.0 s  client_gone
   5.0 s  client_gone
   8.0 s  drip_finished

The first line says FastAPI gave plain def endpoints 40 worker threads here. After that, each endpoint behaved differently:

  • /async-deadline answered HTTP 504 after 2.5 seconds. The handler’s log line confirms that the deadline fired and the outgoing call was cancelled.
  • /sync-deadline answered 504 after 3.3 seconds as seen by the caller, and its handler gave up at 3.0 seconds, which was the first moment a byte arrived after the 2.5-second budget ran out. The thread was freed.
  • /sync-no-deadline is the problem case. Its caller was only willing to wait 4 seconds and left with httpx.ReadTimeout, but the handler carried on and finished the download at 8.0 seconds. That is Step 3 again, inside a service: two seconds of allowed silence never fire when a byte arrives every second. For another 4 seconds after the caller left, one thread of the pool did work nobody was waiting for.

The dependency’s view matches: two clients left early (the cancelled async call and the sync call that gave up), and one full download ran to its end at 8.0 seconds. A service full of endpoints like /sync-no-deadline can turn a slow dependency into an outage of its own. For the error body, use whatever format your API already has; the RFC 9457 tutorial shows a standard one.

How to confirm Step 8 worked

Look for 504 at about 2.5 and 3 seconds, a ReadTimeout for the no-deadline endpoint at 4.0 seconds, and a handler line at 8.0 seconds for it. If the service cannot start, check that port 8766 is free.

Step 9: Test the Helper and Prove the Tests Can Fail

Write the tests

Save this as test_deadline.py. The first three tests check the arithmetic of Budget with a fake clock, so they need no sleeping, and the fourth checks that DeadlineExceeded is a TimeoutError. The next five use the slow server with short waits to check the helper against a real socket. The last four test call_with_retries with small fake functions, and the final one replaces time.sleep to count the pauses.

# test_deadline.py
import time

import pytest
import requests

from deadline import Budget, DeadlineExceeded, TooLarge, call_with_retries, get_with_deadline
from slowserver import serve_in_thread


@pytest.fixture(scope="module")
def base():
    httpd, url = serve_in_thread()
    yield url
    httpd.shutdown()


class FakeClock:
    def __init__(self):
        self.now = 100.0

    def __call__(self):
        return self.now


def test_budget_counts_down_and_never_goes_negative():
    clock = FakeClock()
    budget = Budget(5, clock=clock)
    assert budget.remaining() == 5
    clock.now += 2
    assert budget.remaining() == 3
    clock.now += 10
    assert budget.remaining() == 0
    assert budget.expired()


def test_timeout_is_the_smaller_of_cap_and_remaining():
    clock = FakeClock()
    budget = Budget(5, clock=clock)
    assert budget.timeout(cap=2) == 2
    clock.now += 4
    assert budget.timeout(cap=2) == pytest.approx(1)
    assert budget.timeout() == pytest.approx(1)


def test_timeout_refuses_when_almost_nothing_is_left():
    clock = FakeClock()
    budget = Budget(5, clock=clock)
    clock.now += 4.97
    with pytest.raises(DeadlineExceeded):
        budget.timeout(cap=2)


def test_deadline_exceeded_is_a_timeout_error():
    assert issubclass(DeadlineExceeded, TimeoutError)


def test_fast_answer(base):
    assert get_with_deadline(base + "/fast", Budget(2)) == b"ok"


def test_slow_answer_inside_the_budget(base):
    assert get_with_deadline(base + "/drip?n=4&gap=0.1", Budget(3), read=1) == b"xxxx"


def test_drip_past_the_deadline_stops_near_the_deadline(base):
    t0 = time.perf_counter()
    with pytest.raises(DeadlineExceeded):
        get_with_deadline(base + "/drip?n=40&gap=0.1", Budget(1.0), read=0.5)
    assert time.perf_counter() - t0 < 1.0 + 0.5  # budget plus at most one read timeout


def test_silence_longer_than_the_read_timeout_is_a_read_timeout(base):
    with pytest.raises(requests.ReadTimeout):
        get_with_deadline(base + "/stall?seconds=3", Budget(5), read=0.3)


def test_body_cap(base):
    with pytest.raises(TooLarge):
        get_with_deadline(base + "/drip?n=50&gap=0", Budget(5), max_bytes=10)


def test_retries_return_the_first_success():
    state = {"calls": 0}

    def fn(timeout):
        state["calls"] += 1
        if state["calls"] < 3:
            raise requests.ConnectionError("boom")
        return "fine"

    assert call_with_retries(fn, Budget(5), attempts=3, pause=0.01) == "fine"
    assert state["calls"] == 3


def test_retries_do_not_hide_other_errors():
    calls = []

    def fn(timeout):
        calls.append(timeout)
        raise ValueError("bug")

    with pytest.raises(ValueError):
        call_with_retries(fn, Budget(5), attempts=3, pause=0.01)
    assert len(calls) == 1


def test_retries_stop_when_the_budget_cannot_cover_another_try():
    calls = []

    def fn(timeout):
        calls.append(round(timeout, 1))
        time.sleep(0.5)  # every try uses up half a second
        raise requests.ReadTimeout()

    with pytest.raises(requests.ReadTimeout):
        call_with_retries(fn, Budget(1.0), attempts=5, per_try=0.5, pause=0.3)
    assert len(calls) == 2


def test_an_expired_budget_is_not_retried_even_when_retry_on_is_broad(monkeypatch):
    sleeps = []
    monkeypatch.setattr("deadline.time.sleep", sleeps.append)
    clock = FakeClock()
    budget = Budget(1.0, clock=clock)
    calls = []

    def fn(timeout):
        calls.append(timeout)
        clock.now += 0.98  # leaves 0.02 s, which is below the 0.05 s floor
        raise requests.ReadTimeout()

    with pytest.raises(DeadlineExceeded):
        call_with_retries(fn, budget, attempts=5, pause=0.0, retry_on=(Exception,))
    assert len(calls) == 1
    assert sleeps == [0.0]  # one pause after the first failure, then straight out

Run them with python -m pytest -q test_deadline.py:

............. [100%]
13 passed in 3.35s

Plant bugs on purpose

A test suite that cannot fail proves nothing. Save this as mutants.py. It copies the code to a temporary folder, plants one bug at a time in deadline.py, and checks that at least one test fails for each. It disables Python’s bytecode cache for these runs, so that two planted bugs of the same size cannot reuse a stale cached file.

"""mutants.py: plant one bug at a time in deadline.py and check that the tests notice."""
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path

HERE = Path(__file__).parent
SOURCE = (HERE / "deadline.py").read_text(encoding="utf-8")

CHECK = '''                if budget.expired():
                    raise DeadlineExceeded(f"the {budget.seconds:g} s budget ran out after {len(body)} bytes")
'''
CAP = '''                if len(body) > max_bytes:
                    raise TooLarge(f"more than {max_bytes} bytes")
'''

MUTANTS = [
    ("M1 the download loop never checks the budget", CHECK, ""),
    ("M2 timeout() takes the larger of cap and remaining", "min(cap, left)", "max(cap, left)"),
    ("M3 no cap on the body size", CAP, ""),
    ("M4 retries ignore the budget", "if attempt == attempts or budget.remaining() <= pause:", "if attempt == attempts:"),
    ("M5 the floor is ignored", "if left <= floor:", "if left <= 0:"),
    ("M6 retries also retry DeadlineExceeded", "        except DeadlineExceeded:\n            raise\n", ""),
]


def run(folder):
    env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1", "COLUMNS": "150"}
    proc = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "test_deadline.py"],
                          cwd=folder, capture_output=True, text=True, env=env, timeout=300)
    failed = [line.split("::")[1].split(" ")[0] for line in proc.stdout.splitlines() if line.startswith("FAILED")]
    return proc.returncode, failed


with tempfile.TemporaryDirectory() as tmp:
    for name in ("deadline.py", "slowserver.py", "test_deadline.py"):
        shutil.copy(HERE / name, tmp)
    code, failed = run(tmp)
    print(f"original code: exit {code}, failing tests: {len(failed)}")
    for label, old, new in MUTANTS:
        assert SOURCE.count(old) == 1, label
        (Path(tmp) / "deadline.py").write_text(SOURCE.replace(old, new), encoding="utf-8")
        code, failed = run(tmp)
        verdict = "caught" if code != 0 else "SURVIVED"
        print(f"{label}: {verdict} ({len(failed)} failing: {', '.join(failed)[:90]})")

Run it with python mutants.py. It takes about 35 seconds:

original code: exit 0, failing tests: 0
M1 the download loop never checks the budget: caught (1 failing: test_drip_past_the_deadline_stops_near_the_deadline)
M2 timeout() takes the larger of cap and remaining: caught (2 failing: test_timeout_is_the_smaller_of_cap_and_remaining, test_silence_longer_than_the_read_timeou)
M3 no cap on the body size: caught (1 failing: test_body_cap)
M4 retries ignore the budget: caught (1 failing: test_retries_stop_when_the_budget_cannot_cover_another_try)
M5 the floor is ignored: caught (2 failing: test_timeout_refuses_when_almost_nothing_is_left, test_an_expired_budget_is_not_retried_ev)
M6 retries also retry DeadlineExceeded: caught (1 failing: test_an_expired_budget_is_not_retried_even_when_retry_on_is_broad)

All six planted bugs were caught, each by the test that guards that behavior. The sixth one is worth a story. My first version of the suite had twelve tests, and this bug survived: the line it removes (except DeadlineExceeded: raise) makes no visible difference with the default retry_on, because DeadlineExceeded is not in that tuple. It matters when a caller passes a broad retry_on, such as (Exception,), because the loop would then retry an exhausted budget and keep pausing. The thirteenth test, which uses a broad retry_on and counts pauses, closes that gap. Passing tests do not prove the helper is correct, but a suite that notices these six mistakes is a real safety net.

Run the whole lab at once

The last file runs every command shown in this tutorial and saves each output, which is handy when you change a number and want to see everything again. Save it as run_all.py and run python run_all.py. It takes about two minutes.

"""run_all.py [stems]: run every command shown in the article and save the output to out/NAME.txt."""
import os
import subprocess
import sys
import time
from pathlib import Path

HERE = Path(__file__).parent
OUT = HERE / "out"
OUT.mkdir(exist_ok=True)

COMMANDS = {
    "versions": ["versions.py"],
    "step01_defaults": ["step01_defaults.py"],
    "step02_timeouts": ["step02_timeouts.py"],
    "step03_slow_drip": ["step03_slow_drip.py"],
    "step04_deadline": ["step04_deadline.py"],
    "step05_async_deadline": ["step05_async_deadline.py"],
    "step06_threads": ["step06_threads.py"],
    "step07_budget": ["step07_budget.py"],
    "step08_service": ["step08_service.py"],
    "pytest": ["-m", "pytest", "-q", "-p", "no:cacheprovider", "test_deadline.py"],
    "mutants": ["mutants.py"],
}

wanted = sys.argv[1:] or list(COMMANDS)
env = {**os.environ, "PYTHONIOENCODING": "utf-8", "PYTHONDONTWRITEBYTECODE": "1", "COLUMNS": "100"}
for name in wanted:
    t0 = time.perf_counter()
    proc = subprocess.run([sys.executable, "-u", *COMMANDS[name]], cwd=HERE, capture_output=True,
                          text=True, encoding="utf-8", env=env, timeout=600)
    text = (proc.stdout + proc.stderr).rstrip() + "\n"
    (OUT / f"{name}.txt").write_text(text, encoding="utf-8", newline="\n")
    print(f"{name}: exit {proc.returncode}, {time.perf_counter() - t0:.1f} s, {len(text.splitlines())} lines")

The Three Layers, in One Place

  1. Per-step timeouts. Set connect and read timeouts on every client, so that no call waits unboundedly for silence. Choose the read timeout from the dependency’s latency data (Step 2).
  2. A deadline for the whole call. A read timeout cannot stop a slow drip. Use asyncio.timeout or aiohttp’s total in async code, and a loop built on read1 with a budget check in synchronous requests code (Steps 3 to 5).
  3. A shared budget and a place to stop. Give each incoming request one budget, spend it across all calls and retries, and answer 504 when it runs out (Steps 7 and 8). Put the timeout on the work and not only on the waiter (Step 6).

Common Mistakes

  • Setting no timeout. Three of the four clients tested here wait as long as the server does (Step 1).
  • Treating one number as a total. timeout=2 means two seconds of silence, not two seconds of work (Step 3).
  • Catching only requests.Timeout. A timeout in the middle of a body arrived as a ConnectionError (Step 2).
  • Using timeout=None “to be safe”. The requests documentation describes it as telling Requests “to wait forever for a response”.
  • Putting a blocking call in async def. asyncio.timeout could not interrupt it, and the event loop was blocked as well (Step 5).
  • Timing out only the waiter. The worker thread kept going and the program waited for it (Step 6).
  • Giving every call its own timeout and nothing more. Three 2-second calls took 6 seconds (Step 7).
  • Setting a timeout below normal latency. You will cut off healthy answers. Amazon’s guidance is to pick the percentile of the dependency’s latency that matches the false-timeout rate you accept (Step 2).
  • Forgetting what a timeout does not cover. DNS and TLS handshakes can sit outside it, according to the urllib3 and Amazon documents quoted above.

How to Confirm the Whole Thing Works

Run python run_all.py and check these results against the outputs in this tutorial: the defaults table (7.0, 7.0, 5.0, 7.0 seconds), the slow drip beating every read timeout (10.0 seconds) except aiohttp’s total (2.0 seconds), the helper stopping at 3.0 and 3.6 seconds, three asyncio.timeout lines at 3.0 seconds and the blocking case at 4.0, the threads case at 6.0 versus 2.0 seconds, scenario A at 6.0 versus scenario B at 3.0 seconds, the three endpoints (504 at 2.5 and 3.3 seconds, no-deadline handler finishing at 8.0), 13 passing tests, and six caught mutants. Then point the helper at one real dependency of your own, with timeouts taken from its real latency, and look at what happens when you block that dependency in a staging environment.

What This Lab Did Not Test

  • It ran on Windows 11 with Python 3.13.14. Socket behavior can differ on Linux and macOS in details. On this Windows machine, for example, a connection to a closed local port took about two seconds to be refused, so the lab never connects to one.
  • Everything except one connect test ran over loopback. Real networks add jitter and packet loss.
  • Chunked transfer encoding was not tested. The iter_content behavior in Step 4 was measured only on a response with a Content-Length header.
  • DNS lookups, TLS handshakes, HTTP/2, proxies and redirects were not exercised. The httpx pool timeout and write timeout were described from its documentation and not measured.
  • Retry features inside libraries were not used. This tutorial retried in its own code so the time budget stayed visible.
  • The 1 to 3 second limits are there to keep the lab fast. They are not recommendations for production.

Next Steps

  • Put a circuit breaker in front of a dependency that keeps timing out, so that callers fail fast instead of each paying the timeout.
  • Use a bulkhead to keep one slow dependency from using all your workers.
  • Add backpressure and load shedding so that your own service refuses work early when it is overloaded.
  • Replace the fixed pause in call_with_retries with the exponential backoff and jitter from the retry tutorial.
  • Pass the remaining budget to the next service in a header you define, so the whole chain shares one deadline.

Sources

Tags:

AsyncioFastAPIHTTPPythonReliability EngineeringTimeouts

Share

No Comment! Be the first one.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts