How to Shut Down a FastAPI Service Without Dropping Requests: Handle the Stop Signal, Drain Before You Close, and Survive a Rolling Restart Behind nginx

Every deploy ends with the same small drama: a running process is told to stop while somebody is in the middle of a request. If the process simply vanishes, that somebody gets an error. If it is told to stop politely, the outcome depends on details most tutorials skip: what the server does with the request that is already running, what it does with the request that arrives a moment later, what happens to work you started “in the background,” and how long anyone is willing to wait.

In this tutorial you will stop a small FastAPI service on purpose, again and again, and measure what happens. You will start with a single server and one slow request, then add a time limit for stuck work, then protect work that continues after the response, and finally restart two copies of the service one after the other behind nginx while traffic keeps flowing. At the end you will have a launcher with a drain delay, a test suite that pins the behaviour down, and a set of planted bugs that the suite catches.

What you will learn, in plain words

Four terms come up all the time, so here they are before the first command.

A stop signal is the message an operating system or orchestrator sends to ask a process to finish. On Linux and macOS the usual one is SIGTERM. Kubernetes sends it first and uses a forceful kill only if the process is still there after a deadline, and other process managers follow the same convention. A graceful shutdown is what a process does in response: stop taking new work, finish the work in progress, release resources, and exit. A request that is in flight has been received but not yet answered. To drain a server means to let its in-flight work run out, and in a load-balanced setup it also means making sure the load balancer has stopped sending new work before the process closes its door.

That last point is the one that surprises people. A process can shut down perfectly and still cause errors, because the load balancer and the process learn about the shutdown at different moments. Step 5 is about closing that gap.

Prerequisites

• Operating system: I ran every command on Windows 11 with Python 3.13.14. The code also has a Linux and macOS branch (one line in labkit.py), but I did not run that branch, so treat it as untested.
• Packages: FastAPI 0.162.5 (Starlette 1.10.0), uvicorn 0.54.0, requests 2.34.2, pytest 9.2.0, AnyIO 4.16.0 and Pydantic 2.14.3, all installed from PyPI. Newer versions should behave the same, but the numbers in this article come from these.
• nginx: version 1.30.6. router.py downloads the Windows zip from nginx.org by itself, so you install nothing. On Linux or macOS install nginx with your package manager and make nginx_exe() return its path.
• Knowledge: you can write a FastAPI route and run a Python script. You do not need Docker or Kubernetes; they are only mentioned where their documentation explains why a step matters.

Everything runs on 127.0.0.1. I use the numeric address on purpose: with the server listening on 127.0.0.1, a request to localhost took about 2.1 seconds on this Windows machine, against 0.00 seconds for 127.0.0.1. I did not dig into why.

Step 1: Create the service, the launcher, and the lab kit

You need three small files. The service is an “orders” API in which every part of a shutdown can be observed. The launcher runs it under uvicorn. The lab kit starts and stops server processes the way an orchestrator would.

1.1 The service (app.py)

Read the routes first, because each one exists to test a different thing. /work is an async request that takes as long as you ask. /sync-work is a plain def route, which FastAPI runs in a worker thread. POST /orders simulates a write: it waits, then inserts a row into SQLite. /email/{mode} queues a fake e-mail three different ways. /health and /ready are the two probes an orchestrator uses: health means the process is alive, and ready means it should be sent new requests. The Lifecycle object holds the two facts a shutdown needs: whether we are draining, and which tasks we started ourselves.

# app.py
"""app.py: a small orders service in which every part of a shutdown can be observed."""
import asyncio
import os
import sqlite3
import time
from contextlib import asynccontextmanager

from fastapi import BackgroundTasks, FastAPI, HTTPException, Response
from pydantic import BaseModel

DB_PATH = os.environ.get("LAB_DB", "orders.db")
EVENTS = os.environ.get("LAB_EVENTS", "events.log")
OUTBOX = os.environ.get("LAB_OUTBOX", "outbox.log")
TRACKED_WAIT = float(os.environ.get("LAB_TRACKED_WAIT", "5"))  # how long teardown waits for tracked tasks


def note(message: str) -> None:
    """Append one line to the events file, so the order of things can be read back after the process is gone."""
    with open(EVENTS, "a", encoding="utf-8") as events:
        events.write(f"{time.time():.3f} {message}\n")


class Lifecycle:
    """What the process knows about its own shutdown."""

    def __init__(self) -> None:
        self.draining = False                  # True once a stop signal has arrived and /ready should say no
        self.tasks: set[asyncio.Task] = set()  # work started with spawn(); teardown waits for it

    def spawn(self, coroutine) -> asyncio.Task:
        task = asyncio.create_task(coroutine)
        self.tasks.add(task)                   # a strong reference, and a list teardown can wait on
        task.add_done_callback(self.tasks.discard)
        return task


life = Lifecycle()


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.db = sqlite3.connect(DB_PATH, check_same_thread=False, isolation_level=None)
    app.state.db.execute("create table if not exists orders (id text primary key, at real)")
    note("lifespan startup: database open")
    yield
    note(f"lifespan shutdown begins: {len(life.tasks)} tracked task(s) still running")
    if life.tasks:
        _, abandoned = await asyncio.wait(set(life.tasks), timeout=TRACKED_WAIT)
        note(f"lifespan shutdown: waited for tracked tasks, {len(abandoned)} abandoned")
    app.state.db.close()
    note("lifespan shutdown done: database closed")


app = FastAPI(lifespan=lifespan)


@app.get("/health")
async def health():
    """Liveness: is the process alive? It stays 200 while the process is draining."""
    return {"ok": True}


@app.get("/ready")
async def ready(response: Response):
    """Readiness: should the platform send new requests here? Flips to 503 as soon as draining starts."""
    if life.draining:
        response.status_code = 503
        return {"ready": False}
    return {"ready": True}


@app.get("/work")
async def work(seconds: float = 0.3):
    await asyncio.sleep(seconds)
    return {"pid": os.getpid(), "slept": seconds}


@app.get("/sync-work")
def sync_work(seconds: float = 5.0):
    """A plain def route: FastAPI runs it in a worker thread, where it cannot be cancelled."""
    time.sleep(seconds)
    note(f"sync-work finished its {seconds} s of work")
    return {"pid": os.getpid(), "slept": seconds}


class Order(BaseModel):
    id: str
    work: float = 0.3


@app.post("/orders", status_code=201)
async def create_order(order: Order):
    await asyncio.sleep(order.work)  # stand-in for validation and a call to a payment API
    app.state.db.execute("insert into orders values (?, ?)", (order.id, time.time()))
    return {"id": order.id, "pid": os.getpid()}


async def send_email(tag: str, delay: float) -> None:
    await asyncio.sleep(delay)
    with open(OUTBOX, "a", encoding="utf-8") as outbox:
        outbox.write(f"sent {tag}\n")


@app.post("/email/{mode}", status_code=202)
async def email(mode: str, tag: str, background: BackgroundTasks, delay: float = 1.0):
    """Queue an e-mail three different ways and see which ones survive a shutdown."""
    if mode == "background":
        background.add_task(send_email, tag, delay)      # runs after the response, inside the request
    elif mode == "task":
        asyncio.create_task(send_email(tag, delay))      # fire and forget
    elif mode == "tracked":
        life.spawn(send_email(tag, delay))               # fire and forget, but teardown waits for it
    else:
        raise HTTPException(status_code=404, detail="mode must be background, task or tracked")
    return {"queued": tag, "mode": mode}

Two details are worth noticing now. The lifespan function runs its first half at startup and its second half at shutdown, so it is the natural home for closing the database. And note() writes timestamped lines to a file, so later we can read back exactly when things happened, even after the process is gone.

1.2 The launcher (serve.py)

The launcher builds uvicorn’s Server programmatically so that we can subclass it. For now ignore DrainingServer and --drain-delay; they matter in Step 5. Two other things are happening here. The log format is changed so each line begins with the epoch time, which lets us merge uvicorn’s log with our own events into one timeline. And timeout_graceful_shutdown is passed through from the command line, which Step 3 explains.

# serve.py
"""serve.py: run app.py under uvicorn, with an optional drain delay in front of uvicorn's own graceful shutdown."""
import argparse
import copy
import threading

import uvicorn
from uvicorn.config import LOGGING_CONFIG

import app as service


class DrainingServer(uvicorn.Server):
    drain_delay = 0.0

    def handle_exit(self, sig, frame):
        # First stop signal, drain configured: report "not ready", keep serving, and start the real shutdown later.
        if self.drain_delay > 0 and not service.life.draining:
            service.life.draining = True
            service.note(f"stop signal received: draining for {self.drain_delay} s, /ready now answers 503")
            timer = threading.Timer(self.drain_delay, uvicorn.Server.handle_exit, args=(self, sig, frame))
            timer.daemon = True
            timer.start()
            return
        # No drain configured, or a second signal during the drain (which skips the rest of the wait).
        super().handle_exit(sig, frame)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--port", type=int, default=9001)
    parser.add_argument("--drain-delay", type=float, default=0.0)
    parser.add_argument("--timeout-graceful-shutdown", type=int, default=None)
    parser.add_argument("--access-log", action="store_true")
    args = parser.parse_args()

    # Same log lines as stock uvicorn, but each starts with the epoch time so a timeline can be built later.
    log_config = copy.deepcopy(LOGGING_CONFIG)
    log_config["formatters"]["default"].update(fmt="%(created).3f %(levelprefix)s %(message)s", use_colors=False)
    log_config["formatters"]["access"].update(
        fmt='%(created).3f %(levelprefix)s %(client_addr)s "%(request_line)s" %(status_code)s', use_colors=False)

    config = uvicorn.Config(service.app, host="127.0.0.1", port=args.port, log_config=log_config,
                            access_log=args.access_log, timeout_graceful_shutdown=args.timeout_graceful_shutdown)
    server = DrainingServer(config)
    server.drain_delay = args.drain_delay
    server.run()


if __name__ == "__main__":
    main()

1.3 The lab kit (labkit.py)

The kit starts serve.py as a separate process, waits until /ready answers, and later sends it the stop signal. It also merges the two log sources into a timeline. One Windows detail deserves a sentence: the python.exe inside a virtual environment is a small launcher that starts the real interpreter as a child process (a check with psutil showed one launcher and one child, and the child was the process listening on the port). The kit therefore runs the base interpreter and points it at the virtual environment’s packages, so the process it starts, signals and waits for is the server itself. In a test, Ctrl+Break sent to the launcher reached the server too, so this keeps process ids honest rather than being required for signals. On Windows the stop signal is Ctrl+Break, which uvicorn treats like SIGTERM; on POSIX it is proc.terminate(), which sends SIGTERM.

# labkit.py
"""labkit.py: start and stop lab servers the way an orchestrator does (Windows and POSIX)."""
import os
import signal
import socket
import subprocess
import sys
import time

import requests

HERE = os.path.dirname(os.path.abspath(__file__))
RUN = os.path.join(HERE, "run")
WINDOWS = sys.platform == "win32"
os.makedirs(RUN, exist_ok=True)


def python_for_servers() -> str:
    """On Windows the venv's python.exe is a small launcher that starts the real interpreter as a child process. Run the
    base interpreter and hand it the venv's packages instead, so the process we start, signal and wait for is the server."""
    return os.path.join(sys.base_prefix, "python.exe") if WINDOWS else sys.executable


def start(port: int, *args, name: str, db: str | None = None, env: dict | None = None) -> subprocess.Popen:
    """Run serve.py on a port and return once /ready answers 200. Logs and events go to run/<name>.*"""
    paths = {kind: os.path.join(RUN, f"{name}.{kind}") for kind in ("events", "outbox", "log")}
    for path in paths.values():
        if os.path.exists(path):
            os.remove(path)
    environment = {**os.environ, "PYTHONUNBUFFERED": "1", "PYTHONDONTWRITEBYTECODE": "1",
                   "LAB_DB": os.path.join(RUN, db or f"{name}.db"), "LAB_EVENTS": paths["events"],
                   "LAB_OUTBOX": paths["outbox"], **(env or {})}
    if WINDOWS:
        environment["PYTHONPATH"] = os.path.join(sys.prefix, "Lib", "site-packages")
    with open(paths["log"], "wb") as log:
        proc = subprocess.Popen([python_for_servers(), os.path.join(HERE, "serve.py"), "--port", str(port), *map(str, args)],
                                cwd=HERE, env=environment, stdout=log, stderr=subprocess.STDOUT,
                                creationflags=subprocess.CREATE_NEW_PROCESS_GROUP if WINDOWS else 0)
    proc.paths = paths
    wait_ready(port)
    return proc


def wait_ready(port: int, seconds: float = 20.0) -> None:
    deadline = time.time() + seconds
    while time.time() < deadline:
        try:
            if requests.get(f"http://127.0.0.1:{port}/ready", timeout=0.5).status_code == 200:
                return
        except requests.RequestException:
            pass
        time.sleep(0.05)
    raise RuntimeError(f"port {port} never became ready")


def send_stop(proc: subprocess.Popen) -> None:
    """The stop signal an orchestrator sends first: SIGTERM on POSIX, Ctrl+Break (uvicorn's SIGBREAK) on Windows."""
    if WINDOWS:
        proc.send_signal(signal.CTRL_BREAK_EVENT)
    else:
        proc.terminate()


def hard_kill(proc: subprocess.Popen) -> None:
    proc.kill()


def refused(port: int) -> bool:
    """True when nothing accepts connections on the port any more."""
    try:
        socket.create_connection(("127.0.0.1", port), timeout=5).close()
        return False
    except OSError:
        return True


def read_lines(path: str) -> list[tuple[float, str]]:
    rows = []
    if os.path.exists(path):
        for line in open(path, encoding="utf-8", errors="replace"):
            head, _, rest = line.rstrip("\n").partition(" ")
            try:
                rows.append((float(head), rest.strip()))
            except ValueError:
                continue  # a traceback line, not a timestamped one
    return rows


def timeline(proc: subprocess.Popen, t0: float) -> list[str]:
    """uvicorn's log and the app's own events merged, in seconds since t0 (the moment the signal was sent)."""
    rows = [(t, "uvicorn  " + text) for t, text in read_lines(proc.paths["log"])]
    rows += [(t, "app      " + text) for t, text in read_lines(proc.paths["events"])]
    return [f"{t - t0:+6.2f} s  {text}" for t, text in sorted(rows) if t >= t0 - 0.05]

1.4 Set up the environment

Put the three files in an empty folder and create a virtual environment. On Linux or macOS replace Scripts with bin.

python -m venv .venv
.venv/Scripts/python -m pip install fastapi uvicorn requests pytest
.venv/Scripts/python -c "import fastapi, uvicorn; print(fastapi.__version__, uvicorn.__version__)"

The last command should print the two versions below. If it does, you are ready.

0.162.5 0.54.0

Step 2: Stop the service while a request is in flight

The first experiment is the smallest one that teaches anything. Start the service, send a request that takes 3 seconds, and 0.7 seconds later send the stop signal. Then ask three questions: did the request finish, can a new connection still get in, and how long did the process take to exit? Then repeat the whole thing with a hard kill, which is what happens when something terminates the process without asking.

# step01_stop_signal.py
"""step01_stop_signal.py: stop a server while a request is in flight, once politely and once with a hard kill."""
import threading
import time

import requests

import labkit

PORT = 9101


def scenario(how: str) -> None:
    proc = labkit.start(PORT, "--access-log", name=f"step01-{how}")
    seen = {}

    def slow_request() -> None:
        began = time.time()
        try:
            reply = requests.get(f"http://127.0.0.1:{PORT}/work", params={"seconds": 3}, timeout=20)
            seen["slow"] = f"HTTP {reply.status_code} after {time.time() - began:.2f} s"
        except requests.RequestException as error:
            seen["slow"] = f"{type(error).__name__} after {time.time() - began:.2f} s"

    client = threading.Thread(target=slow_request)
    client.start()
    time.sleep(0.7)  # the request is now in flight
    t0 = time.time()
    threading.Thread(target=lambda: (proc.wait(), seen.update(exit_after=time.time() - t0))).start()
    labkit.send_stop(proc) if how == "signal" else labkit.hard_kill(proc)
    time.sleep(0.4)
    probe = time.time()
    seen["new"] = "refused" if labkit.refused(PORT) else "accepted"
    seen["probe_took"] = time.time() - probe
    code = proc.wait(timeout=30)
    client.join()
    time.sleep(0.2)
    exited = seen["exit_after"]
    print(f"=== {'stop signal' if how == 'signal' else 'hard kill'} at +0.00 s, request started 0.7 s earlier")
    print(f"in-flight 3 s request: {seen['slow']}")
    print(f"new connection 0.4 s after the signal: {seen['new']} (found out in {seen['probe_took']:.2f} s)")
    print(f"process exit code {code}, {exited:.2f} s after the signal")
    if how == "signal":
        print("--- timeline (seconds since the signal)")
        print("\n".join(labkit.timeline(proc, t0)))


scenario("signal")
scenario("kill")

Run it with python step01_stop_signal.py. The first half is the polite stop:

=== stop signal at +0.00 s, request started 0.7 s earlier
in-flight 3 s request: HTTP 200 after 3.02 s
new connection 0.4 s after the signal: refused (found out in 2.06 s)
process exit code 3, 2.43 s after the signal
--- timeline (seconds since the signal)
 +0.03 s  uvicorn  INFO:     Shutting down
 +0.15 s  uvicorn  INFO:     Waiting for connections to close. (CTRL+C to force quit)
 +2.32 s  uvicorn  INFO:     127.0.0.1:60134 "GET /work?seconds=3 HTTP/1.1" 200 OK
 +2.40 s  uvicorn  INFO:     Waiting for application shutdown.
 +2.40 s  app      lifespan shutdown begins: 0 tracked task(s) still running
 +2.41 s  app      lifespan shutdown done: database closed
 +2.41 s  uvicorn  INFO:     Application shutdown complete.
 +2.41 s  uvicorn  INFO:     Finished server process [8876]

Read the timeline from the top, because it is the whole story of a graceful shutdown. The signal arrives at +0.00. Within 0.03 seconds uvicorn logs “Shutting down,” and in that moment it closes its listening socket, which is why the probe 0.4 seconds later finds the port refused. Next uvicorn asks every open connection to finish, and it logs “Waiting for connections to close.” Our 3-second request keeps running and answers at +2.32 seconds with status 200, which the client sees as “HTTP 200 after 3.02 s.” Only after that does uvicorn tell the application to shut down, and only then does our lifespan teardown run and close the database. The order matters: connections drain first, lifespan teardown comes last.

The hard kill is the contrast:

=== hard kill at +0.00 s, request started 0.7 s earlier
in-flight 3 s request: ConnectionError after 0.71 s
new connection 0.4 s after the signal: refused (found out in 2.05 s)
process exit code 1, 0.01 s after the signal

The client gets a ConnectionError after 0.71 seconds, because the process disappeared in the middle of the request. Nothing was logged, no teardown ran, and the exit code is 1. This is the baseline we want to avoid.

Two small observations. First, “found out in 2.06 s” is a quirk of this Windows machine: a connection attempt to a closed local port takes about two seconds to fail, so keep that in mind when we meet it again in Step 5. Second, the polite stop ended with exit code 3 here. uvicorn re-raises the signal it caught once it has finished cleanly (see capture_signals in uvicorn/server.py), so the exit status still says the process ended through a signal; on Windows that shows up as 3. I did not test what Linux reports.

The uvicorn documentation describes the same sequence in its “Graceful Process Shutdown” section: “Close any connections that are not currently waiting on an HTTP response, and wait for any other connections to finalize their HTTP responses.” It also lists a step we will test in Step 4: “Wait for any background tasks to run to completion, such as occurs when the ASGI application has sent the HTTP response, but the asyncio task has not yet run to completion.”

Step 3: Put a ceiling on how long a stopping server waits

A graceful shutdown waits for in-flight requests. That is the point, and it is also a risk: a request that never finishes keeps the process alive forever. The orchestrator will eventually lose patience and kill the process the hard way, so your graceful shutdown ends up as a hard kill with extra delay. The fix is to set your own limit, shorter than the platform’s.

3.1 See the stuck request

step02_stuck_work.py runs seven cases in two groups: four about stuck work, which we read here, and three about work that continues after the response, which we read in Step 4. Each stuck-work case starts a fresh server, sends one long request, sends the stop signal 0.7 seconds later, and watches the process.

# step02_stuck_work.py
"""step02_stuck_work.py: what keeps a stopping server alive, what a timeout cuts, and which after-response work survives."""
import os
import subprocess
import threading
import time

import requests

import labkit


def case(title: str, port: int, server_args: list, method: str, path: str, watch: float, show_log: bool = False) -> None:
    proc = labkit.start(port, *server_args, name=f"step02-{port}")
    seen = {}

    def client() -> None:
        began = time.time()
        try:
            reply = requests.request(method, f"http://127.0.0.1:{port}{path}", timeout=60)
            seen["client"] = f"HTTP {reply.status_code} ({reply.reason}) after {time.time() - began:.2f} s"
        except requests.RequestException as error:
            seen["client"] = f"{type(error).__name__} after {time.time() - began:.2f} s"

    thread = threading.Thread(target=client)
    thread.start()
    time.sleep(0.7)
    t0 = time.time()
    labkit.send_stop(proc)
    try:
        code = proc.wait(timeout=watch)
        outcome = f"process exited {time.time() - t0:5.2f} s after the signal (code {code})"
    except subprocess.TimeoutExpired:
        outcome = f"process STILL RUNNING {watch:.0f} s after the signal, killed by hand"
        labkit.hard_kill(proc)
        proc.wait()
    thread.join()
    print(f"{title}\n    {outcome}\n    client: {seen['client']}")
    if show_log:
        for row in labkit.timeline(proc, t0):
            if "Cancel" in row or "Shutting" in row or "shutdown" in row:
                print("    " + row[:140])
        finished = [text for _, text in labkit.read_lines(proc.paths["events"]) if "sync-work" in text]
        print("    sync-work finished its work:", "yes" if finished else "never (the thread was abandoned)")


def email_mode(mode: str, port: int) -> None:
    proc = labkit.start(port, name=f"step02-email-{mode}")
    reply = requests.post(f"http://127.0.0.1:{port}/email/{mode}", params={"tag": mode, "delay": 2}, timeout=5)
    time.sleep(0.3)
    t0 = time.time()
    labkit.send_stop(proc)
    proc.wait(timeout=30)
    sent = os.path.exists(proc.paths["outbox"]) and "sent" in open(proc.paths["outbox"], encoding="utf-8").read()
    print(f"mode {mode:<10} queued with HTTP {reply.status_code}; process exited {time.time() - t0:.2f} s after the signal; "
          f"e-mail {'SENT' if sent else 'LOST'}")
    if mode in ("task", "tracked"):
        for row in labkit.timeline(proc, t0):
            if "lifespan" in row or "Waiting" in row:
                print("    " + row[:140])


case("async request that needs 6 s, no timeout", 9111, [], "GET", "/work?seconds=6", watch=30)
case("async request that needs 60 s, no timeout", 9112, [], "GET", "/work?seconds=60", watch=5)
case("same 60 s request, --timeout-graceful-shutdown 2", 9113, ["--timeout-graceful-shutdown", 2], "GET", "/work?seconds=60",
     watch=30, show_log=True)
case("plain def route that sleeps 8 s in a thread, --timeout-graceful-shutdown 2", 9114,
     ["--timeout-graceful-shutdown", 2], "GET", "/sync-work?seconds=8", watch=40, show_log=True)
for number, mode in enumerate(("background", "task", "tracked")):
    email_mode(mode, 9150 + number)

Run python step02_stuck_work.py. The first part of the output is about stuck work:

async request that needs 6 s, no timeout
    process exited  5.38 s after the signal (code 3)
    client: HTTP 200 (OK) after 6.01 s
async request that needs 60 s, no timeout
    process STILL RUNNING 5 s after the signal, killed by hand
    client: ConnectionError after 5.73 s
same 60 s request, --timeout-graceful-shutdown 2
    process exited  2.23 s after the signal (code 3)
    client: HTTP 500 (Internal Server Error) after 2.92 s
     +0.06 s  uvicorn  INFO:     Shutting down
     +2.18 s  uvicorn  ERROR:    Cancel 1 running task(s), timeout graceful shutdown exceeded
     +2.18 s  uvicorn  INFO:     Waiting for application shutdown.
     +2.21 s  app      lifespan shutdown begins: 0 tracked task(s) still running
     +2.21 s  app      lifespan shutdown done: database closed
     +2.22 s  uvicorn  INFO:     Application shutdown complete.
    sync-work finished its work: never (the thread was abandoned)
plain def route that sleeps 8 s in a thread, --timeout-graceful-shutdown 2
    process exited  2.21 s after the signal (code 3)
    client: HTTP 500 (Internal Server Error) after 2.90 s
     +0.06 s  uvicorn  INFO:     Shutting down
     +2.19 s  uvicorn  ERROR:    Cancel 1 running task(s), timeout graceful shutdown exceeded
     +2.19 s  uvicorn  INFO:     Waiting for application shutdown.
     +2.20 s  app      lifespan shutdown begins: 0 tracked task(s) still running
     +2.20 s  app      lifespan shutdown done: database closed
     +2.20 s  uvicorn  INFO:     Application shutdown complete.
    sync-work finished its work: never (the thread was abandoned)

The first case is the healthy one: a 6-second request is allowed to finish, the client gets its 200, and the process exits 5.38 seconds after the signal, which is the time the request had left. The second case is the problem. With a 60-second request and no limit, the process is still running 5 seconds after the signal and would have kept running for about 54 seconds more. The reason is in uvicorn/server.py: the shutdown code waits for connections and tasks inside asyncio.wait_for(..., timeout=self.config.timeout_graceful_shutdown), and that setting defaults to None, which means no limit.

The third case sets --timeout-graceful-shutdown 2. The log line you want to remember is Cancel 1 running task(s), timeout graceful shutdown exceeded. The process exits 2.23 seconds after the signal, and the client receives HTTP 500 after 2.92 seconds. The uvicorn settings page describes the option as “Maximum number of seconds to wait for graceful shutdown. After this timeout, the server will start terminating requests.” Notice what “terminating” means for the client: an error response, not a polite retry hint. A timeout is a last resort for stuck work. It is not a way to hand requests over politely, which is what Step 5 is for.

3.2 Plain def routes run in threads

The fourth case uses /sync-work, a plain def route that sleeps 8 seconds. FastAPI runs such routes in a worker thread so they do not block the event loop. With the same 2-second limit the process again exits 2.21 seconds after the signal and the client again gets HTTP 500, but look at the last line: the function never finished its work. Python cannot interrupt a thread that is in the middle of time.sleep, so the shutdown abandons the thread and the process ends around it. For a sleep that is harmless. For a function that is halfway through writing a file or calling a payment provider, it means the work stops at an arbitrary line. Keep blocking routes short, make their side effects safe to repeat, or move long jobs to a queue (see the SQLite job queue tutorial linked at the end).

3.3 Choose the number

The limit has to be shorter than the platform’s deadline. The Kubernetes documentation says: “The default terminationGracePeriodSeconds setting is 30 seconds.” The same page continues: “Once the grace period has expired, the KILL signal is sent to any remaining processes.” Pick a graceful timeout comfortably below the grace period, and remember that the drain delay from Step 5 comes out of the same budget.

Step 4: Do not lose work that happens after the response

Many services answer the client first and finish the real work afterwards: send an e-mail, update a cache, write an audit record. There are three common ways to do that in FastAPI, and they behave differently at shutdown. /email/background uses FastAPI’s BackgroundTasks. /email/task uses a bare asyncio.create_task. /email/tracked also uses create_task but registers the task with our Lifecycle object, and the lifespan teardown waits for everything registered there. Each case queues one e-mail that needs 2 seconds, sends the stop signal 0.3 seconds later, and checks whether the e-mail was written to the outbox file after the process exited.

These cases are the second half of the same output from step02_stuck_work.py:

mode background queued with HTTP 202; process exited 1.80 s after the signal; e-mail SENT
mode task       queued with HTTP 202; process exited 0.13 s after the signal; e-mail LOST
     +0.11 s  app      lifespan shutdown begins: 0 tracked task(s) still running
     +0.11 s  uvicorn  INFO:     Waiting for application shutdown.
     +0.12 s  app      lifespan shutdown done: database closed
mode tracked    queued with HTTP 202; process exited 1.73 s after the signal; e-mail SENT
     +0.11 s  uvicorn  INFO:     Waiting for application shutdown.
     +0.11 s  app      lifespan shutdown begins: 1 tracked task(s) still running
     +1.71 s  app      lifespan shutdown: waited for tracked tasks, 0 abandoned
     +1.71 s  app      lifespan shutdown done: database closed

The BackgroundTasks e-mail was sent, and the process stayed alive for 1.80 seconds to do it. That is uvicorn’s doing: a background task runs inside the request’s own task, and uvicorn waits for its request tasks (they are the “background tasks” in its “Waiting for background tasks to complete” log line). The tracked e-mail was sent too. Its timeline shows the mechanism: lifespan teardown began at +0.11 seconds with “1 tracked task(s) still running,” waited, and finished at +1.71 seconds. The bare create_task e-mail was lost. The process exited 0.13 seconds after the signal because nothing was keeping it alive: uvicorn does not know about tasks you create yourself, and once the server returns, nothing keeps the event loop running long enough for such a task to finish.

The fix is the few lines of Lifecycle.spawn in app.py: keep a reference to every task you start, and wait for the set in your lifespan teardown. The wait is bounded by LAB_TRACKED_WAIT (5 seconds here), so a task that never ends cannot block the exit, and the number it abandons is logged. The FastAPI documentation explains why the teardown is the right place: the code after yield “will be executed once, after having handled possibly many requests.” By the time it runs, uvicorn has already stopped accepting connections and waited for the requests it knows about.

One limit to remember: the tracked wait only helps if your timeouts leave room for it. Your graceful timeout, the tracked wait, and the platform’s grace period all draw on the same clock.

Step 5: Survive a rolling restart behind a load balancer

So far one process stopped politely and nobody noticed. Real deployments run several copies behind a router and replace them one at a time, so the question becomes: while one copy is stopping, does any request still reach it?

5.1 Why the polite stop is still not enough

The Kubernetes documentation describes two things that start together when a pod is deleted. The kubelet has the container runtime “send a TERM signal to process 1 inside each container,” and, in the documentation’s words, “At the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice objects.” Those are two independent announcements travelling by different routes. Your process hears the first one almost immediately. The router hears the second one after some delay. In between, the router can still send requests to a process that has already closed its listening socket, and you saw in Step 2 how uvicorn behaves in that moment: it refuses new connections at once.

I do not have a Kubernetes cluster here, so the lab copies the shape of that race instead of the platform. A tiny “control plane” in the script removes the stopping backend from nginx’s upstream list one second after the stop signal is delivered, and the new copy of the backend is added once its /ready answers. The one second is my assumption, written as PROPAGATION in the script. Your platform’s number is whatever it is; the way to find it is to run a steady request loop against the service while you restart one instance, exactly as this step does.

5.2 The router and the traffic generator

router.py starts nginx with an upstream list that the script can change at run time, like a platform would. Two settings need explaining. proxy_next_upstream decides whether nginx tries the next server when one fails; the Router can run with nginx’s default (error timeout) or with off, which passes the error to the client, as a load balancer without retries would. And max_fails=0 turns off nginx’s own memory of failed servers. The nginx documentation says about max_fails: “By default, the number of unsuccessful attempts is set to 1. The zero value disables the accounting of attempts.” Without that setting nginx would stop sending traffic to a failed server for a while after the first error, which would hide the very gap we want to measure.

# router.py
"""router.py: nginx in front of the backends, with the add and remove a platform's control plane would do."""
import collections
import io
import os
import re
import subprocess
import threading
import time
import zipfile

import requests

import labkit

NGINX_VERSION = "1.30.6"
FRONT = 8280

CONF = """worker_processes 1;
error_log logs/error.log notice;
pid logs/nginx.pid;
events {{ worker_connections 1024; }}
http {{
    access_log off;
    include upstream.conf;
    server {{
        listen 127.0.0.1:{front};
        location / {{
            proxy_pass http://app;
            proxy_next_upstream {retry};
            proxy_connect_timeout 5s;
            proxy_read_timeout 20s;
        }}
    }}
}}
"""


def nginx_exe() -> str:
    dist = os.path.join(labkit.HERE, "nginx_dist")
    exe = os.path.join(dist, f"nginx-{NGINX_VERSION}", "nginx.exe")
    if not os.path.exists(exe):
        data = requests.get(f"https://nginx.org/download/nginx-{NGINX_VERSION}.zip", timeout=120).content
        zipfile.ZipFile(io.BytesIO(data)).extractall(dist)
    return exe


class Router:
    """retry=True keeps nginx's default (try the next server after an error); retry=False passes the error to the client."""

    def __init__(self, ports: list, retry: bool) -> None:
        self.exe = nginx_exe()
        self.prefix = os.path.join(labkit.RUN, "nginx")
        for folder in ("conf", "logs", "temp"):
            os.makedirs(os.path.join(self.prefix, folder), exist_ok=True)
        self.error_log = os.path.join(self.prefix, "logs", "error.log")
        if os.path.exists(self.error_log):
            os.remove(self.error_log)
        self.conf = os.path.join(self.prefix, "conf", "nginx.conf")
        with open(self.conf, "w", newline="\n") as handle:
            handle.write(CONF.format(front=FRONT, retry="error timeout" if retry else "off"))
        self.lock = threading.Lock()
        self.ports = list(ports)
        self.write_upstream()
        self.process = subprocess.Popen([self.exe, "-p", self.prefix, "-c", self.conf],
                                        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
        time.sleep(0.8)

    def write_upstream(self) -> None:
        # max_fails=0 turns off nginx's own "that server failed, skip it for a while" memory, so the router behaves like
        # a platform load balancer that keeps sending traffic until its control plane says otherwise.
        lines = ["upstream app {"] + [f"    server 127.0.0.1:{port} max_fails=0;" for port in self.ports] + ["}"]
        with open(os.path.join(self.prefix, "conf", "upstream.conf"), "w", newline="\n") as handle:
            handle.write("\n".join(lines) + "\n")

    def _reload(self) -> None:
        self.write_upstream()
        subprocess.run([self.exe, "-p", self.prefix, "-c", self.conf, "-s", "reload"], capture_output=True)

    def remove(self, port: int) -> None:
        with self.lock:
            if port in self.ports:
                self.ports.remove(port)
                self._reload()

    def add(self, port: int) -> None:
        with self.lock:
            if port not in self.ports:
                self.ports.append(port)
                self._reload()

    def stop(self) -> None:
        subprocess.run([self.exe, "-p", self.prefix, "-c", self.conf, "-s", "stop"], capture_output=True)
        try:
            self.process.wait(timeout=10)
        except subprocess.TimeoutExpired:
            subprocess.run(["taskkill", "/IM", "nginx.exe", "/F", "/T"], capture_output=True)

    def error_kinds(self) -> collections.Counter:
        """What nginx logged for the failed attempts, with the timestamp, ids and addresses stripped off."""
        kinds = collections.Counter()
        if os.path.exists(self.error_log):
            for line in open(self.error_log, encoding="utf-8", errors="replace"):
                match = re.search(r"\[error\] \d+#\d+: \*\d+ (.*?)(?:, client:|$)", line)
                if match:
                    kinds[re.sub(r"\((\d+): [^)]*\)", r"(\1)", match.group(1))[:150]] += 1
        return kinds

loadgen.py sends 20 requests per second: about half GET /work and half POST /orders, each needing 0.3 seconds of server time, so on average several requests are in flight at any moment. The traffic is open loop: a new request starts every 50 milliseconds whether or not the earlier ones have finished, so a slow failure cannot slow down the arrival of the next request. Every request uses a new connection (Connection: close), so no idle connection can be caught by an nginx reload and blamed on the service. The mix of GET and POST is seeded, so every scenario sees the same requests.

# loadgen.py
"""loadgen.py: open-loop traffic through the router, so a slow failure cannot slow the arrival of the next request."""
import collections
import random
import threading
import time

import requests

from router import FRONT

RATE = 20  # requests per second
CLOSE = {"Connection": "close"}  # a new connection every time, so no idle keep-alive connection can be caught by a reload
OK = ("200", "201")


class Load:
    def __init__(self, tag: str) -> None:
        self.tag, self.rows, self.workers = tag, [], []
        self.halt = threading.Event()
        self.driver = threading.Thread(target=self.drive)

    def one(self, number: int) -> None:
        kind = "POST" if random.Random(number).random() < 0.5 else "GET"  # seeded, so every scenario sees the same mix
        began = time.time()
        try:
            if kind == "GET":
                reply = requests.get(f"http://127.0.0.1:{FRONT}/work", params={"seconds": 0.3}, headers=CLOSE, timeout=30)
            else:
                reply = requests.post(f"http://127.0.0.1:{FRONT}/orders", json={"id": f"{self.tag}-{number}", "work": 0.3},
                                      headers=CLOSE, timeout=30)
            status = str(reply.status_code)
        except requests.RequestException as error:
            status = type(error).__name__
        self.rows.append((kind, status, time.time() - began))

    def drive(self) -> None:
        begun, number = time.time(), 0
        while not self.halt.is_set():
            worker = threading.Thread(target=self.one, args=(number,))
            worker.start()
            self.workers.append(worker)
            number += 1
            time.sleep(max(0.0, begun + number / RATE - time.time()))

    def start(self) -> None:
        self.driver.start()

    def finish(self) -> None:
        self.halt.set()
        self.driver.join()
        for worker in self.workers:
            worker.join()

    def summary(self) -> dict:
        gets = [row for row in self.rows if row[0] == "GET"]
        posts = [row for row in self.rows if row[0] == "POST"]
        return {
            "sent": len(self.rows),
            "get_failed": sum(row[1] not in OK for row in gets),
            "post_failed": sum(row[1] not in OK for row in posts),
            "slow": sum(row[2] > 1.5 for row in self.rows),
            "slowest": max(row[2] for row in self.rows),
            "created": sum(row[1] == "201" for row in posts),
            "statuses": collections.Counter(row[1] for row in self.rows if row[1] not in OK),
        }

5.3 The rolling restart

step03_rolling_restart.py starts two backends, A and B, puts nginx in front, and starts the traffic. Then it restarts A, waits 3.5 seconds, and restarts B. Every server in the scenarios runs with a 10-second graceful timeout, so nothing can hang the experiment. For each restart three things happen at once, as in the Kubernetes description: a timer is started that removes the old backend from the router after PROPAGATION seconds, the stop signal (or a hard kill) is delivered immediately, and when the process has exited a new copy starts on a new port, like a new pod gets a new address, and is added to the router once ready. At the end the script counts how many requests failed, how many rows the two databases hold, and what nginx logged.

Six scenarios are defined. They vary three things: how the old copy is stopped (hard kill or stop signal), whether nginx retries, and how long the server drains before closing its door (--drain-delay, built in Step 5.5).

# step03_rolling_restart.py
"""step03_rolling_restart.py: restart two backends one after the other behind nginx while traffic keeps flowing."""
import os
import sqlite3
import sys
import threading
import time

import labkit
from loadgen import Load
from router import Router

PORTS = {"A": 9201, "B": 9202}
PROPAGATION = 1.0  # seconds the platform needs before it stops sending traffic to a backend that is shutting down


def run(tag: str, how: str, drain: float, retry: bool) -> tuple[dict, list]:
    flags = ["--timeout-graceful-shutdown", 10] + (["--drain-delay", drain] if drain else [])
    for key in PORTS:  # a fresh ledger for every scenario
        path = os.path.join(labkit.RUN, f"{tag}-{key}.db")
        if os.path.exists(path):
            os.remove(path)
    current = dict(PORTS)  # backend name -> port; a restarted backend gets a NEW port, like a new pod gets a new address
    procs = {key: labkit.start(port, *flags, name=f"{tag}-{key}1", db=f"{tag}-{key}.db") for key, port in current.items()}
    router = Router(list(current.values()), retry)
    load = Load(tag)
    load.start()
    begun = time.time()
    marks = []
    time.sleep(1.5)
    for key in PORTS:
        old = current[key]
        marks.append((time.time() - begun, f"stop {key}"))
        # The platform takes the stopping backend out of rotation, but only after a lag...
        threading.Timer(PROPAGATION, lambda port=old, name=key: (router.remove(port), marks.append((time.time() - begun, f"{name} out of rotation")))).start()
        (labkit.hard_kill if how == "kill" else labkit.send_stop)(procs[key])  # ...while the stop is delivered at once
        procs[key].wait(timeout=60)
        current[key] = old + 10
        procs[key] = labkit.start(current[key], *flags, name=f"{tag}-{key}2", db=f"{tag}-{key}.db")
        router.add(current[key])                                               # in rotation once /ready answers
        marks.append((time.time() - begun, f"new {key} in rotation"))
        time.sleep(3.5)  # let requests that were waiting on the dead backend finish before the next backend is touched
    load.finish()
    result = load.summary()
    result["nginx"] = router.error_kinds()
    router.stop()
    for proc in procs.values():
        labkit.hard_kill(proc)
        proc.wait()
    result["rows"] = 0
    for key in PORTS:
        with sqlite3.connect(os.path.join(labkit.RUN, f"{tag}-{key}.db")) as db:
            result["rows"] += db.execute("select count(*) from orders where id like ?", (f"{tag}-%",)).fetchone()[0]
    return result, sorted(marks)


SCENARIOS = [
    ("hard kill, nginx retries (its default)", "kill", 0, True),
    ("hard kill, no retries", "kill", 0, False),
    ("stop signal, nginx retries (its default)", "signal", 0, True),
    ("stop signal, no retries", "signal", 0, False),
    ("stop signal + 0.5 s drain, no retries", "signal", 0.5, False),
    ("stop signal + 2.5 s drain, no retries", "signal", 2.5, False),
]

if __name__ == "__main__":
    chosen = [int(arg) for arg in sys.argv[1:]] or list(range(len(SCENARIOS)))
    results = []
    for index in chosen:
        title, how, drain, retry = SCENARIOS[index]
        result, marks = run(f"r{index}", how, drain, retry)
        results.append((title, result, marks))
        print(f"{title:<42} sent {result['sent']:>3}  GET failed {result['get_failed']:>2}  POST failed {result['post_failed']:>2}  "
              f"slow(>1.5 s) {result['slow']:>2}  slowest {result['slowest']:4.2f} s  201s {result['created']:>3}  "
              f"rows {result['rows']:>3}", flush=True)
    print("--- what the failures looked like (client side, then nginx error log)")
    for title, result, _ in results:
        if result["statuses"] or result["nginx"]:
            print(f"{title}: clients {dict(result['statuses'])}")
            for kind, count in result["nginx"].most_common(3):
                print(f"    nginx x{count}: {kind}")
    print("--- when things happened (seconds since the traffic started)")
    for title, _, marks in results:
        print(f"{title}:")
        print("    " + " | ".join(f"{label} {t:.2f}" for t, label in marks))

5.4 Read the numbers

Run python step03_rolling_restart.py. It takes about two minutes. The first block is the table:

hard kill, nginx retries (its default)     sent 197  GET failed  0  POST failed  3  slow(>1.5 s) 21  slowest 2.45 s  201s 101  rows 101
hard kill, no retries                      sent 196  GET failed 12  POST failed 16  slow(>1.5 s) 21  slowest 2.08 s  201s  88  rows  88
stop signal, nginx retries (its default)   sent 213  GET failed  0  POST failed  0  slow(>1.5 s) 21  slowest 2.41 s  201s 115  rows 115
stop signal, no retries                    sent 211  GET failed 14  POST failed  8  slow(>1.5 s) 22  slowest 2.10 s  201s 106  rows 106
stop signal + 0.5 s drain, no retries      sent 230  GET failed  7  POST failed  6  slow(>1.5 s) 12  slowest 2.08 s  201s 117  rows 117
stop signal + 2.5 s drain, no retries      sent 303  GET failed  0  POST failed  0  slow(>1.5 s)  0  slowest 0.43 s  201s 158  rows 158

Each row sent between 196 and 303 requests (the drain scenario ran longer, so it sent more). “GET failed” and “POST failed” count requests that did not get a 200 or 201. “slow” counts requests that took more than 1.5 seconds. “201s” is how many POSTs the client saw succeed, and “rows” is how many rows the two SQLite files hold afterwards; they match in every row, so no write was committed without being acknowledged. (That match is partly luck of this lab: the insert and the response are microseconds apart. A real handler that commits and then talks to another service can leave a committed write that the client never heard about, which is what idempotency keys are for.)

Read the rows from the top:

• Hard kill, nginx retries. No GET failed, because nginx retried them on the other backend. Three POSTs failed. The nginx documentation explains why: unless the non_idempotent parameter is added to proxy_next_upstream, “requests with a non-idempotent method (POST, LOCK, PATCH) are not passed to the next server if a request has been sent to an upstream server.” The POSTs that were running inside the killed process had been sent, so nginx could not know whether they had taken effect, and it returned an error. The 21 slow requests are the ones nginx first sent to the dead backend; on this Windows host a refused connection takes about 2 seconds to be reported.
• Hard kill, no retries. 28 failures (12 GET and 16 POST). Every request that reached a dead backend failed, whether it was in flight or new.
• Stop signal, nginx retries. Zero failures in this run, but 21 slow requests. (In a second full run of the script, the one I used to check the article from a clean folder, this row had one failed request instead of none, so read zero as almost none.) The retry hid the error and kept the cost: each of those clients waited about 2 seconds for nothing. On Linux a refused connection is reported at once, so I would expect the delay to be much smaller, but I did not measure it.
• Stop signal, no retries. 22 failures, and the failure kinds below show what kind: 22 refused connections and nothing else. The polite stop did its job, because no in-flight request was lost. The failures are requests that arrived at a closed door during the one second before the router noticed.
• Stop signal with a 0.5 second drain. 13 failures. Better, because the server kept answering for half of that second, but half of the gap is still open.
• Stop signal with a 2.5 second drain. Zero failures, none slow, and the slowest request took 0.43 seconds.

Here is what the failures looked like from both sides:

--- what the failures looked like (client side, then nginx error log)
hard kill, nginx retries (its default): clients {'502': 3}
    nginx x21: connect() failed (10061) while connecting to upstream
    nginx x7: WSARecv() failed (10054) while reading response header from upstream
hard kill, no retries: clients {'502': 28}
    nginx x21: connect() failed (10061) while connecting to upstream
    nginx x7: WSARecv() failed (10054) while reading response header from upstream
stop signal, nginx retries (its default): clients {}
    nginx x21: connect() failed (10061) while connecting to upstream
stop signal, no retries: clients {'502': 22}
    nginx x22: connect() failed (10061) while connecting to upstream
stop signal + 0.5 s drain, no retries: clients {'502': 13}
    nginx x12: connect() failed (10061) while connecting to upstream
    nginx x1: upstream prematurely closed connection while reading response header from upstream

The timeline shows the race directly. In the stop-signal row without a drain, the old A is stopped at 1.50 seconds, and the router only removes it at 2.56, so for about a second any request that nginx hands to A meets a closed port. With the 2.5 second drain, A keeps serving until after it has been removed from the router, so the door is never closed on a request.

--- when things happened (seconds since the traffic started)
hard kill, nginx retries (its default):
    stop A 1.50 | new A in rotation 2.14 | A out of rotation 2.55 | stop B 5.64 | new B in rotation 6.31 | B out of rotation 6.69
hard kill, no retries:
    stop A 1.50 | new A in rotation 2.15 | A out of rotation 2.55 | stop B 5.65 | new B in rotation 6.27 | B out of rotation 6.71
stop signal, nginx retries (its default):
    stop A 1.50 | new A in rotation 2.53 | A out of rotation 2.57 | stop B 6.03 | B out of rotation 7.09 | new B in rotation 7.13
stop signal, no retries:
    stop A 1.50 | new A in rotation 2.53 | A out of rotation 2.56 | stop B 6.03 | new B in rotation 7.03 | B out of rotation 7.07
stop signal + 0.5 s drain, no retries:
    stop A 1.50 | A out of rotation 2.55 | new A in rotation 2.91 | stop B 6.41 | B out of rotation 7.46 | new B in rotation 7.97
stop signal + 2.5 s drain, no retries:
    stop A 1.50 | A out of rotation 2.56 | new A in rotation 4.81 | stop B 8.31 | B out of rotation 9.37 | new B in rotation 11.62

I ran each row once for the table above. To see how much the counts move, I ran the two rows that matter most twice more on the same machine. The stop-signal row without a drain failed 22, 20 and 21 requests (the table run, then the two repeats), and the 2.5 second drain failed none in all three. Treat the exact counts as approximate; the pattern is what the experiment shows.

5.5 Add the drain delay to your own server

The drain delay lives in DrainingServer inside serve.py, which you already have. It overrides uvicorn’s handle_exit, the function that runs when a stop signal arrives. If a drain is configured and this is the first signal, it sets life.draining to true, which makes /ready answer 503, writes a note, and starts a timer. The server keeps serving normally while the timer runs. When the timer fires, it calls the original handle_exit, and from there everything in Steps 2 to 4 happens as before. A second stop signal during the drain skips the rest of the wait, because life.draining is already set and the code falls through to the normal path. That is useful when you are at a terminal and lose patience.

To use it, pass the delay when you start the server, for example python serve.py --port 9001 --drain-delay 5 --timeout-graceful-shutdown 10. Choose the numbers with three rules. The drain delay must be longer than your platform’s lag, which you measure as in Step 5.1. The graceful timeout must cover the longest request you are willing to finish. And the sum must fit inside the platform’s deadline: drain delay plus graceful timeout plus the longest teardown (here LAB_TRACKED_WAIT, 5 seconds) must be less than terminationGracePeriodSeconds, which the Kubernetes documentation says defaults to 30 seconds. With 5, 10 and 5 the total is 20 seconds, which fits.

The delay does not have to live in the application. The Kubernetes documentation also describes a preStop hook: “If one of the Pod’s containers has defined a preStop hook and the terminationGracePeriodSeconds in the Pod spec is not set to 0, the kubelet runs that hook inside of the container.” A hook that sleeps for a few seconds creates the same pause before the TERM signal. Doing it in the application has two advantages: it works outside Kubernetes, and the readiness flip is visible to a probe, so you can see in the logs that the drain happened.

5.6 What this lab does not show

• It is nginx, not Kubernetes. The race has the same shape, but the lag in your cluster is its own number.
• Connection refused takes about 2 seconds on this Windows host. On other systems the failing requests fail faster, and the “slow” column would be near zero.
• nginx opens a new connection to the backend for every request here. If you enable keep-alive connections to the upstream, an idle pooled connection can be closed by the server at the moment nginx reuses it. I did not test that.
• I did not test Gunicorn, multiple uvicorn workers, or the restart style the uvicorn documentation describes, in which new server processes listen on the existing socket.
• Each row was run once (and the two key rows three times), so small differences between rows mean little.

Step 6: Test the shutdown contract

Shutdown behaviour is easy to break and hard to notice: nothing fails until the next deploy. So write it down as tests. test_shutdown.py starts real server processes, sends real stop signals, and checks seven promises. An in-flight request finishes and then the listener is gone. While draining, /ready says 503 while /health and ordinary routes still answer 200. A second signal skips the rest of the drain. A graceful timeout cuts a stuck request and the process still exits. A tracked task and a background task are finished before exit, and a bare create_task is lost, which the last test records on purpose so that nobody “fixes” the demonstration by accident.

# test_shutdown.py
"""test_shutdown.py: the shutdown contract of the service, checked against real server processes."""
import os
import subprocess
import threading
import time

import pytest
import requests

import labkit


@pytest.fixture
def server():
    """server(port, *flags) starts serve.py and returns the process; whatever is still running afterwards is killed."""
    started = []

    def make(port: int, *flags) -> subprocess.Popen:
        proc = labkit.start(port, *flags, name=f"test-{port}")
        started.append(proc)
        return proc

    yield make
    for proc in started:
        if proc.poll() is None:
            labkit.hard_kill(proc)
            proc.wait()


def in_background(method: str, url: str, box: dict, **kwargs) -> threading.Thread:
    def call() -> None:
        try:
            box["status"] = requests.request(method, url, timeout=30, **kwargs).status_code
        except requests.RequestException as error:
            box["status"] = type(error).__name__
    thread = threading.Thread(target=call)
    thread.start()
    return thread


def test_in_flight_request_finishes_and_then_the_listener_is_gone(server):
    proc = server(9301)
    box = {}
    thread = in_background("GET", "http://127.0.0.1:9301/work?seconds=2", box)
    time.sleep(0.5)
    labkit.send_stop(proc)
    proc.wait(timeout=8)
    thread.join()
    assert box["status"] == 200
    assert labkit.refused(9301)


def test_ready_says_no_while_the_service_still_works(server):
    proc = server(9302, "--drain-delay", 2)
    labkit.send_stop(proc)
    time.sleep(0.6)
    assert requests.get("http://127.0.0.1:9302/ready", timeout=3).status_code == 503
    assert requests.get("http://127.0.0.1:9302/health", timeout=3).status_code == 200
    assert requests.get("http://127.0.0.1:9302/work?seconds=0", timeout=3).status_code == 200
    proc.wait(timeout=10)
    assert labkit.refused(9302)


def test_a_second_signal_skips_the_rest_of_the_drain(server):
    proc = server(9303, "--drain-delay", 30)
    labkit.send_stop(proc)
    time.sleep(0.5)
    began = time.time()
    labkit.send_stop(proc)
    proc.wait(timeout=8)
    assert time.time() - began < 6


def test_a_timeout_cuts_a_stuck_request_and_the_process_still_exits(server):
    proc = server(9304, "--timeout-graceful-shutdown", 1)
    box = {}
    thread = in_background("GET", "http://127.0.0.1:9304/work?seconds=30", box)
    time.sleep(0.5)
    labkit.send_stop(proc)
    proc.wait(timeout=8)
    thread.join()
    assert box["status"] != 200


def read_outbox(proc: subprocess.Popen) -> str:
    path = proc.paths["outbox"]
    return open(path, encoding="utf-8").read() if os.path.exists(path) else ""


def queue_email_then_stop(server, port: int, mode: str) -> str:
    proc = server(port)
    assert requests.post(f"http://127.0.0.1:{port}/email/{mode}", params={"tag": mode, "delay": 1.5}, timeout=5).status_code == 202
    time.sleep(0.3)
    labkit.send_stop(proc)
    proc.wait(timeout=10)
    return read_outbox(proc)


def test_a_tracked_task_is_finished_before_exit(server):
    assert "sent tracked" in queue_email_then_stop(server, 9305, "tracked")


def test_a_background_task_is_finished_before_exit(server):
    assert "sent background" in queue_email_then_stop(server, 9306, "background")


def test_a_bare_create_task_is_lost(server):
    """Documents the pitfall: nothing keeps the loop alive for it, so the e-mail never goes out."""
    assert "sent task" not in queue_email_then_stop(server, 9307, "task")

Run python -m pytest -q test_shutdown.py:

....... [100%]
7 passed in 18.59s

A passing suite proves little if it cannot fail. mutate.py copies the service seven times, plants one bug in each copy (the draining flag never set, the delay never used, /ready ignoring the flag, spawn() forgetting the task, teardown not waiting, the graceful timeout dropped, a second signal that restarts the drain instead of skipping it) and runs the suite on each copy. A bug counts as caught when the suite fails.

# mutate.py
"""mutate.py: plant one bug at a time in a copy of the service and check that the suite notices."""
import os
import re
import shutil
import subprocess
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
FILES = ["app.py", "serve.py", "labkit.py", "test_shutdown.py"]

MUTANTS = [
    ("draining flag never set", "serve.py", "            service.life.draining = True\n", "            pass\n"),
    ("drain delay never used", "serve.py", "if self.drain_delay > 0 and not service.life.draining:", "if False:"),
    ("/ready ignores draining", "app.py", "    if life.draining:\n        response.status_code = 503", "    if False:\n        response.status_code = 503"),
    ("spawn() forgets the task", "app.py", "        self.tasks.add(task)  ", "        pass  "),
    ("teardown does not wait", "app.py", "    if life.tasks:\n        _, abandoned", "    if False:\n        _, abandoned"),
    ("graceful timeout dropped", "serve.py", "timeout_graceful_shutdown=args.timeout_graceful_shutdown)", "timeout_graceful_shutdown=None)"),
    ("second signal restarts the drain", "serve.py", "if self.drain_delay > 0 and not service.life.draining:", "if self.drain_delay > 0:"),
]


def run_suite(folder: str) -> tuple[int, str]:
    env = {**os.environ, "COLUMNS": "170", "PYTHONDONTWRITEBYTECODE": "1"}
    done = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", "test_shutdown.py"],
                          cwd=folder, capture_output=True, text=True, env=env)
    failed = [re.search(r"::(\w+)", line).group(1) for line in done.stdout.splitlines() if line.startswith("FAILED")]
    summary = [line for line in done.stdout.splitlines() if re.match(r"\d+ (failed|passed|error)", line)]
    lines = [f"first failure: {failed[0]}"] if failed else []
    lines += summary[-1:]
    return done.returncode, " | ".join(lines)


if __name__ == "__main__":
    caught = 0
    for number, (name, filename, old, new) in enumerate(MUTANTS, 1):
        folder = os.path.join(HERE, "run", f"mutant{number}")
        shutil.rmtree(folder, ignore_errors=True)
        os.makedirs(folder)
        for file in FILES:
            shutil.copy(os.path.join(HERE, file), folder)
        text = open(os.path.join(folder, filename), encoding="utf-8").read()
        assert text.count(old) == 1, f"{name}: pattern found {text.count(old)} times"
        open(os.path.join(folder, filename), "w", encoding="utf-8", newline="\n").write(text.replace(old, new))
        code, summary = run_suite(folder)
        caught += code != 0
        print(f"mutant {number} {name:<34} {'CAUGHT' if code else 'SURVIVED':<9} {summary}", flush=True)
    print(f"{caught} of {len(MUTANTS)} planted bugs caught")

Run python mutate.py. It takes about two minutes:

mutant 1 draining flag never set            CAUGHT    first failure: test_ready_says_no_while_the_service_still_works | 1 failed, 1 passed in 6.13s
mutant 2 drain delay never used             CAUGHT    first failure: test_ready_says_no_while_the_service_still_works | 1 failed, 1 passed in 8.28s
mutant 3 /ready ignores draining            CAUGHT    first failure: test_ready_says_no_while_the_service_still_works | 1 failed, 1 passed in 6.13s
mutant 4 spawn() forgets the task           CAUGHT    first failure: test_a_tracked_task_is_finished_before_exit | 1 failed, 4 passed in 14.47s
mutant 5 teardown does not wait             CAUGHT    first failure: test_a_tracked_task_is_finished_before_exit | 1 failed, 4 passed in 14.37s
mutant 6 graceful timeout dropped           CAUGHT    first failure: test_a_timeout_cuts_a_stuck_request_and_the_process_still_exits | 1 failed, 3 passed in 20.11s
mutant 7 second signal restarts the drain   CAUGHT    first failure: test_a_second_signal_skips_the_rest_of_the_drain | 1 failed, 2 passed in 18.97s
7 of 7 planted bugs caught

Check that everything works end to end

You have the whole picture when you can reproduce five results with your own copy of the files. Step 2 should end with the in-flight request answered and the port refused, and the timeline should show lifespan teardown after the connection drained. Step 3 should show a 60-second request outliving the signal without a limit and being cut with HTTP 500 when you set --timeout-graceful-shutdown 2. Step 4 should lose exactly one of the three e-mails, the bare create_task. Step 5 should show failed requests in every row except two, where the count should be zero or very close to it: the stop signal with nginx retrying, which hides the failures as slow requests, and the stop signal with the 2.5 second drain. Step 6 should pass seven tests and catch seven planted bugs. If one of those differs, compare the timeline lines for that step before changing anything: they show the order of events, which is the thing that goes wrong.

Common mistakes

Testing shutdown only with Ctrl+C

Ctrl+C sends SIGINT, and uvicorn treats it specially: its handle_exit sets force_exit when a second SIGINT arrives. Orchestrators send SIGTERM, and a second SIGTERM does not force anything. A shutdown that looks fine at your terminal can behave differently under a deploy, so test with the real signal, as labkit.send_stop does.

Pointing the load balancer at the liveness route

If the platform’s probe calls /health, it never learns that you are draining. The drain only works if the thing that decides where traffic goes watches /ready, or if the delay is long enough that it does not need to.

Expecting lifespan teardown to keep serving

The teardown runs after uvicorn has closed the listening socket and waited for connections. It is the right place to close pools and wait for tracked tasks. It cannot delay the moment the door closes; that is what the drain delay is for.

Making the drain longer than the grace period

If drain delay plus graceful timeout exceeds the platform’s deadline, the forceful kill arrives first and every careful step you wrote is cut short. Add the numbers up on paper.

Trusting retries to cover for the gap

The “stop signal, nginx retries” row shows zero failures, and it is tempting to stop there. But the retry only works for requests the proxy considers safe to repeat, it costs latency, and not every router retries at all. Fix the gap at the source and keep the retry as a second line of defence.

Where to go next

Shutdown is one part of keeping a service healthy under change. If you are building on this, these tutorials on the same site cover the neighbouring problems: backpressure and load shedding for what to do when requests arrive faster than you can finish them, timeouts and deadlines for the client side of a slow response, idempotency keys for making a retried POST safe, a durable SQLite job queue for work that must survive a restart, WebSocket security for what open sockets experience when the server stops (that tutorial measured close code 1012), and zero-downtime schema migrations, because a rolling restart runs the old and new code side by side against one database.

Sources

• uvicorn documentation: Server Behavior (Graceful Process Shutdown) and Settings (--timeout-graceful-shutdown), plus the installed 0.54.0 source of uvicorn/server.py and uvicorn/protocols/http/h11_impl.py.
• FastAPI documentation: Lifespan Events.
• Kubernetes documentation: Pod Lifecycle (termination of Pods, preStop hooks, the 30-second default) and the tutorial Explore Termination Behavior for Pods And Their Endpoints.
• nginx documentation: ngx_http_proxy_module (proxy_next_upstream) and ngx_http_upstream_module (max_fails).
• Python documentation: signal (the signals Windows supports, CTRL_BREAK_EVENT).