How to Return RFC 9457 Problem Details From a FastAPI API and Stop Error Responses From Leaking Secrets

Every API fails sometimes. A conference session sells out, a token expires, a client sends nonsense, a database goes away. What matters is what the caller gets back when that happens. In this tutorial you will build a small FastAPI service called Seat Desk whose every error, from a missing route to a crash, is answered with an RFC 9457 problem details document. You will write the rules down as a contract, measure how far FastAPI’s defaults are from it, close the gap one handler at a time, prove the result with a test suite and eleven deliberate bugs, and finish with a client that reads these errors safely.

You do not need to know the RFC. Every term is defined before it is used. You should be comfortable reading Python and know what an HTTP status code and a response header are.

What problem details are and why they matter

A problem details document is a small JSON object that describes an error in a standard way. RFC 9457 was published in July 2023, it obsoletes RFC 7807, and its JSON form uses the media type application/problem+json. A client that sees that media type knows how to read the body, whichever API sent it.

The RFC defines five members. None of them is required by the data model (the RFC describes them as members a problem details object can have, and a missing type means about:blank), but each has a clear job:

• type is a URI that names the kind of problem. Consumers must treat it “as the problem type’s primary identifier”. The special value about:blank “indicates that the problem has no additional semantics beyond that of the HTTP status code”.
• title is a short human-readable summary of the type. It “SHOULD NOT change from occurrence to occurrence of the problem, except for localization”.
• status repeats the HTTP status code. It is advisory, but “Generators MUST use the same status code in the actual HTTP response, to assure that generic HTTP software that does not understand this format still behaves correctly.”
• detail is an explanation specific to this occurrence. The RFC says it is meant for people: “extensions are more suitable and less error-prone ways to obtain such information”.
• instance is “a URI reference that identifies the specific occurrence of the problem”. When it cannot be dereferenced, “it serves as a unique identifier for the problem occurrence that may be of significance to the server but is opaque to the client”.

On top of these, a problem type may add extension members of its own, such as a list of validation errors or the number of seats that are left. Clients must be prepared for that: “Clients consuming problem details MUST ignore any such extensions that they don’t recognize”.

Why does any of this matter? Because without a standard every client has to guess. As you will see in Step 2, FastAPI answers some errors with {"detail": "Method Not Allowed"}, some with a list of objects under the same detail key, and some with a plain text page. A client that wants to say “this session is full, try another one” ends up matching English sentences, which breaks the day somebody rewords a message. With problem details the client branches on a stable type URI and reads numbers from extension members.

What you need before you start

• Python 3.11 or newer on Windows, macOS or Linux. The package metadata for Starlette 1.9.0 and httpx2 2.16.0 asks for 3.11 or newer.
• A terminal and curl. The commands below use bash syntax; on Windows use Git Bash or adapt the virtual environment activation line.
• Every output in this tutorial was produced on Python 3.13.14 with FastAPI 0.156.8, Starlette 1.9.0, Pydantic 2.14.1, uvicorn 0.54.0, httpx2 2.16.0, requests 2.34.2 and pytest 9.1.1. Pinning those versions below keeps your output identical to mine, apart from generated ids and timings.

mkdir seat-desk && cd seat-desk
python -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install fastapi==0.156.8 uvicorn==0.54.0 httpx2==2.16.0 requests==2.34.2 pytest==9.1.1

FastAPI is the web framework (it brings Starlette and Pydantic with it), uvicorn is the server that runs it, and pytest runs our tests. Starlette’s test client works “using the httpx2 library”, which is why that package is on the list. We use requests for the probe script and the client because it is the library most readers already know.

Put every file in the seat-desk folder. Each Python block below starts with a comment that names its file. When the comment says “(continued)”, append the block to the end of that same file. A block that starts a new file never needs anything from a later step to run.

Step 1: Build Seat Desk with FastAPI’s default errors

Seat Desk lets signed-in attendees reserve seats in conference sessions. It is small on purpose, but it has every kind of failure we care about: unknown ids, wrong methods, missing and invalid tokens, invalid bodies, a session that is full, a rate limit, and a crash.

The data layer

First, an in-memory stand-in for a database. Two sessions exist: s1 with plenty of room and s2 with only two seats left.

# store.py - an in-memory stand-in for the database
from dataclasses import dataclass


@dataclass
class Session:
    id: str
    title: str
    capacity: int
    taken: int

    @property
    def available(self):
        return self.capacity - self.taken


class Store:
    def __init__(self):
        self.sessions = {
            "s1": Session("s1", "Opening keynote", capacity=100, taken=10),
            "s2": Session("s2", "Threat modeling workshop", capacity=20, taken=18),
        }

    def get(self, session_id):
        return self.sessions.get(session_id)

    def reserve(self, session, seats):
        session.taken += seats
        return {"session": session.id, "seats": seats, "seats_available": session.available}

The API

Now the service. Read it once from top to bottom; each piece is short.

# app.py - Seat Desk, a small seat-reservation API
import logging
import math
import time

from fastapi import Depends, FastAPI, HTTPException
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field

from problems import install
from store import Store

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")

TOKENS = {"tok-alice": "alice", "tok-bob": "bob", "tok-carol": "carol"}
bearer = HTTPBearer()


class ReservationIn(BaseModel):
    attendee_email: str = Field(pattern=r"^[^@\s]+@[^@\s]+\.[a-z]{2,}$")
    seats: int = Field(ge=1, le=4)
    invite_code: str | None = Field(default=None, pattern=r"^[A-Z0-9]{8}$")


def create_app(limit=5, window=60, clock=time.monotonic):
    app = FastAPI(title="Seat Desk")
    install(app)
    store = Store()
    hits = {}  # token -> (start of its window, requests in the window)

    def current_user(creds: HTTPAuthorizationCredentials = Depends(bearer)):
        if creds.credentials not in TOKENS:
            raise HTTPException(
                401,
                "The access token is unknown or expired.",
                headers={"WWW-Authenticate": 'Bearer error="invalid_token"'},
            )
        return creds.credentials

    def rate_limit(token: str = Depends(current_user)):
        now = clock()
        start, count = hits.get(token, (now, 0))
        if now - start >= window:
            start, count = now, 0
        if count >= limit:
            wait = math.ceil(window - (now - start))
            raise HTTPException(429, f"Try again in {wait} seconds.", headers={"Retry-After": str(wait)})
        hits[token] = (start, count + 1)
        return token

    @app.get("/sessions/{session_id}")
    def read_session(session_id: str):
        session = store.get(session_id)
        if session is None:
            raise HTTPException(404, f"No session with id {session_id!r}.")
        return {"id": session.id, "title": session.title, "seats_available": session.available}

    @app.post("/sessions/{session_id}/reservations", status_code=201)
    def reserve(session_id: str, body: ReservationIn, token: str = Depends(rate_limit)):
        session = store.get(session_id)
        if session is None:
            raise HTTPException(404, f"No session with id {session_id!r}.")
        if body.seats > session.available:
            raise HTTPException(409, f"Only {session.available} seats are left in this session.")
        return store.reserve(session, body.seats)

    @app.get("/sessions/{session_id}/export")
    def export(session_id: str, token: str = Depends(current_user)):
        # stand-in for a database outage whose message carries a connection string
        raise RuntimeError("could not reach postgresql://admin:[email protected]/seats")

    return app


app = create_app()

What each part does:

• ReservationIn is a Pydantic model that describes the JSON body of a reservation. FastAPI checks every request against it and rejects bad ones before our code runs. invite_code stands in for any secret a client might send, such as a promo code or an API key.
• current_user is a dependency, a function FastAPI runs before the route. It accepts only the three tokens in TOKENS. HTTPBearer reads the Authorization: Bearer ... header for us.
• rate_limit allows five reservation requests per token per 60 seconds. The clock argument lets tests control time later.
• The routes raise HTTPException for the failures a client can cause. The export route simulates a database outage whose error message carries a connection string with a password. That is a realistic thing for a driver to put in an exception, and it is exactly the kind of text that must never reach a client.
• install(app) is the hook for the error layer. It has no visible effect yet because no handler is registered. Each later step adds a handler to problems.py, so app.py never needs any wiring changes.

An empty error layer

Create problems.py with a registry. A handler is a function that turns one kind of exception into a response. The handles decorator records the handler, and install attaches all recorded handlers to an app.

# problems.py - turn every failure into an RFC 9457 problem details response
HANDLERS = {}


def handles(exc_class):
    """Register the decorated function as the handler for exc_class."""

    def register(func):
        HANDLERS[exc_class] = func
        return func

    return register


def install(app):
    """Attach every registered handler to a FastAPI app."""
    for exc_class, handler in HANDLERS.items():
        app.add_exception_handler(exc_class, handler)

Start the server

uvicorn app:app --port 8765 --reload

Leave it running in one terminal. The --reload flag restarts the server when you save a file, so later steps need no manual restarts. Open a second terminal in the same folder, activate the virtual environment there too, and continue. To check that the server is alive, open http://127.0.0.1:8765/sessions/s1 in a browser. You should see the keynote with 90 seats available.

Step 2: Write the contract, then measure the baseline

Before fixing anything, decide what “fixed” means, and write it as code. A check you can run beats a rule you have to remember.

What the contract checks

Each rule comes from the RFCs or is a house rule that we add because it makes clients simpler. RFC 9457 lets members be left out, but a client that cannot rely on type, title, status and instance cannot branch on the kind of problem or quote an id to support, so our contract asks for all four.

# contract.py - what "every error is a problem details document" means, written as code
import re
from http import HTTPStatus

MEDIA_TYPE = "application/problem+json"
SECRETS = ("hunter2", "10.0.0.5", "postgresql://", "Traceback", "RuntimeError")
REQUIRED_HEADERS = {401: "www-authenticate", 405: "allow", 429: "retry-after"}


def violations(resp):
    """Return the rules one error response breaks. An empty list means it conforms."""
    broken = []
    media = resp.headers.get("content-type", "").split(";")[0].strip().lower()
    if media != MEDIA_TYPE:
        broken.append(f"content-type is {media or 'missing'}")
    try:
        doc = resp.json()
    except ValueError:
        return broken + ["body is not JSON"]
    if not isinstance(doc, dict):
        return broken + ["body is not a JSON object"]
    for member, kind in (("type", str), ("title", str), ("status", int), ("instance", str)):
        value = doc.get(member)
        if not isinstance(value, kind) or isinstance(value, bool):
            broken.append(f"{member} is missing or not a {kind.__name__}")
    if "detail" in doc and not isinstance(doc["detail"], str):
        broken.append("detail is not a string")
    if isinstance(doc.get("status"), int) and doc["status"] != resp.status_code:
        broken.append(f"status {doc['status']} differs from HTTP {resp.status_code}")
    kind = doc.get("type")
    if kind == "about:blank":
        phrase = HTTPStatus(resp.status_code).phrase
        if doc.get("title") != phrase:
            broken.append(f"about:blank needs the HTTP phrase {phrase!r} as title")
    elif isinstance(kind, str) and not re.match(r"^[a-z][a-z0-9+.-]*:", kind):
        broken.append("type is a relative URI")
    needed = REQUIRED_HEADERS.get(resp.status_code)
    if needed and needed not in {name.lower() for name in resp.headers}:
        broken.append(f"{needed} header is missing")
    broken += [f"body leaks {word!r}" for word in SECRETS if word in resp.text]
    return broken

violations returns a list of the rules one response breaks, so an empty list means the response conforms. The SECRETS tuple holds the strings our lab must never leak: the invite code hunter2, the database password and address, and the words that give away a stack trace. The 401, 405 and 429 header rules follow RFC 9110: “The server generating a 401 response MUST send a WWW-Authenticate header field” and “The origin server MUST generate an Allow header field in a 405 response containing a list of the target resource’s currently supported methods”.

Ten requests that reach every layer

Next, a list of requests. Each one is chosen to fail in a different layer of the framework, because FastAPI builds its error responses in different places: the router (unknown path, wrong method), dependencies (authentication, rate limit), request validation, our own route code, and the unexpected.

# matrix.py - the ten requests every version of the API is judged by
GOOD = {"attendee_email": "[email protected]", "seats": 2, "invite_code": "ABCD1234"}
ALICE = {"Authorization": "Bearer tok-alice"}
CAROL = {"Authorization": "Bearer tok-carol"}
BAD_BODY = {"attendee_email": "dana", "seats": "two", "invite_code": "hunter2"}

CASES = [
    ("ok", "GET", "/sessions/s1", {}),
    ("session_missing", "GET", "/sessions/nope", {}),
    ("route_missing", "GET", "/venues", {}),
    ("wrong_method", "DELETE", "/sessions/s1", {}),
    ("no_token", "POST", "/sessions/s1/reservations", {"json": GOOD}),
    ("bad_token", "POST", "/sessions/s1/reservations", {"json": GOOD, "headers": {"Authorization": "Bearer nope"}}),
    ("bad_body", "POST", "/sessions/s1/reservations", {"json": BAD_BODY, "headers": ALICE}),
    ("session_full", "POST", "/sessions/s2/reservations", {"json": {**GOOD, "seats": 4}, "headers": ALICE}),
    ("rate_limited", "POST", "/sessions/s1/reservations", {"json": {**GOOD, "seats": 1}, "headers": CAROL, "repeat": 6}),
    ("server_error", "GET", "/sessions/s1/export", {"headers": ALICE}),
]


def run(send, only=None):
    """Send every case with send(method, path, **kwargs) and return {label: last response}."""
    results = {}
    for label, method, path, kwargs in CASES:
        if only and label not in only:
            continue
        kwargs = dict(kwargs)
        for _ in range(kwargs.pop("repeat", 1)):
            results[label] = send(method, path, **kwargs)
    return results

run takes any function that sends a request, so the same list drives a live server now and an in-process test client in Step 7. The repeat entry sends the rate-limit request six times, which is one more than the limit of five.

A probe for the live server

# probe.py - send the matrix to a running server and report what comes back
import json
import sys

import requests

from contract import violations
from matrix import run

BASE = "http://127.0.0.1:8765"


def send(method, path, **kwargs):
    return requests.request(method, BASE + path, timeout=10, **kwargs)


def show(label, resp):
    print(f"--- {label}: HTTP {resp.status_code}")
    print("content-type:", resp.headers.get("content-type"))
    for name in ("allow", "www-authenticate", "retry-after"):
        if name in resp.headers:
            print(f"{name}: {resp.headers[name]}")
    try:
        print(json.dumps(resp.json()))
    except ValueError:
        print(resp.text[:300])


if __name__ == "__main__":
    only = sys.argv[1:]
    results = run(send, only or None)
    if only:
        for label, resp in results.items():
            show(label, resp)
    else:
        print(f"{'case':16} {'status':6} {'content-type':26} contract")
        for label, resp in results.items():
            media = resp.headers.get("content-type", "-")[:26]
            if resp.status_code < 400:
                verdict = "not an error"
            else:
                broken = violations(resp)
                verdict = "ok" if not broken else f"{len(broken)} broken, first: {broken[0]}"
            print(f"{label:16} {resp.status_code:<6} {media:26} {verdict}")

Run it without arguments for a one-line verdict per request:

python probe.py

case             status content-type               contract
ok               200    application/json           not an error
session_missing  404    application/json           5 broken, first: content-type is application/json
route_missing    404    application/json           5 broken, first: content-type is application/json
wrong_method     405    application/json           5 broken, first: content-type is application/json
no_token         401    application/json           5 broken, first: content-type is application/json
bad_token        401    application/json           5 broken, first: content-type is application/json
bad_body         422    application/json           7 broken, first: content-type is application/json
session_full     409    application/json           5 broken, first: content-type is application/json
rate_limited     429    application/json           5 broken, first: content-type is application/json
server_error     500    text/plain; charset=utf-8  2 broken, first: content-type is text/plain

Every one of the nine error responses breaks the contract. Seven break five rules each (the wrong media type plus the four missing members), the 422 breaks seven, and the 500 breaks two. Now look at the raw responses. Passing request names to the probe prints them in full:

python probe.py session_missing wrong_method no_token bad_body server_error

--- session_missing: HTTP 404
content-type: application/json
{"detail": "No session with id 'nope'."}
--- wrong_method: HTTP 405
content-type: application/json
allow: GET
{"detail": "Method Not Allowed"}
--- no_token: HTTP 401
content-type: application/json
www-authenticate: Bearer
{"detail": "Not authenticated"}
--- bad_body: HTTP 422
content-type: application/json
{"detail": [{"type": "string_pattern_mismatch", "loc": ["body", "attendee_email"], "msg": "String should match pattern '^[^@\\s]+@[^@\\s]+\\.[a-z]{2,}$'", "input": "dana", "ctx": {"pattern": "^[^@\\s]+@[^@\\s]+\\.[a-z]{2,}$"}}, {"type": "int_parsing", "loc": ["body", "seats"], "msg": "Input should be a valid integer, unable to parse string as an integer", "input": "two"}, {"type": "string_pattern_mismatch", "loc": ["body", "invite_code"], "msg": "String should match pattern '^[A-Z0-9]{8}$'", "input": "hunter2", "ctx": {"pattern": "^[A-Z0-9]{8}$"}}]}
--- server_error: HTTP 500
content-type: text/plain; charset=utf-8
Internal Server Error

Three different shapes come out of one small service:

1. Errors you raise with HTTPException, plus the router’s own 404 and 405 and the 401 from HTTPBearer, are {"detail": "some text"} with the media type application/json. Note that headers such as allow and www-authenticate are already correct; we must not lose them.
2. A validation failure is also called detail, but now it is a list of objects, so the same key holds a string or a list depending on the error. Worse, each entry has an input member that sends the rejected value back, and the value here is the secret invite code hunter2. A secret that a client just submitted now travels into every log, proxy and error tracker that stores response bodies.
3. The crash is not JSON at all: Internal Server Error as text/plain. A client that always calls response.json() on an error will raise an exception of its own.

That is the baseline. Steps 3, 5 and 6 raise the number of conforming responses from zero to nine, and Step 4 makes the typed ones more useful.

Step 3: Turn the framework’s own errors into problem details

Most errors in a typical FastAPI app are HTTPExceptions, so one handler fixes a large share of them without any change to the routes. Append this to problems.py:

# problems.py (continued)
import logging
import uuid
from http import HTTPStatus

from fastapi import Request
from fastapi.responses import JSONResponse
from starlette.exceptions import HTTPException as StarletteHTTPException

log = logging.getLogger("problems")
MEDIA_TYPE = "application/problem+json"
BASE_URI = "https://api.example.com/problems/"
STANDARD_MEMBERS = {"type", "title", "status", "detail", "instance"}


def new_instance():
    """An opaque id for one occurrence of a problem (RFC 9457, section 3.1.5)."""
    return f"urn:uuid:{uuid.uuid4()}"


def phrase_for(status):
    """The HTTP status phrase ('Not Found' for 404), which RFC 9457 wants as the title of about:blank."""
    try:
        return HTTPStatus(status).phrase
    except ValueError:
        return "Error"


def problem_response(status, title, type_uri="about:blank", detail=None, headers=None, instance=None, **extensions):
    """Build the response for one occurrence of a problem."""
    clash = STANDARD_MEMBERS & extensions.keys()
    if clash:
        raise ValueError(f"extension members cannot reuse {sorted(clash)}")
    body = {"type": type_uri, "title": title, "status": status}
    if detail:
        body["detail"] = detail
    body["instance"] = instance or new_instance()
    body.update(extensions)
    log.info("problem %s status=%s instance=%s", type_uri, status, body["instance"])
    return JSONResponse(body, status_code=status, headers=headers, media_type=MEDIA_TYPE)


@handles(StarletteHTTPException)
async def on_http_exception(request: Request, exc: StarletteHTTPException):
    phrase = phrase_for(exc.status_code)
    detail = exc.detail if isinstance(exc.detail, str) and exc.detail != phrase else None
    return problem_response(exc.status_code, phrase, detail=detail, headers=exc.headers)

How it works

problem_response builds the document in a fixed order: the five standard members first, then any extension members. It refuses extension names that would overwrite a standard member. It also writes one log line per problem, including the new instance. That is the value a client can quote and a developer can search for.

instance is a urn:uuid: URI. It is not meant to be fetched: as the RFC says, a non-dereferenceable instance is an identifier “opaque to the client”.

on_http_exception is registered for Starlette’s HTTPException, not FastAPI’s, and that choice matters. The FastAPI documentation on handling errors explains that FastAPI’s class inherits from Starlette’s, and gives the rule: “you should register it for Starlette’s HTTPException”. Starlette raises its own class for unknown paths and wrong methods, so a handler registered only for FastAPI’s class never sees those.

The handler uses the HTTP phrase as the title and the about:blank type, which is what the RFC prescribes when a problem has no meaning beyond its status. It adds detail only when the exception carries text that says more than the phrase does, and it passes exc.headers through so that WWW-Authenticate and Allow survive.

Check it

Save the file. The server reloads. Run the probe again:

python probe.py

case             status content-type               contract
ok               200    application/json           not an error
session_missing  404    application/problem+json   ok
route_missing    404    application/problem+json   ok
wrong_method     405    application/problem+json   ok
no_token         401    application/problem+json   ok
bad_token        401    application/problem+json   ok
bad_body         422    application/json           7 broken, first: content-type is application/json
session_full     409    application/problem+json   ok
rate_limited     429    application/problem+json   ok
server_error     500    text/plain; charset=utf-8  2 broken, first: content-type is text/plain

Seven of nine now pass. Only the validation failure and the crash are left, and those need their own handlers. Look at three of the fixed responses:

python probe.py session_missing wrong_method no_token

--- session_missing: HTTP 404
content-type: application/problem+json
{"type": "about:blank", "title": "Not Found", "status": 404, "detail": "No session with id 'nope'.", "instance": "urn:uuid:00061b6d-83bc-4505-a207-013a48256921"}
--- wrong_method: HTTP 405
content-type: application/problem+json
allow: GET
{"type": "about:blank", "title": "Method Not Allowed", "status": 405, "instance": "urn:uuid:f2be6444-d19b-42c1-9119-6ccd36d1aefe"}
--- no_token: HTTP 401
content-type: application/problem+json
www-authenticate: Bearer
{"type": "about:blank", "title": "Unauthorized", "status": 401, "detail": "Not authenticated", "instance": "urn:uuid:44521134-bfaa-4989-b0fd-c9056f41daf6"}

Notice what carried over. The 405 still has allow: GET and the 401 still has www-authenticate: Bearer, because the handler forwards the exception’s headers. Notice also that the 405 has no detail: “Method Not Allowed” would only repeat the title. Step 9 shows what happens if you register the wrong class or forget the headers.

Step 4: Give your own errors a type and extension members

The 409 “Only 2 seats are left in this session.” is correct, but the number 2 is buried in an English sentence. A client that wants to offer “book 2 seats instead” would have to parse the text, which the RFC discourages. The fix is a problem type: a named kind of problem with its own URI, a fixed title, a fixed status, and extension members for the data.

The RFC says a new problem type definition “MUST document” three things: a type URI, a title, and the HTTP status code it is used with. It also says a problem type URI should lead somewhere that “explains how to resolve the problem”, and that a type definition may specify a Retry-After header. Append this to problems.py:

# problems.py (continued)
from typing import NamedTuple


class ProblemType(NamedTuple):
    status: int
    title: str
    advice: str


TYPES = {
    "session-not-found": ProblemType(404, "Session not found", "Check the session id and try again."),
    "session-full": ProblemType(409, "Session is full", "Ask for fewer seats (see seats_available) or pick another session."),
    "invalid-token": ProblemType(401, "Invalid access token", "Get a new token and repeat the request."),
    "rate-limited": ProblemType(429, "Too many requests", "Wait for the number of seconds in the Retry-After header."),
    "validation-error": ProblemType(422, "Your request is not valid", "Fix the fields listed in errors and send the request again."),
}


class Problem(Exception):
    """Raise this from a route or a dependency to answer with a typed problem."""

    def __init__(self, slug, detail=None, headers=None, **extensions):
        super().__init__(slug)
        self.slug, self.detail, self.headers, self.extensions = slug, detail, headers, extensions


@handles(Problem)
async def on_problem(request: Request, exc: Problem):
    kind = TYPES[exc.slug]
    return problem_response(kind.status, kind.title, BASE_URI + exc.slug, exc.detail, exc.headers, **exc.extensions)

TYPES is the registry: slug, status, title and a line of advice for each type. Problem is an exception you raise from a route or a dependency with the slug, a human sentence for detail, optional headers, and any extension members as keyword arguments. on_problem looks up the type and builds the response, so a title can never drift from one raise site to another.

The BASE_URI from Step 3 is a placeholder. Replace it with a URL you control. The RFC suggests a type URI that “ought to be under your control and stable over time”, because changing it later breaks every client that branches on it. Relative type URIs are legal but easy to get wrong, so we use absolute ones, as the RFC recommends.

Use the new types in the routes

A few edits to app.py turn the typed failures into Problems and add a small page that serves each type’s documentation, so the type URIs resolve. In a diff, a line that starts with a minus sign is removed, a line that starts with a plus sign is added, and the other lines show where you are in the file. Apply it by hand or with patch:

--- app.py
+++ app.py
@@ -1,3 +1,4 @@
 # app.py - Seat Desk, a small seat-reservation API
+import html
 import logging
 import math
@@ -5,8 +6,9 @@
 
 from fastapi import Depends, FastAPI, HTTPException
+from fastapi.responses import HTMLResponse
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from pydantic import BaseModel, Field
 
-from problems import install
+from problems import TYPES, Problem, install
 from store import Store
 
@@ -31,6 +33,6 @@
     def current_user(creds: HTTPAuthorizationCredentials = Depends(bearer)):
         if creds.credentials not in TOKENS:
-            raise HTTPException(
-                401,
+            raise Problem(
+                "invalid-token",
                 "The access token is unknown or expired.",
                 headers={"WWW-Authenticate": 'Bearer error="invalid_token"'},
@@ -45,5 +47,5 @@
         if count >= limit:
             wait = math.ceil(window - (now - start))
-            raise HTTPException(429, f"Try again in {wait} seconds.", headers={"Retry-After": str(wait)})
+            raise Problem("rate-limited", f"Try again in {wait} seconds.", headers={"Retry-After": str(wait)})
         hits[token] = (start, count + 1)
         return token
@@ -53,5 +55,5 @@
         session = store.get(session_id)
         if session is None:
-            raise HTTPException(404, f"No session with id {session_id!r}.")
+            raise Problem("session-not-found", f"No session with id {session_id!r}.")
         return {"id": session.id, "title": session.title, "seats_available": session.available}
 
@@ -60,7 +62,12 @@
         session = store.get(session_id)
         if session is None:
-            raise HTTPException(404, f"No session with id {session_id!r}.")
+            raise Problem("session-not-found", f"No session with id {session_id!r}.")
         if body.seats > session.available:
-            raise HTTPException(409, f"Only {session.available} seats are left in this session.")
+            raise Problem(
+                "session-full",
+                f"Only {session.available} seats are left in this session.",
+                seats_available=session.available,
+                seats_requested=body.seats,
+            )
         return store.reserve(session, body.seats)
 
@@ -70,4 +77,11 @@
         raise RuntimeError("could not reach postgresql://admin:[email protected]/seats")
 
+    @app.get("/problems/{slug}", response_class=HTMLResponse, include_in_schema=False)
+    def problem_docs(slug: str):
+        kind = TYPES.get(slug)
+        if kind is None:
+            raise HTTPException(404)
+        return f"<h1>{html.escape(kind.title)}</h1><p>HTTP status {kind.status}.</p><p>{html.escape(kind.advice)}</p>"
+
     return app
 

Check it

python probe.py session_missing session_full bad_token rate_limited

--- session_missing: HTTP 404
content-type: application/problem+json
{"type": "https://api.example.com/problems/session-not-found", "title": "Session not found", "status": 404, "detail": "No session with id 'nope'.", "instance": "urn:uuid:5246f447-596a-4655-a148-0b394a6348e7"}
--- bad_token: HTTP 401
content-type: application/problem+json
www-authenticate: Bearer error="invalid_token"
{"type": "https://api.example.com/problems/invalid-token", "title": "Invalid access token", "status": 401, "detail": "The access token is unknown or expired.", "instance": "urn:uuid:d510c236-786f-476d-aebe-d3ce4164f093"}
--- session_full: HTTP 409
content-type: application/problem+json
{"type": "https://api.example.com/problems/session-full", "title": "Session is full", "status": 409, "detail": "Only 2 seats are left in this session.", "instance": "urn:uuid:11594c26-9ac0-4668-8b36-efa3973c3fab", "seats_available": 2, "seats_requested": 4}
--- rate_limited: HTTP 429
content-type: application/problem+json
retry-after: 60
{"type": "https://api.example.com/problems/rate-limited", "title": "Too many requests", "status": 429, "detail": "Try again in 60 seconds.", "instance": "urn:uuid:d16ea6fa-8c61-4f15-ae04-a6134e6fb8a7"}

The summary table does not change in this step, because those cases already conformed. What changed is the information in them:

• type is now a URL that names the problem, and title is the same sentence every time.
• The 409 carries seats_available and seats_requested as numbers. The extension names are three or more characters, start with a letter and use only letters and underscores, which is what the RFC recommends so that they can be serialized in formats other than JSON.
• The invalid token keeps its WWW-Authenticate header, now with the challenge error="invalid_token" that RFC 6750 defines for “The access token provided is expired, revoked, malformed, or invalid for other reasons”.
• The 429 carries retry-after: 60. The value is a whole number of seconds, the form RFC 9110 calls delay-seconds.

To confirm that the type URIs resolve, open http://127.0.0.1:8765/problems/session-full in a browser. In production the same path would sit under your real BASE_URI.

Step 5: Report validation errors without echoing the input

FastAPI’s default 422 gives a client three kinds of useful information: what was wrong, where, and which value was rejected. The third one is the risk. The RFC’s own example for validation errors shows the safer shape: an errors extension that is a list of objects, each with a detail that describes the issue and a pointer that locates it in the request body using a JSON Pointer, written as a URI fragment such as #/age. No input value appears in it. That example also uses status 422, as FastAPI does. Append this to problems.py:

# problems.py (continued)
from urllib.parse import quote

from fastapi.exceptions import RequestValidationError


def pointer(loc):
    """('body', 'items', 0, 'a/b') becomes '#/items/0/a~1b': a JSON Pointer (RFC 6901) as a URI fragment."""
    parts = [str(part).replace("~", "~0").replace("/", "~1") for part in loc[1:]]
    return "#" + "".join("/" + quote(part, safe="~!$&'()*+,;=:@") for part in parts)


def describe(error):
    """One entry of the errors extension: the message and the location, never the input."""
    where = error["loc"][0]
    item = {"detail": error["msg"]}
    if where == "body":
        # a JSON syntax error is located by character offset, not by a path of member names
        item["pointer"] = "#" if error["type"] == "json_invalid" else pointer(error["loc"])
    else:
        item["parameter"] = str(error["loc"][-1])
        item["location"] = str(where)
    return item


@handles(RequestValidationError)
async def on_validation_error(request: Request, exc: RequestValidationError):
    kind = TYPES["validation-error"]
    errors = [describe(error) for error in exc.errors()]
    detail = f"{len(errors)} problem(s) found in the request."
    return problem_response(kind.status, kind.title, BASE_URI + "validation-error", detail, errors=errors)

How it works

• pointer builds the JSON Pointer. RFC 6901 requires the characters ~ and / inside a name to be written as ~0 and ~1, and the URI fragment form requires “percent-encoding those characters not allowed by the fragment rule”, which is why a field called first name becomes #/first%20name.
• describe keeps only the message and the location of each error. The input and ctx members that Pydantic provides never leave the server. Errors in the query string, the path or a header have no body location, so they get a parameter name and a location instead.
• A broken JSON document is reported by character offset, not by member path, so its pointer is #, which means “the whole document”.
• The detail line counts the problems, and the details of each live in errors, so the standard detail never has to hold a list.

Check it

python probe.py

case             status content-type               contract
ok               200    application/json           not an error
session_missing  404    application/problem+json   ok
route_missing    404    application/problem+json   ok
wrong_method     405    application/problem+json   ok
no_token         401    application/problem+json   ok
bad_token        401    application/problem+json   ok
bad_body         422    application/problem+json   ok
session_full     409    application/problem+json   ok
rate_limited     429    application/problem+json   ok
server_error     500    text/plain; charset=utf-8  2 broken, first: content-type is text/plain

python probe.py bad_body

--- bad_body: HTTP 422
content-type: application/problem+json
{"type": "https://api.example.com/problems/validation-error", "title": "Your request is not valid", "status": 422, "detail": "3 problem(s) found in the request.", "instance": "urn:uuid:07869c4e-091c-44be-83af-f37e663ad843", "errors": [{"detail": "String should match pattern '^[^@\\s]+@[^@\\s]+\\.[a-z]{2,}$'", "pointer": "#/attendee_email"}, {"detail": "Input should be a valid integer, unable to parse string as an integer", "pointer": "#/seats"}, {"detail": "String should match pattern '^[A-Z0-9]{8}$'", "pointer": "#/invite_code"}]}

The three bad fields are listed with pointers, the secret hunter2 is gone, and the 422 passes. Only the crash remains.

The messages themselves still reveal the pattern that a field must match. That is useful for a client developer and is the kind of detail that the RFC says “ought to focus on helping the client correct the problem”. If your patterns are confidential, replace the Pydantic message with your own.

Step 6: Hide crashes behind an instance id

A crash is not the client’s fault, and the client cannot fix it. The RFC is direct about this: “Problem details are not a debugging tool for the underlying implementation”, and generators are “encouraged to avoid making implementation details such as a stack dump available through the HTTP interface”. So the response says that something went wrong and gives an id, and the full story goes to the log under the same id. Append this to problems.py:

# problems.py (continued)
@handles(Exception)
async def on_unexpected(request: Request, exc: Exception):
    instance = new_instance()
    log.error("unhandled %s instance=%s %s %s", type(exc).__name__, instance, request.method, request.url.path, exc_info=exc)
    return problem_response(
        500,
        phrase_for(500),
        instance=instance,
        detail="The server hit an unexpected error. Quote the instance value when you contact support.",
    )

Why a handler for Exception is special

A handler registered for Exception is not run by the same layer as the others. Starlette’s middleware documentation describes ServerErrorMiddleware as “Ensures that application exceptions may return a custom 500 page, or display an application traceback in DEBUG mode” and adds that it is “always the outermost middleware layer”. That layer sends our response and then raises the exception again. The source says why: “We always continue to raise the exception. This allows servers to log the error, or allows test clients to optionally raise the error within the test case.”

Two consequences follow. The server will print its own traceback after our response, and a test that calls the crash route needs special care. A related warning comes from Starlette’s exceptions page: “If debug is enabled and an error occurs, then instead of using the installed 500 handler, Starlette will respond with a traceback response.” Never run production with debug on.

Check it

python probe.py

case             status content-type               contract
ok               200    application/json           not an error
session_missing  404    application/problem+json   ok
route_missing    404    application/problem+json   ok
wrong_method     405    application/problem+json   ok
no_token         401    application/problem+json   ok
bad_token        401    application/problem+json   ok
bad_body         422    application/problem+json   ok
session_full     409    application/problem+json   ok
rate_limited     429    application/problem+json   ok
server_error     500    application/problem+json   ok

All nine error responses conform. Look at the crash:

python probe.py server_error

--- server_error: HTTP 500
content-type: application/problem+json
{"type": "about:blank", "title": "Internal Server Error", "status": 500, "detail": "The server hit an unexpected error. Quote the instance value when you contact support.", "instance": "urn:uuid:b370b219-4f08-455e-95da-355fa9ad639f"}

The client learns that it was a 500 and gets an id. Now look in the terminal where uvicorn is running. Between the stack frames (left out here) you will find these lines for that request:

ERROR problems: unhandled RuntimeError instance=urn:uuid:b370b219-4f08-455e-95da-355fa9ad639f GET /sessions/s1/export
RuntimeError: could not reach postgresql://admin:[email protected]/seats
INFO problems: problem about:blank status=500 instance=urn:uuid:b370b219-4f08-455e-95da-355fa9ad639f
ERROR:    Exception in ASGI application
RuntimeError: could not reach postgresql://admin:[email protected]/seats

The first line is our handler: the exception type, the same instance that the client received, the method and the path. The next line ends the traceback that our handler logged, and it is where the secret lives, which is where it belongs. The INFO line is the log entry that problem_response writes for every problem. The last two lines are uvicorn’s own report of the exception that Starlette raised again.

Testing a crash

That re-raise has a trap for your tests. Starlette’s test client re-raises server exceptions by default. Its documentation says: “Occasionally you might want to test the content of 500 error responses, rather than allowing the client to raise the server exception.” The remedy is raise_server_exceptions=False. This small script shows both behaviors:

# step_testclient.py - why a crash test needs raise_server_exceptions=False
import logging

from fastapi.testclient import TestClient

from app import create_app
from matrix import ALICE

logging.disable(logging.CRITICAL)  # keep this script's output free of server log lines
app = create_app()

try:
    TestClient(app).get("/sessions/s1/export", headers=ALICE)
except RuntimeError as err:
    print("default TestClient re-raised after the handler ran:", err)

resp = TestClient(app, raise_server_exceptions=False).get("/sessions/s1/export", headers=ALICE)
print(resp.status_code, resp.headers["content-type"])
print(resp.json())

python step_testclient.py

default TestClient re-raised after the handler ran: could not reach postgresql://admin:[email protected]/seats
500 application/problem+json
{'type': 'about:blank', 'title': 'Internal Server Error', 'status': 500, 'detail': 'The server hit an unexpected error. Quote the instance value when you contact support.', 'instance': 'urn:uuid:88cc088c-a769-43e6-a6c3-4e9c5e22fbd1'}

With the default client the test sees a RuntimeError, carrying the secret, even though the handler had already produced a perfect 500 response. With raise_server_exceptions=False the test receives that response. We use the second form in every test below.

Step 7: Turn the contract into a test suite

The probe needs a running server and a human to read its output. A test suite runs in a second and fails loudly. Create test_problems.py:

# test_problems.py - the contract, checked one request at a time
import logging

import pytest
from fastapi.testclient import TestClient

from app import create_app
from contract import MEDIA_TYPE, violations
from matrix import ALICE, CAROL, CASES, GOOD, run
from problems import BASE_URI, TYPES, describe, pointer, problem_response

ERROR_CASES = [label for label, *_ in CASES if label != "ok"]


class Clock:
    def __init__(self):
        self.now = 1000.0

    def __call__(self):
        return self.now


@pytest.fixture
def clock():
    return Clock()


@pytest.fixture
def client(clock):
    # raise_server_exceptions=False: a crash must come back as a response, not as an exception in the test
    return TestClient(create_app(clock=clock), raise_server_exceptions=False)


@pytest.fixture
def results(client):
    return run(client.request)


@pytest.mark.parametrize("label", ERROR_CASES)
def test_every_error_conforms(results, label):
    assert violations(results[label]) == []


def test_session_full_carries_its_extensions(results):
    body = results["session_full"].json()
    assert body["type"] == BASE_URI + "session-full"
    assert (body["seats_available"], body["seats_requested"]) == (2, 4)


def test_validation_errors_point_at_the_fields(results):
    errors = results["bad_body"].json()["errors"]
    assert {error["pointer"] for error in errors} == {"#/attendee_email", "#/seats", "#/invite_code"}
    assert all(set(error) == {"detail", "pointer"} for error in errors)


def test_a_broken_json_document_is_pointed_at_as_a_whole(client):
    headers = {**ALICE, "Content-Type": "application/json"}
    resp = client.post("/sessions/s1/reservations", content='{"seats": ', headers=headers)
    assert resp.status_code == 422
    assert resp.json()["errors"][0]["pointer"] == "#"


def test_secrets_in_bad_input_are_not_echoed(client):
    resp = client.post("/sessions/s1/reservations", headers=ALICE, json={**GOOD, "invite_code": "Sup3rSecret!"})
    assert resp.status_code == 422
    assert "Sup3rSecret" not in resp.text


def test_instance_is_new_for_every_occurrence(client):
    first, second = (client.get("/sessions/nope").json()["instance"] for _ in range(2))
    assert first != second
    assert first.startswith("urn:uuid:")


def test_title_is_stable_while_detail_is_specific(client):
    one, other = client.get("/sessions/nope").json(), client.get("/sessions/other").json()
    assert one["title"] == other["title"]
    assert one["detail"] != other["detail"]


def test_a_crash_keeps_its_secrets_in_the_log(client, caplog):
    with caplog.at_level(logging.INFO, logger="problems"):
        resp = client.get("/sessions/s1/export", headers=ALICE)
    instance = resp.json()["instance"]
    assert resp.status_code == 500
    assert f"instance={instance}" in caplog.text
    assert "hunter2" in caplog.text
    assert "hunter2" not in resp.text


def test_retry_after_counts_down(client, clock):
    def send():
        return client.post("/sessions/s1/reservations", headers=CAROL, json={**GOOD, "seats": 1})

    assert [send().status_code for _ in range(5)] == [201] * 5
    blocked = send()
    assert (blocked.status_code, blocked.headers["retry-after"]) == (429, "60")
    clock.now += 45
    assert send().headers["retry-after"] == "15"
    clock.now += 15
    assert send().status_code == 201


def test_extension_members_cannot_shadow_standard_ones():
    with pytest.raises(ValueError):
        problem_response(400, "Bad Request", type="https://evil.example/")


@pytest.mark.parametrize(
    "loc, expected",
    [
        (("body",), "#"),
        (("body", "seats"), "#/seats"),
        (("body", 0, "name"), "#/0/name"),
        (("body", "a/b", "c~d"), "#/a~1b/c~0d"),
        (("body", "first name"), "#/first%20name"),
    ],
)
def test_json_pointers(loc, expected):
    assert pointer(loc) == expected


def test_query_problems_name_the_parameter():
    error = {"loc": ("query", "limit"), "msg": "Input should be less than or equal to 50", "type": "less_than_equal"}
    assert describe(error) == {"detail": error["msg"], "parameter": "limit", "location": "query"}


@pytest.mark.parametrize("slug", sorted(TYPES))
def test_every_type_uri_resolves(client, slug):
    page = client.get("/problems/" + slug)
    assert page.status_code == 200
    assert TYPES[slug].title in page.text


OPERATION = {
    "session_missing": ("get", "/sessions/{session_id}"),
    "no_token": ("post", "/sessions/{session_id}/reservations"),
    "bad_token": ("post", "/sessions/{session_id}/reservations"),
    "bad_body": ("post", "/sessions/{session_id}/reservations"),
    "session_full": ("post", "/sessions/{session_id}/reservations"),
    "rate_limited": ("post", "/sessions/{session_id}/reservations"),
    "server_error": ("get", "/sessions/{session_id}/export"),
}


def test_openapi_describes_every_error_the_matrix_saw(client, results):
    paths = client.get("/openapi.json").json()["paths"]
    for label, (method, template) in OPERATION.items():
        declared = paths[template][method]["responses"]
        status = str(results[label].status_code)
        assert status in declared, f"{label}: {status} is not documented for {method.upper()} {template}"
        assert MEDIA_TYPE in declared[status]["content"], f"{label}: {status} is not described as {MEDIA_TYPE}"

What the tests do

• The client fixture builds a fresh app for every test, with a fake Clock so that the rate limit can be tested without waiting. results runs the whole matrix against it.
• test_every_error_conforms is the contract: one test per error case, each asserting that violations is empty.
• The other tests cover what the contract cannot see. The 409 carries its extensions. Validation errors point at the right fields, a broken JSON document points at #, and a secret in a bad field is not echoed. instance is new every time while title stays stable and detail changes. A crash keeps its secret in the log and out of the response, and the log line carries the same instance. Retry-After counts down as the fake clock advances. Extension members cannot shadow standard ones, JSON Pointers escape ~, / and spaces, and every type URI leads to a documentation page.
• The last test compares the API documentation with reality, and we will meet it in a moment.

Run it

python -m pytest -q --tb=no -rf test_problems.py

.............................F [100%]
=== short test summary info ===
FAILED test_problems.py::test_openapi_describes_every_error_the_matrix_saw - AssertionError: session_missing: 404 is not documented for GET /sessions/{session_id}
1 failed, 29 passed in 0.92s

Twenty-nine tests pass and one fails, which is useful: the code behaves, but the generated API documentation does not know about the new error shapes. The failing message says that a 404 is not documented for GET /sessions/{session_id}. That is Step 8.

Step 8: Document the errors so the generated docs tell the truth

FastAPI builds an OpenAPI description of your API from the code, and tools use it to generate clients, render documentation and fuzz-test the service. Out of the box it documents a 422 as application/json in the shape of its default HTTPValidationError, which is no longer what we send, and it knows nothing about the other failures. FastAPI’s additional responses page says: “You can declare additional responses, with additional status codes, media types, descriptions, etc.” We use that to describe every error as application/problem+json. Append this to problems.py:

# problems.py (continued)
PROBLEM_SCHEMA = {
    "type": "object",
    "properties": {
        "type": {"type": "string", "format": "uri-reference"},
        "title": {"type": "string"},
        "status": {"type": "integer"},
        "detail": {"type": "string"},
        "instance": {"type": "string", "format": "uri-reference"},
    },
    "required": ["type", "title", "status", "instance"],
}


def documented(*statuses):
    """Build the responses= argument: each status is described as application/problem+json."""
    return {
        status: {"description": phrase_for(status), "content": {MEDIA_TYPE: {"schema": PROBLEM_SCHEMA}}}
        for status in statuses
    }

PROBLEM_SCHEMA describes the standard members and marks the four that our contract requires. It is written inline so that it can be attached directly to the application/problem+json media type. documented(401, 404) builds the responses argument for a route.

Now declare the errors on each route. This diff adds three responses arguments and one import:

--- app.py
+++ app.py
@@ -10,5 +10,5 @@
 from pydantic import BaseModel, Field
 
-from problems import TYPES, Problem, install
+from problems import TYPES, Problem, documented, install
 from store import Store
 
@@ -51,5 +51,5 @@
         return token
 
-    @app.get("/sessions/{session_id}")
+    @app.get("/sessions/{session_id}", responses=documented(404))
     def read_session(session_id: str):
         session = store.get(session_id)
@@ -58,5 +58,5 @@
         return {"id": session.id, "title": session.title, "seats_available": session.available}
 
-    @app.post("/sessions/{session_id}/reservations", status_code=201)
+    @app.post("/sessions/{session_id}/reservations", status_code=201, responses=documented(401, 404, 409, 422, 429))
     def reserve(session_id: str, body: ReservationIn, token: str = Depends(rate_limit)):
         session = store.get(session_id)
@@ -72,5 +72,5 @@
         return store.reserve(session, body.seats)
 
-    @app.get("/sessions/{session_id}/export")
+    @app.get("/sessions/{session_id}/export", responses=documented(401, 500))
     def export(session_id: str, token: str = Depends(current_user)):
         # stand-in for a database outage whose message carries a connection string

Check it

This script prints what the generated documentation says about a rejected reservation, before and after:

# step_openapi.py - what the generated API docs promise for a rejected reservation
import logging

from fastapi.testclient import TestClient

from app import create_app as documented_app
from app_v2 import create_app as plain_app

logging.getLogger().handlers.clear()  # keep this script's output free of server log lines


def describe_422(create):
    schema = TestClient(create()).get("/openapi.json").json()
    content = schema["paths"]["/sessions/{session_id}/reservations"]["post"]["responses"]["422"]["content"]
    for media, body in content.items():
        found = body["schema"]
        print(f"  {media}: {found.get('$ref') or 'inline schema, required: ' + ', '.join(found['required'])}")


print("before documenting the errors (app_v2.py):")
describe_422(plain_app)
print("after (app.py):")
describe_422(documented_app)

python step_openapi.py

before documenting the errors (app_v2.py):
  application/json: #/components/schemas/HTTPValidationError
after (app.py):
  application/problem+json: inline schema, required: type, title, status, instance

Before, a generated client would have been told that a 422 is an HTTPValidationError with a detail list. After, it is a problem document. Run the tests again:

python -m pytest -q test_problems.py

.............................. [100%]
30 passed in 0.95s

All thirty pass. The documentation test is the one that will keep them honest later: add a new error to a route without documenting it and the suite fails.

Step 9: Break it on purpose

A test suite that has never failed has never been shown to work. This script copies the finished code into scratch folders, injects one bug into each copy, and reports whether the tests notice. It also shows what the contract matrix alone sees, so you can tell which checks do the work. Create mutations.py:

# mutations.py - break the error layer on purpose and count which checks notice
import os
import pathlib
import re
import shutil
import subprocess
import sys

LAB = pathlib.Path(__file__).resolve().parent
SOURCES = ["app.py", "client.py", "contract.py", "matrix.py", "problems.py", "store.py", "test_client.py", "test_problems.py"]

MUTANTS = [
    ("handlers registered for FastAPI's HTTPException", "problems.py",
     "@handles(StarletteHTTPException)", '@handles(__import__("fastapi").HTTPException)'),
    ("exc.headers dropped from framework errors", "problems.py",
     "detail=detail, headers=exc.headers)", "detail=detail, headers=None)"),
    ("validation errors echo the input", "problems.py",
     'item = {"detail": error["msg"]}', 'item = {"detail": error["msg"], "input": error.get("input")}'),
    ("500 detail carries the exception text", "problems.py",
     'detail="The server hit an unexpected error. Quote the instance value when you contact support.",',
     'detail=f"{type(exc).__name__}: {exc}",'),
    ("body status differs from the HTTP status", "problems.py",
     'body = {"type": type_uri, "title": title, "status": status}',
     'body = {"type": type_uri, "title": title, "status": 400}'),
    ("title changes with every occurrence", "problems.py",
     "return problem_response(kind.status, kind.title, BASE_URI + exc.slug,",
     'return problem_response(kind.status, f"{kind.title}: {exc.detail}", BASE_URI + exc.slug,'),
    ("one fixed instance for every occurrence", "problems.py",
     'body["instance"] = instance or new_instance()',
     'body["instance"] = "urn:uuid:00000000-0000-0000-0000-000000000000"'),
    ("plain application/json instead of problem+json", "problems.py",
     "media_type=MEDIA_TYPE)", 'media_type="application/json")'),
    ("JSON Pointer escaping removed", "problems.py",
     'str(part).replace("~", "~0").replace("/", "~1")', "str(part)"),
    ("Retry-After header forgotten", "app.py",
     'headers={"Retry-After": str(wait)})', "headers=None)"),
    ("422 left out of the OpenAPI documentation", "app.py",
     "documented(401, 404, 409, 422, 429)", "documented(401, 404, 409, 429)"),
]

# what the matrix itself sees: the first rule broken by the first failing case
MATRIX_VIEW = """
import logging
from fastapi.testclient import TestClient
from app import create_app
from contract import violations
from matrix import run

logging.getLogger().handlers.clear()
results = run(TestClient(create_app(), raise_server_exceptions=False).request)
for label, resp in results.items():
    broken = violations(resp) if resp.status_code >= 400 else []
    if broken:
        print(label + ': ' + broken[0])
        break
"""


def run_tests(folder):
    env = dict(os.environ, COLUMNS="200", PYTHONUTF8="1")
    done = subprocess.run([sys.executable, "-m", "pytest", "-q", "--tb=no", "-rf", "-p", "no:cacheprovider"],
                          cwd=folder, capture_output=True, text=True, env=env)
    failed = re.findall(r"^FAILED (\S+)", done.stdout, flags=re.M)
    seen = subprocess.run([sys.executable, "-c", MATRIX_VIEW], cwd=folder, capture_output=True, text=True, env=env)
    return failed, seen.stdout.strip() or "the matrix sees nothing wrong"


def main():
    scratch = LAB / "mut"
    shutil.rmtree(scratch, ignore_errors=True)
    rows = []
    for name, target, old, new in [("none, the finished code", None, None, None)] + MUTANTS:
        folder = scratch / f"m{len(rows)}"
        folder.mkdir(parents=True)
        for source in SOURCES:
            shutil.copy(LAB / source, folder / source)
        if target:
            text = (folder / target).read_text(encoding="utf-8")
            assert text.count(old) == 1, (name, text.count(old))
            (folder / target).write_text(text.replace(old, new), encoding="utf-8", newline="\n")
        failed, seen = run_tests(folder)
        short = [item.split("::")[1] for item in failed]
        rows.append((name, len(failed), short, seen))
    for number, (name, count, short, seen) in enumerate(rows):
        print(f"[{number}] {name}")
        print(f"    tests failing: {count}" + (f", first: {short[0]}" if short else ""))
        print(f"    matrix says:   {seen}")
    shutil.rmtree(scratch, ignore_errors=True)
    print("undetected by the test suite:", [row[0] for row in rows[1:] if row[1] == 0] or "none")
    print("invisible to the matrix alone:", [row[0] for row in rows[1:] if row[3].startswith("the matrix sees nothing")] or "none")


if __name__ == "__main__":
    main()

python mutations.py

[0] none, the finished code
    tests failing: 0
    matrix says:   the matrix sees nothing wrong
[1] handlers registered for FastAPI's HTTPException
    tests failing: 2, first: test_every_error_conforms[route_missing]
    matrix says:   route_missing: content-type is application/json
[2] exc.headers dropped from framework errors
    tests failing: 2, first: test_every_error_conforms[wrong_method]
    matrix says:   wrong_method: allow header is missing
[3] validation errors echo the input
    tests failing: 4, first: test_every_error_conforms[bad_body]
    matrix says:   bad_body: body leaks 'hunter2'
[4] 500 detail carries the exception text
    tests failing: 2, first: test_every_error_conforms[server_error]
    matrix says:   server_error: body leaks 'hunter2'
[5] body status differs from the HTTP status
    tests failing: 9, first: test_every_error_conforms[session_missing]
    matrix says:   session_missing: status 400 differs from HTTP 404
[6] title changes with every occurrence
    tests failing: 1, first: test_title_is_stable_while_detail_is_specific
    matrix says:   the matrix sees nothing wrong
[7] one fixed instance for every occurrence
    tests failing: 1, first: test_instance_is_new_for_every_occurrence
    matrix says:   the matrix sees nothing wrong
[8] plain application/json instead of problem+json
    tests failing: 9, first: test_every_error_conforms[session_missing]
    matrix says:   session_missing: content-type is application/json
[9] JSON Pointer escaping removed
    tests failing: 1, first: test_json_pointers[loc3-#/a~1b/c~0d]
    matrix says:   the matrix sees nothing wrong
[10] Retry-After header forgotten
    tests failing: 2, first: test_every_error_conforms[rate_limited]
    matrix says:   rate_limited: retry-after header is missing
[11] 422 left out of the OpenAPI documentation
    tests failing: 1, first: test_openapi_describes_every_error_the_matrix_saw
    matrix says:   the matrix sees nothing wrong
undetected by the test suite: none
invisible to the matrix alone: ['title changes with every occurrence', 'one fixed instance for every occurrence', 'JSON Pointer escaping removed', '422 left out of the OpenAPI documentation']

The tests catch all eleven bugs. A few rows are worth reading closely.

• Bugs 1 and 2 are the two shortcuts hinted at in Step 3. Registering the handler for FastAPI’s HTTPException lets the router’s 404 and 405 slip through in the old format. Forgetting exc.headers keeps the right shape but loses the Allow and WWW-Authenticate headers that the RFCs require. Both look right in a quick manual test of a route that raises HTTPException.
• Bugs 3 and 4 are leaks. Echoing the input and putting the exception text into detail are caught by the secret check.
• Bug 5 shows why status needs a check: the RFC itself warns that duplicating the code brings “the possibility of disagreement between the two”. Nine tests fail when it happens.
• Four bugs are invisible to the matrix alone: a title that changes with each occurrence, one fixed instance, broken pointer escaping, and a missing entry in the OpenAPI document. Only the targeted tests from Step 7 and Step 8 catch them. That is the reason the suite has both a broad contract check and narrow tests.

Step 10: Read problem details on the client side

A server that speaks problem details is only half the job. The client must read the document without trusting the network too much. The RFC gives the rules: a member must be ignored when its “value type does not match the specified type”, unknown extensions must be ignored, the type URI is the identifier to branch on, and consumers “SHOULD NOT automatically dereference the type URI”. The HTTP status code is what generic software trusts, so the client uses it and treats the body’s status as a hint. Create client.py:

# client.py - read problem details without trusting the server more than RFC 9457 does
from dataclasses import dataclass, field
from http import HTTPStatus

MEDIA_TYPE = "application/problem+json"
STANDARD_MEMBERS = ("type", "title", "status", "detail", "instance")


@dataclass(eq=False)
class ApiError(Exception):
    status: int
    title: str
    type: str = "about:blank"
    detail: str | None = None
    instance: str | None = None
    retry_after: int | None = None
    extensions: dict = field(default_factory=dict)
    is_problem: bool = True

    def __str__(self):
        return f"{self.status} {self.title} ({self.type}): {self.detail}"


def phrase_for(status):
    try:
        return HTTPStatus(status).phrase
    except ValueError:
        return "Error"


def member(doc, name, kind):
    """RFC 9457, section 3.1: a member whose value has the wrong type must be ignored."""
    value = doc.get(name)
    return value if isinstance(value, kind) and not isinstance(value, bool) else None


def parse_error(resp):
    """Turn a failed response into an ApiError, whatever the server or a proxy sent."""
    status = resp.status_code  # generic HTTP software trusts this one; the body's status is only advisory
    retry = resp.headers.get("retry-after", "")
    retry_after = int(retry) if retry.isdigit() else None
    media = resp.headers.get("content-type", "").split(";")[0].strip().lower()
    try:
        doc = resp.json() if media == MEDIA_TYPE else None
    except ValueError:
        doc = None
    if not isinstance(doc, dict):
        return ApiError(status, phrase_for(status), retry_after=retry_after, is_problem=False)
    return ApiError(
        status,
        member(doc, "title", str) or phrase_for(status),
        type=member(doc, "type", str) or "about:blank",
        detail=member(doc, "detail", str),
        instance=member(doc, "instance", str),
        retry_after=retry_after,
        extensions={key: value for key, value in doc.items() if key not in STANDARD_MEMBERS},
    )


def call(send, method, url, **kwargs):
    """Send a request with send(method, url, ...). Return the JSON body or raise ApiError."""
    resp = send(method, url, timeout=10, **kwargs)
    if resp.status_code >= 400:
        raise parse_error(resp)
    return resp.json()

parse_error never raises on a malformed response. It returns an ApiError in every case, with is_problem set to False when the body was not a problem document at all, for example an HTML error page from a proxy or a plain text answer from middleware. retry_after is read only when it is a whole number of seconds, because Retry-After can also be a date and this client chooses not to guess.

Test the parser

# test_client.py - the consumer side: read what the server sent, whatever it is
import requests

from client import parse_error


def make(status, content_type, body, headers=None):
    resp = requests.Response()
    resp.status_code = status
    resp._content = body.encode()
    resp.headers.update({"Content-Type": content_type, **(headers or {})})
    return resp


def test_reads_a_problem_and_keeps_its_extensions():
    body = '{"type": "https://x.test/full", "title": "Full", "status": 409, "seats_available": 2}'
    err = parse_error(make(409, "application/problem+json; charset=utf-8", body))
    assert err.is_problem
    assert err.type == "https://x.test/full"
    assert err.extensions == {"seats_available": 2}


def test_members_of_the_wrong_type_are_ignored():
    err = parse_error(make(400, "application/problem+json", '{"type": 7, "title": ["x"], "status": "400", "detail": null}'))
    assert (err.type, err.title, err.detail, err.status) == ("about:blank", "Bad Request", None, 400)


def test_the_http_status_wins_over_the_body():
    err = parse_error(make(404, "application/problem+json", '{"title": "Gone fishing", "status": 200}'))
    assert err.status == 404


def test_gateway_html_is_not_a_problem():
    err = parse_error(make(502, "text/html", "<html>Bad gateway</html>"))
    assert (err.is_problem, err.status, err.title) == (False, 502, "Bad Gateway")


def test_a_problem_media_type_with_a_broken_body_is_not_a_problem():
    err = parse_error(make(500, "application/problem+json", "not json"))
    assert (err.is_problem, err.status) == (False, 500)


def test_a_json_array_is_not_a_problem():
    assert not parse_error(make(400, "application/problem+json", "[1, 2]")).is_problem


def test_retry_after_is_read_as_seconds():
    assert parse_error(make(429, "text/plain", "slow down", {"Retry-After": "30"})).retry_after == 30
    assert parse_error(make(429, "text/plain", "slow down", {"Retry-After": "Wed, 21 Oct 2026 07:28:00 GMT"})).retry_after is None

python -m pytest -q

..................................... [100%]
37 passed in 0.97s

All thirty-seven tests pass: thirty for the server and seven for the client. The client tests build responses by hand: a wrong-typed member, a body that contradicts the HTTP status, gateway HTML, a problem media type with a broken body, a JSON array, and both forms of Retry-After.

A client that branches on the type

Here is the payoff. This script reserves too many seats, reads the number that the server sent, asks again for the seats that are left, then keeps reserving until it hits the rate limit, and finally triggers the crash. Stop the server with Ctrl+C and start it again (or restart it with uvicorn app:app --port 8765) so that the in-memory data is fresh.

# step_client.py - a client that branches on the problem type, not on the text
import requests

from client import ApiError, call

BASE = "http://127.0.0.1:8765"
ALICE = {"Authorization": "Bearer tok-alice"}
CAROL = {"Authorization": "Bearer tok-carol"}


def reserve(session, seats, headers):
    body = {"attendee_email": "[email protected]", "seats": seats}
    return call(requests.request, "POST", f"{BASE}/sessions/{session}/reservations", json=body, headers=headers)


try:
    reserve("s2", 4, ALICE)
except ApiError as err:
    print("refused:", err)
    if err.type.endswith("/session-full"):
        left = err.extensions["seats_available"]
        print(f"asking for the {left} seats that are left instead")
        print(reserve("s2", left, ALICE))

for attempt in range(1, 8):
    try:
        left = reserve("s1", 1, CAROL)["seats_available"]
        print(f"attempt {attempt}: reserved, {left} seats left")
    except ApiError as err:
        if err.type.endswith("/rate-limited"):
            print(f"attempt {attempt}: rate limited, retry after {err.retry_after} seconds")
            break
        raise

try:
    call(requests.request, "GET", f"{BASE}/sessions/s1/export", headers=ALICE)
except ApiError as err:
    print("crash:", err)
    print("instance to quote to support:", err.instance)

python step_client.py

refused: 409 Session is full (https://api.example.com/problems/session-full): Only 2 seats are left in this session.
asking for the 2 seats that are left instead
{'session': 's2', 'seats': 2, 'seats_available': 0}
attempt 1: reserved, 89 seats left
attempt 2: reserved, 88 seats left
attempt 3: reserved, 87 seats left
attempt 4: reserved, 86 seats left
attempt 5: reserved, 85 seats left
attempt 6: rate limited, retry after 60 seconds
crash: 500 Internal Server Error (about:blank): The server hit an unexpected error. Quote the instance value when you contact support.
instance to quote to support: urn:uuid:68d01f39-7051-43a4-ac8b-555b655cbc96

The code never matches an English sentence. It tests err.type.endswith("/session-full"), reads seats_available from err.extensions, and uses err.retry_after as a number. On the crash it prints the instance it would quote to support. If the server reworded every message tomorrow, this client would not notice.

Step 11: Check the whole service and know where the contract still leaks

End to end with curl

Restart the server so that the data is fresh, and check three responses with curl -si, which prints the status line and headers too. A date: header is omitted from the output here.

curl -si -X POST http://127.0.0.1:8765/sessions/s2/reservations -H "Authorization: Bearer tok-alice" -H "Content-Type: application/json" -d '{"attendee_email":"[email protected]","seats":4}'

HTTP/1.1 409 Conflict
server: uvicorn
content-length: 244
content-type: application/problem+json

{"type":"https://api.example.com/problems/session-full","title":"Session is full","status":409,"detail":"Only 2 seats are left in this session.","instance":"urn:uuid:af0ac232-d121-4bf9-b581-77b84ba535c3","seats_available":2,"seats_requested":4}

curl -si http://127.0.0.1:8765/sessions/s1/export -H "Authorization: Bearer tok-alice"

HTTP/1.1 500 Internal Server Error
server: uvicorn
content-length: 224
content-type: application/problem+json

{"type":"about:blank","title":"Internal Server Error","status":500,"detail":"The server hit an unexpected error. Quote the instance value when you contact support.","instance":"urn:uuid:3454b8aa-d11c-4552-af41-fb9f6c1c90b5"}

curl -si -X DELETE http://127.0.0.1:8765/sessions/s1

HTTP/1.1 405 Method Not Allowed
server: uvicorn
allow: GET
content-length: 123
content-type: application/problem+json

{"type":"about:blank","title":"Method Not Allowed","status":405,"instance":"urn:uuid:9ef6ad1d-d6d0-4c61-a3ef-8837795bcae4"}

Three layers, one format: a typed domain error with an extension, a crash with nothing but an id, and a router error that kept its allow header.

Responses your handlers cannot control

Exception handlers shape the responses that your application produces. A response that a middleware or a proxy writes by itself escapes them. Starlette 1.9.0 ships a request body limit middleware, and it makes a good specimen. This script adds a 1,024-byte limit and sends a 5 KB body twice, once with a declared Content-Length and once as a chunked upload without one:

# step_edges.py - a response that never passes through the exception handlers
import logging

from fastapi.testclient import TestClient
from starlette.middleware.body_limit import RequestBodyLimitMiddleware

from app import create_app
from client import parse_error
from matrix import ALICE

logging.getLogger().handlers.clear()  # keep this script's output free of server log lines
ran = []


class Tap(logging.Handler):
    def emit(self, record):
        ran.append(record.getMessage())


logging.getLogger("problems").setLevel(logging.INFO)
logging.getLogger("problems").addHandler(Tap())

app = create_app()
app.add_middleware(RequestBodyLimitMiddleware, max_body_size=1024)
client = TestClient(app, raise_server_exceptions=False)


def chunks():
    yield b'{"attendee_email": "[email protected]", "seats": 1, "padding": "'
    yield b"x" * 5000
    yield b'"}'


def post(label, **kwargs):
    before = len(ran)
    resp = client.post("/sessions/s1/reservations", **kwargs)
    err = parse_error(resp)
    print(f"{label}: HTTP {resp.status_code} {resp.headers['content-type']} is_problem={err.is_problem} handler_ran={len(ran) > before}")
    print("   ", resp.text[:110])


big = {"attendee_email": "[email protected]", "seats": 1, "padding": "x" * 5000}
post("declared length", headers=ALICE, json=big)
post("chunked, no length", headers={**ALICE, "Content-Type": "application/json"}, content=chunks())

python step_edges.py

declared length: HTTP 413 text/plain; charset=utf-8 is_problem=False handler_ran=True
    Content Too Large
chunked, no length: HTTP 413 application/problem+json is_problem=True handler_ran=True
    {"type":"about:blank","title":"Content Too Large","status":413,"instance":"urn:uuid:eabd89b1-14bc-4565-83d0-b8

The two answers differ. With a declared length the middleware answers on its own in plain text, and its source shows that it replaces whatever response the application starts to send when the declared length is over the limit. Our handler did run (handler_ran=True) but its answer was discarded. When the body is chunked and has no declared length, the middleware raises an HTTPException that our handler converts into a proper problem document.

The same thing happens with anything that answers before your application does, such as a reverse proxy, a load balancer or a gateway, which typically speak HTML or plain text. You cannot fix that from FastAPI, so you design for it: the client from Step 10 reports is_problem=False with the right status and a title taken from the HTTP phrase, and carries on. If you control the proxy, configure its error pages to return application/problem+json too.

Common mistakes and how to catch them

• Registering the handler for FastAPI’s HTTPException. Routing errors keep the old shape. Catch it with the route_missing and wrong_method cases (Step 9, bug 1).
• Dropping exc.headers. The body looks right and the protocol is broken. Catch it with the header rules in the contract (bug 2).
• Echoing validation input or exception text. Use a canary string in a test, as hunter2 does here (bugs 3 and 4).
• A status that disagrees with the HTTP status, or a title that changes per occurrence. Both defeat the clients and dashboards that group problems by type (bugs 5 and 6).
• Relative type URIs. They resolve differently depending on the request URL, and the contract rejects them.
• Testing a crash with the default test client. The test fails with the exception instead of checking the response (Step 6).
• Letting the OpenAPI document describe the old shapes. Generated clients and fuzzers then disagree with the server (Step 8).
• Debug mode in production. It bypasses the 500 handler, as the Starlette documentation says. Call the crash route on your staging server and check what comes back.

Confirm everything works

With the server running, the final probe should report no broken rules, python -m pytest -q should report 37 passed, and python mutations.py should report no bug as undetected:

python probe.py

case             status content-type               contract
ok               200    application/json           not an error
session_missing  404    application/problem+json   ok
route_missing    404    application/problem+json   ok
wrong_method     405    application/problem+json   ok
no_token         401    application/problem+json   ok
bad_token        401    application/problem+json   ok
bad_body         422    application/problem+json   ok
session_full     409    application/problem+json   ok
rate_limited     429    application/problem+json   ok
server_error     500    application/problem+json   ok

Where to go next

• Add problem types as your API grows. A stale version in the ETag and If-Match tutorial is a natural version-conflict type, and an invalid cursor in the keyset pagination tutorial could be an invalid-cursor type with the offending parameter named.
• Give the body size limit from the secure file uploads tutorial a type of its own, and decide how your clients should treat the plain text 413 that the middleware sends.
• Let a tool check your documentation against reality. Schemathesis compares real responses with the schemas your OpenAPI document declares, so the documented helper from Step 8 pays off there.
• Pair this with the access control checks in the IDOR tutorial and the response filtering in the mass assignment tutorial, and put a 503 problem with Retry-After in front of an overloaded service as in the load shedding tutorial.
• Read the sources: RFC 9457 is short, and its section on defining new problem types is worth reading before you mint your own.