How to Find Bugs Your Tests Miss in Python With Property-Based Testing and Hypothesis

What you will build, and why example tests are not enough

Many unit tests are examples. You pick an input, work out the answer by hand, and assert that the code returns it. That works until the bug lives in an input you did not think of: an interval that sits inside another one, a bill that does not divide evenly, a username written with a look-alike letter. In this tutorial you will test your code a different way. Instead of writing down inputs, you write down a rule that must hold for every input, and a library invents hundreds or thousands of inputs to try to break it.

That approach is called property-based testing, and the Python library for it is Hypothesis. You will use Hypothesis 6.169.0 (published on PyPI on October 11, 2026, and it needs Python 3.11 or newer) with pytest 9.1.1 on Python 3.13.14 and Windows 11. The Hypothesis documentation is careful about what the method is for: “Property-based testing is a powerful addition to unit testing. It is not always a replacement.”

You will write five small pieces of code that each hide a bug or a wrong assumption, and watch Hypothesis find every one of them: a function that merges overlapping intervals and loses part of one, a bill splitter that loses cents, a query-string round trip that drops empty fields, a username normalizer that gives a different answer the second time you run it, and an LRU cache that evicts the wrong entry after one particular sequence of calls. Along the way you will learn how to read a failure report, why the input you get back is so small, and how to pin a failure so that it cannot return. Then you will set up the settings that make the tests useful in continuous integration (CI), meet a health check, and measure how many inputs Hypothesis needed to find each bug. That measurement surprised me more than anything else in this tutorial. Every output below came from running the code in this article.

Six terms to know before we start

Property. A rule about your code that should hold for every valid input, such as “the output of sorted() is in order”.

Strategy. A recipe that tells Hypothesis how to build inputs. st.integers() builds whole numbers, and st.lists(st.integers()) builds lists of them.

Shrinking. When Hypothesis finds an input that breaks a property, it tries simpler variations of that input that still break it, and reports the simplest one it can reach. You get a two-item list instead of a list of forty items.

Example database. A folder where Hypothesis remembers failing inputs, so the next run tries them first.

Oracle. A slow but obviously correct way to compute the expected answer. A property can compare your fast code against an oracle instead of against a hand-picked answer.

Idempotent. An operation is idempotent if doing it twice gives the same result as doing it once, like pressing the lift button on a floor you are already on.

Prerequisites

• Python 3.11 or newer. Hypothesis 6.169.0 declares requires_python >=3.11. Everything here ran on Python 3.13.14.
• The basics of pytest: files named test_*.py, functions named test_*, and plain assert statements. You do not need any Hypothesis experience.
• A terminal and an empty folder. The code needs no network, database or web framework; it uses the standard library plus Hypothesis and pytest.
• About an hour. Two scripts take a noticeable moment to run: step05_standard.py (about ten seconds on my machine) and step08_measure.py (about a minute).

Create the project

Make a folder, create a virtual environment inside it, activate it, and install the two packages:

mkdir properties
cd properties
python -m venv .venv
.venv\Scripts\activate
python -m pip install hypothesis==6.169.0 pytest==9.1.1

On macOS and Linux the activation line is source .venv/bin/activate. Every file below goes in the properties folder and every command is run from there. Save the next script as step00_versions.py and run it with python step00_versions.py to confirm what you installed.

"""step00_versions.py: print the versions this tutorial was written with."""
import sys

import hypothesis
import pytest

print("python", sys.version.split()[0])
print("hypothesis", hypothesis.__version__)
print("pytest", pytest.__version__)

Expected output on the machine used for this article:

python 3.13.14
hypothesis 6.169.0
pytest 9.1.1

Step 1: Write a property and see what Hypothesis generates

Before hunting bugs, get comfortable with the two parts every Hypothesis test has: a strategy that makes inputs, and the @given decorator that feeds them to your test function.

1.1 Look at the inputs Hypothesis makes

Save this as step01_peek.py and run it with python step01_peek.py.

"""step01_peek.py: print what Hypothesis generates for a list of integers."""
from hypothesis import given, settings, strategies as st


@settings(max_examples=8, database=None, derandomize=True)
@given(st.lists(st.integers()))
def show(xs):
    print(xs)


show()

With the pinned versions you should see these eight lists. Other versions of Hypothesis print other numbers.

[]
[-30247374765868732185896774795264, -2950663462827152375808]
[5758, -2618033, -3618, 100000000, -24825549425694830940389376, -357, 31314819]
[-65059516776490, 2030280945791670, 22147, -155707, -593462790, -51802680, -661582178]
[-4657550]
[14553, 572, 5227, -43]
[-118575813, -1248289, 1478, -1494539304546, 16263]
[-321828, 58784157834, -234, 564, -366704]

Here is what the code does. st.lists(st.integers()) is a strategy that means “a list of whole numbers”. @given(...) turns show into a Hypothesis test: when you call show() without arguments, Hypothesis calls it for you many times and passes a generated list as xs. max_examples=8 limits the run to eight calls (the default is a hundred; the quickstart says “By default, Hypothesis generates 100 random inputs.”), database=None keeps Hypothesis from saving anything to disk, and derandomize=True makes the run deterministic, so that you see the same lists as printed here.

Now look at the lists, because this is the first lesson. The first list is empty. The others have between one and seven items, and the numbers range from small to enormous (the first non-empty list contains a number with more than thirty digits). You chose none of this. A person writing examples by hand rarely thinks of an empty list or of a thirty-digit number, and Hypothesis tried both in the first two inputs of this run.

1.2 Test sorted() with two properties

A property is a rule about the output that holds for every valid input. For sorted() two rules come to mind: the result is in order, and it contains exactly the items the input contained. Save this as test_sorted.py.

# test_sorted.py
from collections import Counter

from hypothesis import given, strategies as st


@given(st.lists(st.integers()))
def test_sorted_output_is_in_order(xs):
    result = sorted(xs)
    assert all(a <= b for a, b in zip(result, result[1:]))


@given(st.lists(st.integers()))
def test_sorted_output_has_the_same_items_as_the_input(xs):
    assert Counter(sorted(xs)) == Counter(xs)

Run it with pytest:

python -m pytest -q test_sorted.py

.. [100%]
2 passed in 0.16s

Each test ran 100 times with different lists, and none of them broke the rule. Look at what each property leaves out. A function that always returns an empty list satisfies the first rule, and a function that returns its input unchanged satisfies the second. Only the two together pin down sorting. Choosing properties that are strong enough is an important skill in this style of testing, and Step 2 shows what happens when a property is too weak.

Check before moving on: you should see 2 passed. If you see ModuleNotFoundError: No module named 'hypothesis', your virtual environment is not active.

Step 2: Find a bug in merge_intervals

Calendars, booking systems and range lists often need to merge overlapping ranges. A closed interval (start, end) includes both of its ends, so (1, 4) and (4, 5) touch at 4 and merge into (1, 5). This step writes the function the way many people would, tests it the usual way, and then lets Hypothesis in.

2.1 The function and the tests you would normally write

Save the first version as intervals_v1.py. The _v1 in the name is deliberate: this version has a bug, and you will keep it next to the fixed version so that you can watch both.

"""intervals_v1.py: merge overlapping closed intervals (first version)."""


def merge_intervals(intervals):
    """Merge overlapping closed intervals: [(1, 3), (2, 6)] becomes [(1, 6)]."""
    merged = []
    for start, end in sorted(intervals):
        if merged and start <= merged[-1][1]:
            merged[-1] = (merged[-1][0], end)
        else:
            merged.append((start, end))
    return merged

The function sorts the intervals by start. Then it walks through them: if the next interval starts at or before the end of the last merged one, the two overlap and are combined; otherwise the next interval starts a new group. Now write the tests you would normally write, five examples that cover overlapping, separate, touching, unsorted and empty input. Save them as test_intervals_examples.py.

# test_intervals_examples.py
from intervals_v1 import merge_intervals


def test_overlapping_pair_merges():
    assert merge_intervals([(1, 3), (2, 6)]) == [(1, 6)]


def test_separate_intervals_stay_separate():
    assert merge_intervals([(1, 2), (5, 6)]) == [(1, 2), (5, 6)]


def test_touching_intervals_merge():
    assert merge_intervals([(1, 4), (4, 5)]) == [(1, 5)]


def test_unsorted_input_is_handled():
    assert merge_intervals([(8, 10), (1, 3), (2, 4)]) == [(1, 4), (8, 10)]


def test_empty_input():
    assert merge_intervals([]) == []

python -m pytest -q test_intervals_examples.py

..... [100%]
5 passed in 0.01s

All five pass, and the code looks finished. Keep that feeling in mind.

2.2 A property that is too weak

Now describe what a correct merge must do for every input. First you need a strategy that builds interval lists. Save this as interval_strategies.py; every later test in this tutorial imports it.

"""interval_strategies.py: how Hypothesis should build interval lists for the tests."""
from hypothesis import strategies as st

# A closed interval (start, end) with start <= end. Small numbers make overlaps likely.
interval = st.tuples(st.integers(-20, 20), st.integers(-20, 20)).map(
    lambda pair: (min(pair), max(pair))
)

intervals = st.lists(interval, max_size=8)

The interval strategy draws two whole numbers between -20 and 20 and sorts the pair so that the start is never larger than the end; .map(...) is how you transform what a strategy produces. The intervals strategy builds lists of up to eight of them. Two choices matter here. First, the numbers are small on purpose. With only 41 possible values, intervals often share an endpoint or touch, such as one that ends at 4 next to one that starts at 4, and shared endpoints are where interval code often breaks. With numbers up to a billion, two intervals would almost never share an endpoint. Second, short lists keep failure reports easy to read.

A typical first property is “the output is sorted and the merged intervals are separated from each other”. Save it as test_intervals_weak.py.

# test_intervals_weak.py
from hypothesis import given

from interval_strategies import intervals
from intervals_v1 import merge_intervals


@given(intervals)
def test_output_is_sorted_and_separated(xs):
    out = merge_intervals(xs)
    for (_, end), (start, _) in zip(out, out[1:]):
        assert end < start

python -m pytest -q test_intervals_weak.py

. [100%]
1 passed in 0.14s

It passes, so the function is fine? It is not, and this is the trap. The property says nothing about which numbers the merged intervals cover, so a function that quietly shortens an interval satisfies it. A property that wrong code also satisfies tells you very little, no matter how many inputs Hypothesis tries.

2.3 A property with an oracle

A stronger rule is “merging must not change which whole numbers are covered”. To check it you need an oracle, a slow but obviously correct way to answer the question. Here the oracle is a set that lists every whole number inside any interval. Save this as test_intervals_coverage_v1.py.

# test_intervals_coverage_v1.py
from hypothesis import given

from interval_strategies import intervals
from intervals_v1 import merge_intervals


def covered(xs):
    """Every whole number inside at least one closed interval."""
    return {n for start, end in xs for n in range(start, end + 1)}


@given(intervals)
def test_merging_keeps_exactly_the_covered_points(xs):
    assert covered(merge_intervals(xs)) == covered(xs)

python -m pytest -q test_intervals_coverage_v1.py

F [100%]
=== FAILURES ===
___ test_merging_keeps_exactly_the_covered_points ___
test_intervals_coverage_v1.py:14: in test_merging_keeps_exactly_the_covered_points
    def test_merging_keeps_exactly_the_covered_points(xs):
                   ^^^
test_intervals_coverage_v1.py:15: in test_merging_keeps_exactly_the_covered_points
    assert covered(merge_intervals(xs)) == covered(xs)
E   assert {-2, -1} == {-2, -1, 0}
E     
E     Extra items in the right set:
E     0
E     Use -v to get more diff
E   Failing test case: test_merging_keeps_exactly_the_covered_points(
E       xs=[(-2, 0), (-1, -1)],
E   )
=== short test summary info ===
FAILED test_intervals_coverage_v1.py::test_merging_keeps_exactly_the_covered_points - assert {-2, -1} == {-2, -1, 0}
1 failed in 0.27s

Read the report from the bottom up. Failing test case is the input that broke the property: the list [(-2, 0), (-1, -1)]. The assertion above it shows the two sets, {-2, -1} from the merged result and {-2, -1, 0} from the original intervals, and pytest points out the extra item: the number 0 has disappeared.

Walk through it by hand. Sorted, the input is (-2, 0) and then (-1, -1). The second interval starts inside the first, so the code merges them, but it replaces the end of the merged interval with the end of the second interval, -1, instead of keeping the larger of the two ends, 0. The result (-2, -1) no longer covers 0.

The input in the report is tiny, and you did not shrink it by hand. Hypothesis did. The next script, step02_unshrunk.py, runs the same search twice with the same random seed (@seed(5)). The first search skips the shrink phase through settings(phases=...), so it reports the first failing input as found. The second search includes the shrink phase. Python 3.11 lets a library attach notes to an exception, and Hypothesis puts the failing input in a note, so the script can print it.

"""step02_unshrunk.py: the first failing input Hypothesis finds, with and without the shrink phase."""
from hypothesis import Phase, given, seed, settings

from interval_strategies import intervals
from intervals_v1 import merge_intervals

SEED = 5


def covered(xs):
    return {n for start, end in xs for n in range(start, end + 1)}


def first_failure(phases):
    @seed(SEED)
    @settings(phases=phases, database=None)
    @given(intervals)
    def search(xs):
        assert covered(merge_intervals(xs)) == covered(xs)

    try:
        search()
    except AssertionError as error:
        return "\n".join(error.__notes__)


print("without the shrink phase:")
print(first_failure([Phase.generate]))
print("with the shrink phase:")
print(first_failure([Phase.generate, Phase.shrink]))

without the shrink phase:
Failing test case: search(
    xs=[(-18, 2), (-8, 4), (14, 20), (-7, 14), (-9, -7), (-6, -6)],
)
with the shrink phase:
Failing test case: search(
    xs=[(0, 0), (-1, 1)],
)

The raw failing input had six intervals with numbers up to 20 in size, and it is not obvious which of them matter. Shrinking reduced it to two intervals with numbers no larger than 1 in size. That is why Hypothesis reports are readable: the bug is the same, but the noise around it is gone. Different runs can end at different small inputs: the script above ended at [(0, 0), (-1, 1)] while the pytest run before it ended at [(-2, 0), (-1, -1)], because shrinking stops at the first input it cannot simplify further. Both have the same shape: one interval nested inside another.

2.4 Fix it, then keep the test

The fix is to keep the larger of the two ends. Save the fixed function as intervals.py, with no _v1 in its name.

"""intervals.py: merge overlapping closed intervals (fixed)."""


def merge_intervals(intervals):
    """Merge overlapping closed intervals: [(1, 3), (2, 6)] becomes [(1, 6)]."""
    merged = []
    for start, end in sorted(intervals):
        if merged and start <= merged[-1][1]:
            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
        else:
            merged.append((start, end))
    return merged

Now write the final test file, test_intervals.py, with four properties of a correct merge: the coverage oracle, “sorted and separated” (so the weak property is still there, as a second line of defence), “merging twice changes nothing” (merging is idempotent) and “the order of the input does not matter”.

# test_intervals.py
from hypothesis import event, example, given, strategies as st

from interval_strategies import intervals
from intervals import merge_intervals


def covered(xs):
    """Every whole number inside at least one closed interval."""
    return {n for start, end in xs for n in range(start, end + 1)}


@given(intervals)
@example([(-2, 0), (-1, -1)])  # the input Hypothesis reported for the first version
def test_merging_keeps_exactly_the_covered_points(xs):
    out = merge_intervals(xs)
    event("something merged" if len(out) < len(xs) else "nothing to merge")
    assert covered(out) == covered(xs)


@given(intervals)
def test_output_is_sorted_and_separated(xs):
    out = merge_intervals(xs)
    for (_, end), (start, _) in zip(out, out[1:]):
        assert end < start


@given(intervals)
def test_merging_twice_changes_nothing(xs):
    once = merge_intervals(xs)
    assert merge_intervals(once) == once


@given(intervals, st.randoms(use_true_random=False))
def test_input_order_does_not_matter(xs, rnd):
    shuffled = list(xs)
    rnd.shuffle(shuffled)
    assert merge_intervals(shuffled) == merge_intervals(xs)

Four things in this file are new. @example([(-2, 0), (-1, -1)]) pins the failing input from the report. The Hypothesis documentation says “@example adds an explicit input which Hypothesis will run every time, in addition to the randomly generated test cases.” so this case is now tested on every run, whatever the random search does. event(...) labels each input with a short text, which lets you check that your strategy really produces the situations you care about. st.randoms(use_true_random=False) gives the test a random-number generator that Hypothesis controls; the documentation says its values “will actually be Hypothesis generated values (and will be shrunk accordingly for any failing test case)”. Finally, the order-independence property is a metamorphic property: it compares the function with itself on a changed input, so it needs no oracle.

Run it with the statistics option, which prints what Hypothesis did:

python -m pytest -q --hypothesis-show-statistics test_intervals.py

.... [100%]
=== Hypothesis Statistics ===
test_intervals.py::test_merging_keeps_exactly_the_covered_points:

  - during generate phase (0.07 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 100 passing, 0 failing, and 9 invalid test cases
    - Events:
      * 78.90%, something merged
      * 12.84%, nothing to merge

  - Stopped because settings.max_examples=100

The output has one such block for each of the four tests, and the last line says 4 passed. The figures change from run to run. The documentation explains where the event lines come from: “These events appear in observability output, as well as the output of our pytest plugin when run with --hypothesis-show-statistics.” Here they tell you that most generated inputs contained something that had to be merged, which is what you want. If an event said that nearly every input was “nothing to merge”, your strategy would be testing the boring case. The “invalid” count is the number of generated inputs Hypothesis discarded before running the property to the end; the page “How many times will Hypothesis run my test?” in the documentation lists the reasons.

Check before moving on: test_intervals.py passes, while test_intervals_coverage_v1.py still fails, because it tests the old file. That is the point of keeping both.

Step 3: Conservation laws, split a bill

Some bugs are about things that must add up. When a group splits a restaurant bill, the shares must sum to the total, to the cent. A rule of this kind, something that has to stay true however the code is used, is an invariant. Save the first version as billing_v1.py.

"""billing_v1.py: split a restaurant bill (first version)."""


def split_bill(total_cents, people):
    """Split a bill into one share per person, in whole cents."""
    if people < 1:
        raise ValueError("people must be at least 1")
    share = total_cents // people
    return [share] * people

The function rejects groups of fewer than one person and gives everyone total_cents // people, whole-number division that drops the remainder. Write the invariant as a property in test_billing_v1.py. This time the test passes the strategies by keyword, so you can see which one feeds which argument.

# test_billing_v1.py
from hypothesis import given, strategies as st

from billing_v1 import split_bill


@given(
    total_cents=st.integers(min_value=0, max_value=10**9),
    people=st.integers(min_value=1, max_value=50),
)
def test_shares_add_up_to_the_bill(total_cents, people):
    assert sum(split_bill(total_cents, people)) == total_cents

python -m pytest -q test_billing_v1.py

F [100%]
=== FAILURES ===
___ test_shares_add_up_to_the_bill ___
test_billing_v1.py:8: in test_shares_add_up_to_the_bill
    total_cents=st.integers(min_value=0, max_value=10**9),
               ^^^
test_billing_v1.py:12: in test_shares_add_up_to_the_bill
    assert sum(split_bill(total_cents, people)) == total_cents
E   assert 0 == 1
E    +  where 0 = sum([0, 0])
E    +    where [0, 0] = split_bill(1, 2)
E   Failing test case: test_shares_add_up_to_the_bill(
E       total_cents=1,
E       people=2,
E   )
=== short test summary info ===
FAILED test_billing_v1.py::test_shares_add_up_to_the_bill - assert 0 == 1
1 failed in 0.17s

Hypothesis found a bill of one cent split between two people: each share is 0 and the sum is 0, so one cent has vanished. A single dropped cent looks harmless in a test and costs real money across a million bills, which is why a conservation property is worth writing for anything that handles amounts. A hand-written example such as splitting 100 cents between 4 people passes, because only totals that do not divide evenly expose the bug.

The fix gives the first few people one extra cent each. divmod(total, people) returns the share everyone gets and the number of cents left over. Save the fixed version as billing.py.

"""billing.py: split a restaurant bill (fixed)."""


def split_bill(total_cents, people):
    """Split a bill into one share per person, in whole cents. The first shares absorb the leftover cents."""
    if people < 1:
        raise ValueError("people must be at least 1")
    base, extra = divmod(total_cents, people)
    return [base + 1] * extra + [base] * (people - extra)

The final test file, test_billing.py, has three properties: the sum is preserved, the group has one share per person and no share differs from another by more than one cent, and a group of nobody is rejected with a ValueError. That last property settles what the function does with invalid input, a decision you should make on purpose. Without the guard, Hypothesis would have reported a ZeroDivisionError for a group of nobody as its very first failure.

# test_billing.py
import pytest
from hypothesis import example, given, strategies as st

from billing import split_bill

bills = st.integers(min_value=0, max_value=10**9)
groups = st.integers(min_value=1, max_value=50)


@given(bills, groups)
@example(1, 2)  # the case Hypothesis found in the first version
def test_shares_add_up_to_the_bill(total_cents, people):
    assert sum(split_bill(total_cents, people)) == total_cents


@given(bills, groups)
def test_one_share_per_person_and_no_share_differs_by_more_than_a_cent(total_cents, people):
    shares = split_bill(total_cents, people)
    assert len(shares) == people
    assert max(shares) - min(shares) <= 1


@given(bills, st.integers(max_value=0))
def test_a_group_of_nobody_is_rejected(total_cents, people):
    with pytest.raises(ValueError):
        split_bill(total_cents, people)

python -m pytest -q test_billing.py

... [100%]
3 passed in 0.17s

Step 4: Round trips, and bugs in code you did not write

A round trip is a conversion followed by its inverse: encode and decode, serialize and parse. The Hypothesis documentation recommends starting here: “Look for round-trip properties: encode/decode, serialize/deserialize, etc. These property-based tests tend to be both powerful and easy to write.” Python’s urllib.parse module turns a dictionary into a query string with urlencode and back with parse_qs. Because parse_qs returns a list of values for each name, the expected result is each value wrapped in a list. Save this as test_roundtrip_v1.py.

# test_roundtrip_v1.py
from urllib.parse import parse_qs, urlencode

from hypothesis import given, strategies as st

queries = st.dictionaries(st.text(), st.text())


@given(queries)
def test_query_string_round_trip(params):
    assert parse_qs(urlencode(params)) == {k: [v] for k, v in params.items()}

python -m pytest -q test_roundtrip_v1.py

F [100%]
=== FAILURES ===
___ test_query_string_round_trip ___
test_roundtrip_v1.py:10: in test_query_string_round_trip
    def test_query_string_round_trip(params):
                   ^^^
test_roundtrip_v1.py:11: in test_query_string_round_trip
    assert parse_qs(urlencode(params)) == {k: [v] for k, v in params.items()}
E   AssertionError: assert {} == {'': ['']}
E     
E     Right contains 1 more item:
E     {'': ['']}
E     Use -v to get more diff
E   Failing test case: test_query_string_round_trip(
E       params={'': ''},
E   )
=== short test summary info ===
FAILED test_roundtrip_v1.py::test_query_string_round_trip - AssertionError: assert {} == {'': ['']}
1 failed in 0.78s

The failing input is a dictionary with an empty name and an empty value. urlencode writes it as =, and parse_qs reads that back as nothing at all. This is not a mistake in CPython. The documentation of parse_qs says so: “The default false value indicates that blank values are to be ignored and treated as if they were not included.” A form with one empty field, such as urlencode({'a': '', 'b': 'x'}), comes back as {'b': ['x']} with the empty field gone. The code is behaving as documented, and your test assumed a stricter contract. A property test finds that kind of mismatch between what you assume and what a library promises, in code that you did not write.

The fix is to ask for the behaviour you assumed. Pass keep_blank_values=True. Save the corrected test as test_roundtrip.py.

# test_roundtrip.py
from urllib.parse import parse_qs, urlencode

from hypothesis import given, strategies as st

queries = st.dictionaries(st.text(), st.text())


@given(queries)
def test_query_string_round_trip(params):
    decoded = parse_qs(urlencode(params), keep_blank_values=True)
    assert decoded == {k: [v] for k, v in params.items()}

python -m pytest -q test_roundtrip.py

. [100%]
1 passed in 0.77s

Note what a pass means here. Hypothesis tried 100 dictionaries of arbitrary text and found no further disagreement. The documentation of st.text() says what the strategy can produce: “The default alphabet strategy can generate the full unicode range but excludes surrogate characters because they are invalid in the UTF-8 encoding.” That range includes the characters that matter in a query string, such as & and =, so the strategy is able to test them.

Step 5: Idempotence, and a username that is two different names

Websites that let people choose a username usually want “Admin”, “ADMIN” and “admin” to be the same name. So they compute a canonical form of each name, a standard spelling used for comparisons, and refuse a registration whose canonical form already exists. Getting this wrong can be a security problem and not only an annoyance. Django’s 3.0.1 release notes (December 18, 2019) describe CVE-2019-19844, “Potential account hijack via password reset form”: “By submitting a suitably crafted email address making use of Unicode characters, that compared equal to an existing user email when lower-cased for comparison, an attacker could be sent a password reset token for the matched account.” The mechanism there is different from the one in this step, but the lesson carries over: when two code paths disagree about whether two strings are the same, an attacker can use the gap.

A canonical form should be idempotent: canonicalizing a name that is already canonical must change nothing. If it does, two parts of a system that each canonicalize the same stored value, for example a signup service and a login service, can end up with different keys for the same person.

5.1 A canonical form that looks right

Two standard-library tools do the work. str.casefold() is a stronger kind of lowercasing; the Python documentation says “Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string.” unicodedata.normalize("NFKC", ...) replaces characters with their standard compatibility equivalents, so the mathematical bold letter U+1D400 becomes a plain capital A and the service mark sign U+2120 becomes the two letters SM. A natural first version lowercases first and normalizes second. Save it as usernames_v1.py.

"""usernames_v1.py: the form of a username used to decide whether two names are the same (first version)."""
import unicodedata


def canonical_username(name):
    """Lowercase the name for comparison, then fold look-alike characters together."""
    return unicodedata.normalize("NFKC", name.casefold())

The property is the definition above in code: canonicalize once, canonicalize the result, and expect no change. Save it as test_usernames_v1.py. One line is new. @seed(0) fixes the random seed. Hypothesis normally picks a new seed on every run, and the rest of this step explains why this test needs a pinned seed to behave the same way for you as it did for me.

# test_usernames_v1.py
from hypothesis import given, seed, strategies as st

from usernames_v1 import canonical_username


@seed(0)  # a fixed seed, so that you see the same unlucky run as this article
@given(st.text())
def test_canonical_form_is_stable(name):
    once = canonical_username(name)
    assert canonical_username(once) == once

python -m pytest -q test_usernames_v1.py

. [100%]
1 passed in 0.73s

It passes. The function is nevertheless wrong, and what happens next is worth slowing down for.

5.2 Ask for more inputs

Write the same test again with a bigger budget. @settings(max_examples=3000) asks Hypothesis for 3,000 inputs instead of 100. Save this as test_usernames_v1_deep.py.

# test_usernames_v1_deep.py
from hypothesis import given, settings, strategies as st

from usernames_v1 import canonical_username


@settings(max_examples=3000)
@given(st.text())
def test_canonical_form_is_stable(name):
    once = canonical_username(name)
    assert canonical_username(once) == once

python -m pytest -q test_usernames_v1_deep.py

F [100%]
=== FAILURES ===
___ test_canonical_form_is_stable ___
test_usernames_v1_deep.py:8: in test_canonical_form_is_stable
    @given(st.text())
                   ^^^
test_usernames_v1_deep.py:11: in test_canonical_form_is_stable
    assert canonical_username(once) == once
E   AssertionError: assert 'a' == 'A'
E     
E     - A
E     + a
E   Failing test case: test_canonical_form_is_stable(
E       name='𝐀',
E   )
=== short test summary info ===
FAILED test_usernames_v1_deep.py::test_canonical_form_is_stable - AssertionError: assert 'a' == 'A'
1 failed in 0.88s

Now it fails. Your report may show a different character than mine, because shrinking can stop at different small inputs. In my runs Hypothesis reported U+1D400 (mathematical bold capital A, shown above), U+1D468 (mathematical bold italic capital A) and U+2120 (the service mark sign). All three have the same story. casefold() has no case mapping for them, so it leaves them alone. NFKC then converts them into ordinary capital letters. The first pass therefore returns a name with a capital letter in it (A, or SM for the service mark), and the second pass lowercases it (a or sm). The canonical form was not stable, because normalizing ran after the case folding and created capital letters that nothing folded again. The Unicode Standard describes the reason in its definition of caseless matching: “Compatibility caseless matching requires an extra cycle of case folding and normalization for each string compared, because the NFKD normalization of a compatibility character such as U+3392 SQUARE MHZ may result in a sequence of alphabetic characters which must again be case folded (and normalized) to be compared correctly.”

5.3 Fix the order, then measure how rare the bug was

The simplest repair is to normalize first and case-fold last, so that nothing runs after the case folding. Save it as usernames.py.

"""usernames.py: the form of a username used to decide whether two names are the same (fixed)."""
import unicodedata


def canonical_username(name):
    """Fold look-alike characters together first, then lowercase for comparison."""
    return unicodedata.normalize("NFKC", name).casefold()

This is a simple recipe and not a complete one, as the Unicode quotation above hints, and 5.4 checks it against the standard. First, answer the question the passing test in 5.1 raised: how rare is the bug? This script tries every single Unicode code point (all but the surrogates, which cannot be encoded) on both versions and counts the ones whose canonical form is not stable. It needs both usernames_v1.py and usernames.py. Save it as step05_scan.py and run it with python step05_scan.py.

"""step05_scan.py: try every single code point on both versions of canonical_username."""
import sys
import unicodedata

import usernames
import usernames_v1


def unstable_code_points(canonical):
    found = []
    for code in range(sys.maxunicode + 1):
        if 0xD800 <= code <= 0xDFFF:  # surrogates cannot be encoded, skip them
            continue
        once = canonical(chr(code))
        if canonical(once) != once:
            found.append(code)
    return found


print("Unicode data in this Python:", unicodedata.unidata_version)
for label, module in (("casefold, then NFKC", usernames_v1), ("NFKC, then casefold", usernames)):
    bad = unstable_code_points(module.canonical_username)
    print(f"{label}: {len(bad)} unstable code points out of 1,112,064")
    for code in bad[:3]:
        print(f"    U+{code:04X} {unicodedata.name(chr(code), '?')}")

print()
fake = usernames_v1.canonical_username(chr(0x1D468) + "dmin")
print("first pass of the fake name:", fake)
print("second pass of the fake name:", usernames_v1.canonical_username(fake))
print("first pass of the real name:", usernames_v1.canonical_username("Admin"))

Unicode data in this Python: 15.1.0
casefold, then NFKC: 637 unstable code points out of 1,112,064
    U+037A GREEK YPOGEGRAMMENI
    U+03D2 GREEK UPSILON WITH HOOK SYMBOL
    U+03D3 GREEK UPSILON WITH ACUTE AND HOOK SYMBOL
NFKC, then casefold: 0 unstable code points out of 1,112,064

first pass of the fake name: Admin
second pass of the fake name: admin
first pass of the real name: admin

Of 1,112,064 code points, 637 break the first version and none breaks the fixed one (the data in Python 3.13.14 is Unicode 15.1.0, so other Python versions can give other counts). My arithmetic: 637 of 1,112,064 is about 0.06 percent. A search that draws 100 strings is unlikely to include one of those characters, which is why the 100-example test in 5.1 passed.

The last three lines show why this matters. The fake name is U+1D468 followed by dmin. With the first version it canonicalizes to Admin, with a capital A, which differs from the canonical form of the real name, admin, so a uniqueness check on the canonical form would accept the registration. Canonicalize the stored Admin one more time, as any second service that normalizes its input would, and you get admin: the fake account now collides with the real one. This is a sketch of how the gap could be used, not a report of an attack, and a real system has other checks. It shows why idempotence earns a property.

5.4 Keep the test and check the fix against the standard

Save the final test as test_usernames.py. It keeps the 3,000-input budget, because Step 8 shows that 100 is not enough, and it pins the look-alike admin name with @example.

# test_usernames.py
from hypothesis import example, given, settings, strategies as st

from usernames import canonical_username

# U+1D468 MATHEMATICAL BOLD ITALIC CAPITAL A, one of the characters that broke the first version.
MATH_A = chr(0x1D468)


@settings(max_examples=3000)
@given(st.text())
@example(MATH_A + "dmin")  # the look-alike admin name from the scan
def test_canonical_form_is_stable(name):
    once = canonical_username(name)
    assert canonical_username(once) == once

python -m pytest -q test_usernames.py

. [100%]
1 passed in 1.39s

Passing one property is not the same as being correct, so one more check is worth making. The Unicode Standard defines compatibility caseless matching with a longer recipe: NFKD(toCasefold(NFKD(toCasefold(NFD(X))))), called D146 in chapter 3 of the standard. The script below implements that recipe as standard_form and asks three questions: does canonical_username ever move a code point into a different standard class, does it group the code points exactly as the standard does, and does it keep the standard class of 3,000 random strings? Save it as step05_standard.py and run it.

"""step05_standard.py: does canonical_username agree with the Unicode Standard's definition of caseless matching?"""
import sys
import unicodedata

from hypothesis import given, settings, strategies as st

from usernames import canonical_username


def standard_form(text):
    """Unicode Standard D146: NFKD(toCasefold(NFKD(toCasefold(NFD(X)))))."""
    step = unicodedata.normalize("NFD", text).casefold()
    step = unicodedata.normalize("NFKD", step).casefold()
    return unicodedata.normalize("NFKD", step)


print("Unicode data in this Python:", unicodedata.unidata_version)
code_points = [n for n in range(sys.maxunicode + 1) if not 0xD800 <= n <= 0xDFFF]

moved = [n for n in code_points
         if standard_form(canonical_username(chr(n))) != standard_form(chr(n))]
print("code points that canonical_username moves to another standard class:", len(moved))

ours, standard = {}, {}
for n in code_points:
    ours.setdefault(canonical_username(chr(n)), set()).add(n)
    standard.setdefault(standard_form(chr(n)), set()).add(n)
same = {frozenset(v) for v in ours.values()} == {frozenset(v) for v in standard.values()}
print(f"groups of equal names: ours {len(ours)}, standard {len(standard)}, identical grouping: {same}")


@settings(max_examples=3000, database=None)
@given(st.text())
def never_changes_the_class(name):
    assert standard_form(canonical_username(name)) == standard_form(name)


never_changes_the_class()
print("3000 random strings: canonical_username never changed the standard class")

Unicode data in this Python: 15.1.0
code points that canonical_username moves to another standard class: 0
groups of equal names: ours 1106760, standard 1106760, identical grouping: True
3000 random strings: canonical_username never changed the standard class

For every single code point, the simple recipe groups names exactly as the standard does, and it held for 3,000 random strings. That is evidence and not a proof: the check covers single code points and random text, not every possible string, and it depends on the Unicode version in your Python. If your product must follow the standard to the letter, use the standard’s recipe instead and keep this property test as a guard.

Step 6: Test a whole object with a state machine, an LRU cache

So far you tested functions that take an input and return an output. Objects have a history. A cache can be right after any single call and wrong after one particular sequence of calls, and example tests find such sequences only if you think of them. Hypothesis can generate sequences of operations too. Its documentation says “A RuleBasedStateMachine gives you a structured way to define state machines.” You describe the operations (rules), the facts that must always hold (invariants) and a simple model of what the object should contain, and Hypothesis tries random sequences of rules.

The object under test is an LRU (least recently used) cache. It keeps at most capacity entries and, when it is full, drops the entry that was used longest ago. If you want the full story of how such a cache works, the sxz.io tutorial on building an LRU cache from scratch in Python covers it. Here you only need the behaviour. Save the first version as lru_v1.py.

"""lru_v1.py: a least-recently-used cache (first version)."""
from collections import OrderedDict


class LRUCache:
    """Keeps at most `capacity` items and drops the one that was used longest ago."""

    def __init__(self, capacity):
        if capacity < 1:
            raise ValueError("capacity must be at least 1")
        self.capacity = capacity
        self._data = OrderedDict()

    def get(self, key, default=None):
        if key not in self._data:
            return default
        self._data.move_to_end(key)
        return self._data[key]

    def put(self, key, value):
        if key in self._data:
            self._data[key] = value
            return
        if len(self._data) >= self.capacity:
            self._data.popitem(last=False)
        self._data[key] = value

    def __len__(self):
        return len(self._data)

get moves the key to the end of the ordered dictionary, which marks it as recently used. put evicts the oldest entry when a new key arrives and the cache is full. Look closely at the first branch of put, and see whether you can spot the problem before the test does.

Now write the state machine. Save it as test_lru_v1.py.

# test_lru_v1.py
from hypothesis import seed, strategies as st
from hypothesis.stateful import RuleBasedStateMachine, invariant, rule

from lru_v1 import LRUCache

CAPACITY = 2
keys = st.integers(0, 3)


@seed(2)  # pinned so that your report matches the one in the article
class LRUMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.cache = LRUCache(CAPACITY)
        self.order = []  # keys from least to most recently used
        self.values = {}  # key -> value

    @rule(key=keys, value=st.integers())
    def put(self, key, value):
        self.cache.put(key, value)
        if key in self.order:
            self.order.remove(key)
        self.order.append(key)
        self.values[key] = value
        if len(self.order) > CAPACITY:
            oldest = self.order.pop(0)
            del self.values[oldest]

    @rule(key=keys)
    def get(self, key):
        assert self.cache.get(key, "missing") == self.values.get(key, "missing")
        if key in self.order:
            self.order.remove(key)
            self.order.append(key)

    @invariant()
    def never_over_capacity(self):
        assert len(self.cache) <= CAPACITY


TestLRU = LRUMachine.TestCase

The machine holds the cache under test and a model: a list of keys ordered from least to most recently used, and a dictionary of values. The model is simple and built differently from the cache (a list and a dictionary instead of an OrderedDict), so it is unlikely to share the cache’s mistakes. The put and get rules call the real cache, update the model by the plain rules of LRU, and check that get agrees with the model. The @invariant() method runs after every step. The last line, TestLRU = LRUMachine.TestCase, is what lets pytest find the machine. The capacity is only 2 and the keys come from 0 to 3, so the cache fills up and evicts constantly. The @seed(2) line before the class pins the random seed, as in Step 5, so that your report matches the one printed below; remove it when you reuse the machine for your own code.

python -m pytest -q test_lru_v1.py

F [100%]
=== FAILURES ===
___ TestLRU.runTest ___
test_lru_v1.py:32: in get
    assert self.cache.get(key, "missing") == self.values.get(key, "missing")
E   AssertionError: assert 'missing' == 0
E    +  where 'missing' = get(0, 'missing')
E    +    where get = <lru_v1.LRUCache object at 0x00000287B083F7D0>.get
E    +      where <lru_v1.LRUCache object at 0x00000287B083F7D0> = LRUMachine({}).cache
E    +  and   0 = <built-in method get of dict object at 0x00000287B0AD6F80>(0, 'missing')
E    +    where <built-in method get of dict object at 0x00000287B0AD6F80> = {0: 0, 2: 0}.get
E    +      where {0: 0, 2: 0} = LRUMachine({}).values
E   Failing test case:
E   state = LRUMachine()
E   state.never_over_capacity()
E   state.put(key=0, value=0)
E   state.never_over_capacity()
E   state.put(key=1, value=0)
E   state.never_over_capacity()
E   state.put(key=0, value=0)
E   state.never_over_capacity()
E   state.put(key=2, value=0)
E   state.never_over_capacity()
E   state.get(key=0)
E   state.teardown()
=== short test summary info ===
FAILED test_lru_v1.py::TestLRU::runTest - AssertionError: assert 'missing' == 0
1 failed in 1.54s

I removed the lines that point inside Hypothesis itself, together with their source lines, to keep the report readable. The sequence at the bottom is the shrunk failing run. The state.never_over_capacity() lines between the steps are the invariant being checked after each one, and state.teardown() is cleanup. Walk through the five real steps with a capacity of 2. After put(0) and put(1) the cache holds 0 and then 1, so 0 is the oldest. The second put(0) updates the value, but the first branch of put returns without marking the key as recently used, so the cache still thinks 0 is the oldest, while the model has moved it to the end. put(2) then evicts 0 from the cache, where the model evicts 1. The final get(0) returns 'missing' where the model expected 0.

The report has five steps even though Hypothesis can run up to fifty per test (the documented default is stateful_step_count = 50), because shrinking dropped the steps that were not needed to reproduce the failure. To find this by hand you would have had to think of writing the same key a second time while another key sits between the two writes.

The fix is to treat an update of an existing key as a use. Save the corrected cache as lru.py.

"""lru.py: a least-recently-used cache (fixed)."""
from collections import OrderedDict


class LRUCache:
    """Keeps at most `capacity` items and drops the one that was used longest ago."""

    def __init__(self, capacity):
        if capacity < 1:
            raise ValueError("capacity must be at least 1")
        self.capacity = capacity
        self._data = OrderedDict()

    def get(self, key, default=None):
        if key not in self._data:
            return default
        self._data.move_to_end(key)
        return self._data[key]

    def put(self, key, value):
        if key in self._data:
            self._data.move_to_end(key)
        elif len(self._data) >= self.capacity:
            self._data.popitem(last=False)
        self._data[key] = value

    def __len__(self):
        return len(self._data)

The test for it is the same machine pointed at the fixed module. Save it as test_lru.py by copying test_lru_v1.py and changing the import line to this:

# test_lru.py (copy test_lru_v1.py and change the import)
from lru import LRUCache

python -m pytest -q test_lru.py

. [100%]
1 passed in 0.55s

Step 7: Make it part of your workflow

A property test that you run once is a demo. This step covers what turns it into a habit: pinning failures, understanding the example database, setting up a CI profile, and reading a health check.

7.1 Pin every failure you find

You already did this in the final test files: @example([(-2, 0), (-1, -1)]) in Step 2, @example(1, 2) in Step 3 and the look-alike name in Step 5. The reason is in the documentation, which says that Hypothesis may invalidate its saved failures when you upgrade it, and that changes to a test’s source “may also change its database key, invalidating its stored entries”. It concludes: “We therefore recommend against relying on the database for the correctness of your tests.” Pin the already-shrunk input from the report, because explicit examples do not shrink: Hypothesis runs them as written.

7.2 The example database

The database is still useful for quick work on your own machine. The documentation says “When a test fails, Hypothesis automatically saves the failure so it can be replayed later.” The settings documentation says that if you configure nothing, “a DirectoryBasedExampleDatabase is created in the current working directory under .hypothesis/examples”. Add .hypothesis/ to your .gitignore: it is a cache and not part of your code. The next script runs the failing property from Step 2 three times against a temporary database and records which generated input was the first to fail. Save it as step07_database.py and run it.

"""step07_database.py: with the example database on, a failing input is replayed first."""
import tempfile

from hypothesis import given, settings
from hypothesis.database import DirectoryBasedExampleDatabase

from interval_strategies import intervals
from intervals_v1 import merge_intervals


def covered(xs):
    return {n for start, end in xs for n in range(start, end + 1)}


state = {"calls": 0, "first_failure": None}

with tempfile.TemporaryDirectory() as folder:
    database = DirectoryBasedExampleDatabase(folder)

    @settings(database=database)
    @given(intervals)
    def search(xs):
        state["calls"] += 1
        try:
            assert covered(merge_intervals(xs)) == covered(xs)
        except AssertionError:
            if state["first_failure"] is None:
                state["first_failure"] = state["calls"]
            raise

    for run in (1, 2, 3):
        state.update(calls=0, first_failure=None)
        try:
            search()
        except AssertionError:
            pass
        print(f"run {run}: first failing input was number {state['first_failure']}, "
              f"{state['calls']} inputs in total including shrinking")

run 1: first failing input was number 17, 87 inputs in total including shrinking
run 2: first failing input was number 1, 2 inputs in total including shrinking
run 3: first failing input was number 1, 2 inputs in total including shrinking

On the first run, the first failing input was number 17 (yours will differ), and the search used 87 inputs in total, including the ones spent on shrinking. On runs 2 and 3 the very first input was the saved failure, and the whole search took two inputs. That is the difference between waiting for the search to rediscover a bug and getting it back immediately while you fix it.

7.3 Settings profiles for your laptop and for CI

A settings profile is a named bundle of settings. Hypothesis ships two: default (100 inputs, random, with the database on) and ci. The documentation says: “If the CI environment variable is set to any value, the ci profile is active by default.” It adds that Hypothesis detects various vendor-specific CI variables too. The built-in ci profile makes runs deterministic (derandomize=True), turns off the example database and the per-example deadline (200 milliseconds in the default profile), prints a reproduction blob when a test fails, and suppresses the too_slow health check.

You can keep that profile and change one number, which is what the documentation shows with its own “run more examples in CI” example. Save this as conftest.py in the project folder; pytest loads it before your tests.

# conftest.py
from hypothesis import settings

# Hypothesis already switches to its built-in "ci" profile when the CI environment variable is set.
# Keep that profile and run more examples in it.
settings.register_profile("ci", settings.get_profile("ci"), max_examples=1000)

# A bigger, random budget for a nightly run: python -m pytest --hypothesis-profile=thorough
settings.register_profile("thorough", settings.get_profile("default"), max_examples=5000)

The ci line keeps everything in the built-in profile and raises the budget to 1,000 inputs. The thorough profile is a bigger budget for a nightly run. It names the default profile as its parent on purpose, so that it stays random even on a machine where the CI variable is set. See the difference on one test, with --hypothesis-show-statistics and -k add_up to pick the bill test from Step 3:

python -m pytest -q --hypothesis-show-statistics test_billing.py -k add_up

. [100%]
=== Hypothesis Statistics ===
test_billing.py::test_shares_add_up_to_the_bill:

  - during generate phase (0.03 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 100 passing, 0 failing, and 0 invalid test cases

  - Stopped because settings.max_examples=100


1 passed, 2 deselected in 0.10s

To run the same test as CI would, set the CI variable. In bash that is CI=true python -m pytest .... In PowerShell run $env:CI = "true" first and Remove-Item Env:CI afterwards. The nightly profile is a command-line option:

CI=true python -m pytest -q --hypothesis-show-statistics test_billing.py -k add_up
python -m pytest -q --hypothesis-show-statistics test_billing.py -k add_up --hypothesis-profile=thorough

. [100%]
=== Hypothesis Statistics ===
test_billing.py::test_shares_add_up_to_the_bill:

  - during generate phase (0.33 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 1000 passing, 0 failing, and 0 invalid test cases

  - Stopped because settings.max_examples=1000


1 passed, 2 deselected in 0.40s

. [100%]
=== Hypothesis Statistics ===
test_billing.py::test_shares_add_up_to_the_bill:

  - during generate phase (1.58 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 5000 passing, 0 failing, and 0 invalid test cases

  - Stopped because settings.max_examples=5000


1 passed, 2 deselected in 1.66s

The three runs used 100, 1,000 and 5,000 inputs, and the largest took about 1.6 seconds for the bill splitter. Cheap properties can afford a big budget.

Failures that come from CI

The ci profile also turns on print_blob. Run the failing bill test as CI would, and the report gains a line you have not seen yet:

E   Failing test case: test_shares_add_up_to_the_bill(
E       total_cents=1,
E       people=2,
E   )
E   
E   You can reproduce this test case by temporarily adding @reproduce_failure('6.169.0', b'AEEBQQI=') as a decorator on your test function

The @reproduce_failure decorator replays the exact failure on your machine, which the documentation calls “useful for locally replaying failures found by CI”. It comes with a warning: “the binary blob is not stable across Hypothesis versions, so you should not leave this decorator on your tests permanently. Use @example with an explicit input instead.” That is the reason this tutorial pins readable examples and not blobs. For a random failure on your own machine, a seed does the job; the documentation says “If using pytest, you can alternatively pass --hypothesis-seed on the command line.”

Why not derandomize everywhere?

A deterministic run tests the same inputs every time, which keeps CI stable but stops it from looking in new places. The documentation says that setting a seed “overrides settings.derandomize, which is designed to enable deterministic CI tests rather than reproducing observed failures.” A sensible split is a derandomized run on every push and a random, bigger run on a schedule, the thorough profile above.

7.4 Health checks

Hypothesis also watches how your strategies behave. Suppose you only want user IDs that start with user_ and write the strategy with a filter. Save this as test_health_v1.py.

# test_health_v1.py
from hypothesis import given, strategies as st


@given(st.text().filter(lambda s: s.startswith("user_")))
def test_user_ids_keep_their_prefix(user_id):
    assert user_id.removeprefix("user_") == user_id[len("user_"):]

python -m pytest -q test_health_v1.py

F [100%]
=== FAILURES ===
___ test_user_ids_keep_their_prefix ___
test_health_v1.py:6: in test_user_ids_keep_their_prefix
    def test_user_ids_keep_their_prefix(user_id):
                   ^^^
E   hypothesis.errors.FailedHealthCheck: It looks like this test is filtering out a lot of inputs. 0 inputs were generated successfully, while 50 inputs were filtered out. 
E   
E   An input might be filtered out by calls to assume(), strategy.filter(...), or occasionally by Hypothesis internals.
E   
E   Applying this much filtering makes input generation slow, since Hypothesis must discard inputs which are filtered out and try generating it again. It is also possible that applying this much filtering will distort the domain and/or distribution of the test, leaving your testing less rigorous than expected.
E   
E   If you expect this many inputs to be filtered out during generation, you can disable this health check with @settings(suppress_health_check=[HealthCheck.filter_too_much]). See https://hypothesis.readthedocs.io/en/latest/reference/api.html#hypothesis.HealthCheck for details.
--- Hypothesis ---
You can reproduce this failure by adding @seed(191780702218710956336292321039213486566) to this test, or by running pytest with --hypothesis-seed=191780702218710956336292321039213486566.
=== short test summary info ===
FAILED test_health_v1.py::test_user_ids_keep_their_prefix - hypothesis.errors.FailedHealthCheck: It looks like this test is filtering out a lot of inputs. 0 inputs wer...
1 failed in 0.75s

The test never ran. Random text almost never starts with user_, so Hypothesis threw away 50 inputs without finding a single usable one and stopped. The documentation describes this check: “Check for when the test is filtering out too many test cases, either through use of assume() or .filter(), or occasionally for Hypothesis internal reasons.” It also tells you not to treat health checks as correctness errors: “Health checks are proactive warnings, not correctness errors”, and the output offers a way to switch the check off. Here the better answer is to fix the strategy: build values that satisfy the rule, instead of generating values and discarding the ones that do not. Save this as test_health.py.

# test_health.py
from hypothesis import given, strategies as st


@given(st.text().map(lambda tail: "user_" + tail))
def test_user_ids_keep_their_prefix(user_id):
    assert user_id.removeprefix("user_") == user_id[len("user_"):]

python -m pytest -q test_health.py

. [100%]
1 passed in 0.73s

Step 8: How many inputs did it take?

Every pass in this tutorial hides a question: if the bug had been there, how likely was the search to find it? Step 5 showed that the answer can be “not very”. This step measures it. The script runs 50 independent searches for each bug, with seeds 0 to 49 and the database off, and records the number of the first generated input that broke the property. It uses the first versions of the code from Steps 2, 3 and 5, the query-string round trip from Step 4, and one more property, a JSON round trip of floats, which the text below explains. Save it as step08_measure.py.

"""step08_measure.py: how many generated inputs pass before Hypothesis finds each bug in the first versions?"""
import json
import statistics
import sys
from urllib.parse import parse_qs, urlencode

from hypothesis import HealthCheck, given, seed, settings, strategies as st

from billing_v1 import split_bill
from interval_strategies import intervals
from intervals_v1 import merge_intervals
from usernames_v1 import canonical_username

RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 50


def covered(xs):
    return {n for start, end in xs for n in range(start, end + 1)}


def first_failure(strategy, holds, seed_value, budget):
    """One search with a fixed seed. Returns the number of the first input that broke `holds`, or None."""
    state = {"calls": 0, "first": None}

    @seed(seed_value)
    @settings(database=None, max_examples=budget, deadline=None, suppress_health_check=list(HealthCheck))
    @given(strategy)
    def search(x):
        state["calls"] += 1
        if not holds(x):
            if state["first"] is None:
                state["first"] = state["calls"]
            raise AssertionError

    try:
        search()
    except AssertionError:
        pass
    return state["first"]


def stable(name):
    return canonical_username(canonical_username(name)) == canonical_username(name)


CASES = [
    ("merge_intervals loses points", intervals,
     lambda xs: covered(merge_intervals(xs)) == covered(xs), (100,)),
    ("split_bill loses cents", st.tuples(st.integers(0, 10**9), st.integers(1, 50)),
     lambda t: sum(split_bill(*t)) == t[0], (100,)),
    ("query string drops blank values", st.dictionaries(st.text(), st.text()),
     lambda d: parse_qs(urlencode(d)) == {k: [v] for k, v in d.items()}, (100,)),
    ("username form is unstable (any text)", st.text(), stable, (100, 1000, 5000)),
    ("JSON round trip of floats breaks on NaN", st.floats(),
     lambda x: json.loads(json.dumps(x)) == x, (100, 1000)),
    ("username form is unstable (letters and digits)", st.text(st.characters(categories=("L", "N"))),
     stable, (100, 1000)),
]

print(f"{'bug':<48}{'budget':>7}{'found':>9}{'median':>8}{'max':>7}")
for label, strategy, holds, budgets in CASES:
    for budget in budgets:
        hits = [first_failure(strategy, holds, s, budget) for s in range(RUNS)]
        found = [h for h in hits if h is not None]
        median = f"{statistics.median(found):.0f}" if found else "-"
        worst = str(max(found)) if found else "-"
        print(f"{label:<48}{budget:>7}{len(found):>6}/{RUNS}{median:>8}{worst:>7}", flush=True)

Run it with python step08_measure.py. It took about a minute on my machine. Because the seeds are fixed, a second run printed exactly the same table. These are the numbers from my run:

What the table says

The first three bugs were found in all 50 searches, inside the default 100 inputs. The median search needed 9 inputs for the interval bug, 3 for the bill splitter and 2 for the query string, and the slowest searches needed 31, 6 and 2. These bugs live in a large part of the input space, and 100 inputs is plenty.

The username bug is different. The property that passed in Step 5 found the bug in only 10 of 50 searches at 100 inputs. At 1,000 inputs it was found in 49 of 50 searches, and at 5,000 in all 50 (the slowest search needed its 1,244th input). The median search that found it needed about 220 inputs, which suggests that roughly one input in a few hundred triggers the bug. That fits the scan in Step 5, where only about 0.06 percent of code points were unstable.

The JSON row is a second rare case, and its property is a trap of a different kind. json.loads(json.dumps(x)) == x looks like a natural round trip, but st.floats() can produce nan, which is not equal to itself, so the property fails whenever nan turns up. That happened within 100 inputs in 39 of 50 searches. The JSON specification, RFC 8259, says that such values do not belong in the format: “Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.” Python’s json.dumps writes NaN anyway, because its allow_nan option defaults to true and then “their JavaScript equivalents (NaN, Infinity, -Infinity) are used”. With allow_nan=False the same call raises a ValueError, “in strict compliance with the JSON specification”. The sensible repair is to use st.floats(allow_nan=False, allow_infinity=False) for the property and allow_nan=False in the call, so that a bad value raises an error instead of producing invalid JSON.

The last two rows are a negative result worth knowing. I guessed that a narrower alphabet (letters and digits only) would make the Unicode bug easier to hit. It did not: 12 of 50 searches found it at 100 inputs and 49 of 50 at 1,000, about the same as with unrestricted text. Spending more inputs helped, and narrowing the alphabet did not.

What to do about bugs that hide in a small corner

Three habits follow from the numbers. Spend more inputs when the test is cheap; the 5,000-input profile in 7.3 took about 1.6 seconds for the bill splitter. Run a bigger budget on a schedule with the thorough profile, which is random, so that it keeps looking in new places while the derandomized CI run stays stable. And pin every failure with @example, because a bug that takes luck to find should never need luck to find a second time. Mixing ordinary text with a hand-picked list of troublesome characters in your strategy may help as well; I did not measure that here.

Common mistakes and gotchas

A property that repeats the code

If the test computes the expected value with the same logic as the function, it passes for the same reasons the function is right or wrong. The coverage oracle in Step 2 works because it answers the question by a different, slower route (a set of points), and the LRU model in Step 6 works because a list and a dictionary are built differently from an OrderedDict.

A property that is too weak

Step 2 showed a property that the buggy function satisfied. Ask of each property: could a wrong implementation pass this? If you can name one, strengthen the property or add another.

Filtering instead of building

Step 7.4 showed the health check that catches .filter() or assume() discarding almost everything. Use .map() or a composed strategy to build valid inputs directly. Check the event statistics from Step 2 to make sure the interesting cases appear.

No decision about invalid input

Step 3 included a property for the group of nobody. Decide what your function does with inputs outside its contract, write that down as a property, and keep the main properties inside the contract.

Hidden state and flaky tests

Hypothesis expects the test to behave the same way when it replays an input. The documentation on flaky failures says “Hypothesis raises an exception when it detects flakiness”, and lists common causes that include “Decisions based on global state.” and a test that depends on filesystem or database state which is not reset between inputs. Reset state inside the test, and keep randomness and clocks out of the code under test or inject them.

Treating a pass as proof

Step 5 and the table in Step 8 are the argument. A pass means that the search did not find a counterexample in its budget. For bugs in a small corner of the input space the budget matters as much as the property.

Verify it all works end to end

Here is the whole tutorial as a list of commands, with the result to expect from each. The folder also holds the first-version files that fail on purpose, so the last command names the files that should pass.

python step00_versions.py                                             # versions
python -m pytest -q test_sorted.py                                    # 2 passed
python -m pytest -q test_intervals_examples.py                        # 5 passed
python -m pytest -q test_intervals_weak.py                            # 1 passed
python -m pytest -q test_intervals_coverage_v1.py                     # 1 failed, on purpose
python step02_unshrunk.py                                             # six intervals, then two
python -m pytest -q test_billing_v1.py                                # 1 failed, on purpose
python -m pytest -q test_roundtrip_v1.py                              # 1 failed, on purpose
python -m pytest -q test_usernames_v1.py                              # 1 passed (seed 0)
python -m pytest -q test_usernames_v1_deep.py                         # 1 failed, on purpose
python step05_scan.py                                                 # 637 and 0
python step05_standard.py                                             # three agreement lines
python -m pytest -q test_lru_v1.py                                    # 1 failed, on purpose
python -m pytest -q test_health_v1.py                                 # 1 failed, on purpose (health check)
python step07_database.py                                             # run 1 slow, runs 2 and 3 instant
python -m pytest -q test_sorted.py test_intervals.py test_billing.py test_roundtrip.py test_usernames.py test_lru.py test_health.py

The last command is the suite of fixed code and final tests:

............. [100%]
13 passed in 2.46s

Run it once more as CI would, with the CI variable set. It uses the 1,000-input profile and takes longer, because each property runs 1,000 inputs instead of 100 (the username test keeps its own 3,000):

............. [100%]
13 passed in 14.80s

If a script says No module named 'intervals_v1', you are running it from a different folder than the one that holds the file. If a failure shows a different small input than the one printed here, that is normal: shrinking can end at different small inputs. If test_usernames_v1.py fails instead of passing, check that the @seed(0) line is there; without it, roughly one run in five fails, as Step 8 shows.

Where this stops working, and what to learn next

Property-based testing has limits that matter before you rely on it.

• It finds bugs and does not prove their absence. The table in Step 8 is the evidence: a bug in a small corner of the input space can survive a hundred inputs. Raise the budget, run a random nightly profile and pin what you find.
• Some code has no easy property. Code that mostly calls a network or draws pixels offers few rules to state. Properties work best on parsers, converters, data structures, arithmetic on money or time, and anything with an inverse or a simple model.
• Each property runs many times, so keep it fast. The default profile lists a deadline of 200 milliseconds per test case, and the built-in ci profile switches that deadline off. Slow properties make slow suites.
• Stateful tests are harder to read. The model and the rules are real code that you must keep correct. Start with a small model.
• Reports can differ between runs. Shrinking stops at a local minimum, so the same bug can appear as different small inputs.

To keep learning, read the Hypothesis quickstart, the tutorial and the API reference. Other tutorials on this site pair well with this one. The same idea in another language is in the Rust proptest tutorial. If you are replacing old code, an old implementation can serve as your oracle, as in the differential testing tutorial, and the characterization tests tutorial covers locking down behaviour before you refactor. A property such as “the resolved path always stays inside the base folder” would fit the code in the path traversal tutorial. And if code comes from an AI assistant, properties are a fast way to question it; see how to review AI-generated Python code.

Sources and method

The Hypothesis quotations come from the 6.169.0 documentation: the quickstart, the introduction, settings, replaying failed tests and flaky failures pages, the API reference, the strategies reference and How many times will Hypothesis run my test?. The release date and Python requirement come from the PyPI page. The other sources are the Python documentation for str.casefold, urllib.parse.parse_qs and json.dumps, chapter 3 of the Unicode Standard (definition D146), RFC 8259 and the Django 3.0.1 release notes.

Everything ran on Python 3.13.14, Windows 11, Hypothesis 6.169.0 and pytest 9.1.1, and the outputs above were captured from those runs. The measurement script uses seeds 0 to 49 with the database off. The Unicode scans cover every code point except surrogates, using the Unicode 15.1.0 data that ships with Python 3.13. The claims about what a pass proves are limited to the inputs and budgets stated in each step.