How to Find and Fix a Memory Leak in a Python Service With tracemalloc

What you will build, and why memory leaks matter

A Python service that behaves on the day you deploy it and then slowly eats memory over the following days is one of the most common production problems there is. The symptoms are familiar: the operating system kills the process, a container is restarted by its memory limit, or a graph climbs like a staircase and never comes back down. Restarting the service hides the problem for a while. Finding the cause takes a method, and the method in this tutorial uses only the standard library.

You will start by running a small report service that leaks memory on purpose. You will measure the growth, switch on the standard library’s tracemalloc module, and compare two snapshots of memory to find the exact lines of code that produce the growth. Then comes the step most guides skip: working out which object is still holding that memory, because the holder, not the line that allocated the memory, is what you have to change. You will fix three different leaks (a dictionary that never forgets, a cache that keeps every object it has ever seen, and exceptions that remember all their variables), measure what it costs to leave the tool switched on in a live service, learn two situations where tracemalloc and the operating system disagree about how much memory you use, and finish with a pytest check that fails when a leak comes back.

You do not need any profiling experience. Every step shows the exact code, the output that was captured when it was run, and a plain explanation of what the output means and why the step matters.

Six terms to know before we start

Memory leak. Memory the program no longer needs but still holds on to, so the total only ever grows. In Python this is almost never memory that has been lost. CPython normally frees an object when the last reference to it goes away, so a leak is an object that something still refers to by mistake.

Reference and holder. A reference is any variable, list slot, dictionary entry, attribute or cache entry that points at an object. The holder is whatever keeps that pointer alive. To fix a leak you remove the holder or put a limit on it.

Memory block and snapshot. Every object lives in one or more blocks of memory. A snapshot is the record tracemalloc makes, at one moment, of every block that is still allocated together with the line of code that allocated it. Subtracting an older snapshot from a newer one shows what was added in between.

Resident memory (RSS). The physical memory the operating system says the process occupies. It is the number that monitoring graphs and container limits use. The psutil documentation, the package this tutorial uses to read it, says: “rss: aka RSS. On UNIX matches the top RES column. On Windows maps to WorkingSetSize.”

Cache. A store of results kept so the program does not have to compute them again. A cache is only safe when its keys repeat (otherwise it never gets a hit) and when its size has a limit.

Bounded. Having a maximum size. Most leak fixes come down to making something bounded or making it forget.

Prerequisites

• Python 3.10 or newer. Everything here ran on Python 3.13.14 on Windows 11. tracemalloc is part of the standard library, so the main technique needs nothing extra.
• Two packages: psutil to read the process’s resident memory (version 7.2.2 was used) and pytest for the test in Step 10 (version 9.1.1).
• Comfort with functions, classes, dictionaries and lists. You do not need to know anything about memory management or profilers.
• A terminal and an empty folder. Every file below goes in the same folder and every command is run from there. All numbers in this article were measured on Windows 11, so on Linux or macOS expect different figures, and note that the Step 9 script was only run on Windows.

Create the project

mkdir leakhunt
cd leakhunt
python -m venv .venv
.venv\Scripts\activate
python -m pip install psutil==7.2.2 pytest==9.1.1

On macOS and Linux the activation line is source .venv/bin/activate.

Step 1: Build a leaky service and watch its memory climb

You cannot learn to find a leak without one to find, so this step builds a service with three leaks inside it and proves that it leaks before any tool is involved.

1.1 Check your versions

Save this small script and run it with python step00_versions.py. It prints the Python version and the package versions so you can compare them with the ones used here.

# step00_versions.py - print the versions this lab ran on
import platform

import psutil
import pytest

print(f"Python {platform.python_version()} on {platform.system()} {platform.release()}")
print(f"psutil {psutil.__version__}, pytest {pytest.__version__}")

Expected output on the machine used for this article:

Python 3.13.14 on Windows 11
psutil 7.2.2, pytest 9.1.1

1.2 Create the service

The service is one function, handle_request, that imitates a web request. For each request id it does five things. It makes a Renderer for one of fifty customers. It renders a report page, a string of about 22 KB, using a theme (a dictionary of one hundred colors). It stores the page in RESULT_CACHE under the customer and request id. It parses an upload, which uses a 40,000-byte buffer and fails for every fifth request. And it remembers every failure in RECENT_ERRORS so someone can look at them later.

Three of those decisions are leaks, and the comments in the file say which. Real leaks are never labeled, so once you have read the comments, try to forget them: the purpose of the tutorial is to find them with tools. Save the file as service_v1.py and do not fix anything yet.

# service_v1.py - a small report service with three planted memory leaks
import functools

RESULT_CACHE = {}  # leak 1: one entry per request, never removed
RECENT_ERRORS = []  # leak 3: failed requests are kept "for debugging"


def build_rows(request_id, count):
    return ",".join(f"row-{request_id:05d}-{n:04d}" for n in range(count))


class Renderer:
    def __init__(self, customer):
        self.customer = customer

    @functools.lru_cache(maxsize=None)  # leak 2: the cache key includes self
    def theme(self):
        return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}

    def render(self, request_id):
        theme = self.theme()
        header = f"<h1>Report for {self.customer}, {len(theme)} colors</h1>"
        body = build_rows(request_id, 1500)
        return header + body


def parse_upload(request_id):
    buffer = bytearray(40_000)
    if request_id % 5 == 0:
        raise ValueError(f"upload {request_id} is corrupt")
    return len(buffer)


def handle_request(request_id):
    customer = f"customer-{request_id % 50}"
    renderer = Renderer(customer)
    page = renderer.render(request_id)
    RESULT_CACHE[(customer, request_id)] = page
    try:
        parse_upload(request_id)
    except ValueError as exc:
        RECENT_ERRORS.append(exc)
    return len(page)

1.3 Measure the symptom

The first question is only whether memory grows. This script sends the service five batches of one thousand requests and prints the resident memory after each batch. It takes the module name as an argument so you can reuse it on the fixed versions later.

# symptom.py - drive a service module with batches of requests and print the process memory after each batch
import importlib
import os
import sys

import psutil

service = importlib.import_module(sys.argv[1])
process = psutil.Process(os.getpid())


def resident_mib():
    return process.memory_info().rss / 2**20


print(f"{'start':>14}: {resident_mib():7.1f} MiB resident")
sent = 0
for _ in range(5):
    for request_id in range(sent, sent + 1000):
        service.handle_request(request_id)
    sent += 1000
    print(f"{sent:>7} requests: {resident_mib():7.1f} MiB resident")

python symptom.py service_v1

Expected output (your numbers will differ a little):

         start:    20.3 MiB resident
   1000 requests:    66.2 MiB resident
   2000 requests:   111.3 MiB resident
   3000 requests:   156.3 MiB resident
   4000 requests:   201.4 MiB resident
   5000 requests:   246.5 MiB resident

Each batch of one thousand requests adds about 45 MiB and nothing ever comes back. At a modest one hundred requests per second that would be roughly 4.5 MiB every second. This is the shape of a leak: a line that rises in step with the amount of work done, not with time or with the number of customers. It also shows the limit of the measurement. Resident memory tells you that memory is leaking. It cannot tell you what is leaking or where in the code it comes from. That is the job of the next step.

Step 2: Turn on tracemalloc and compare two snapshots

The Python documentation describes the module like this: “The tracemalloc module is a debug tool to trace memory blocks allocated by Python.” Among the things it provides, the documentation lists “Compute the differences between two snapshots to detect memory leaks”. That is exactly the technique you will use: take a snapshot, do some work, take another snapshot, and ask which lines of code are responsible for the difference.

2.1 Write a small helper module

Three small functions are shared by the later scripts, so put them in one file. take_snapshot wraps tracemalloc.take_snapshot(). top_growth calls Snapshot.compare_to, which the documentation describes as “Compute the differences with an old snapshot.” and which returns a list of StatisticDiff objects with this ordering: “The result is sorted from the biggest to the smallest by: absolute value of StatisticDiff.size_diff”. Each statistic has a size_diff (how many bytes were added), a count_diff (how many blocks were added) and a traceback (where they were allocated). The helper then drops three kinds of noise: allocations made by tracemalloc itself, by linecache (the module the helper uses to read source lines), and by Python’s import system, and it hides anything that grew by less than 20 KiB. fmt turns one statistic into a line of text.

Notice that the helper filters the results of the comparison instead of calling Snapshot.filter_traces on the snapshot first. Step 8 measures why: on a large snapshot, filtering the traces was by far the slowest part.

# snaps.py - helpers shared by the demos: skip the tools' own allocations, and format one line per statistic
import linecache
import os
import tracemalloc

NOISE = {
    tracemalloc.__file__,
    linecache.__file__,
    "<frozen importlib._bootstrap>",
    "<frozen importlib._bootstrap_external>",
    "<unknown>",
}


def take_snapshot():
    return tracemalloc.take_snapshot()


def top_growth(after, before, key_type="lineno", limit=6, min_kib=20):
    """The statistics that grew most, without allocations made by tracemalloc, linecache or the import system,
    and without anything that grew by less than min_kib."""
    stats = after.compare_to(before, key_type)
    quiet = (s for s in stats if s.size_diff >= min_kib * 1024 and not any(f.filename in NOISE for f in s.traceback))
    return list(quiet)[:limit]


def where(frame):
    return f"{os.path.basename(frame.filename)}:{frame.lineno}"


def source(frame):
    return linecache.getline(frame.filename, frame.lineno).strip()


def fmt(stat):
    frame = stat.traceback[0]
    return f"{where(frame):<20}{stat.size_diff / 1024:>10,.1f} KiB{stat.count_diff:>+10,d} blocks   {source(frame)}"

2.2 Compare two snapshots

The next script starts tracing, sends 400 warm-up requests, takes a baseline snapshot, sends 1,000 more requests and takes a second snapshot. Two details matter. First, tracing is started before the service runs, because the documentation says the module “should be started as early as possible”: blocks allocated before tracemalloc.start() are not traced and will never appear in a diff. Second, the warm-up lets caches that are meant to fill up do so before the baseline, so that they are not mistaken for leaks. Step 10 turns this idea into a test.

# diff_report.py - compare two tracemalloc snapshots taken 1,000 requests apart
import importlib
import sys
import tracemalloc

from snaps import fmt, take_snapshot, top_growth

service = importlib.import_module(sys.argv[1])

tracemalloc.start()
for request_id in range(400):  # warm-up: let caches that are meant to fill up do so before the baseline
    service.handle_request(request_id)
before = take_snapshot()

for request_id in range(400, 1400):
    service.handle_request(request_id)
after = take_snapshot()

growing = top_growth(after, before)
if growing:
    print(f"{'where':<20}{'growth':>14}{'blocks':>17}   code")
    for stat in growing:
        print(fmt(stat))
else:
    print("no line grew by 20 KiB or more")
current, peak = tracemalloc.get_traced_memory()
print(f"traced now: {current / 2**20:.1f} MiB, peak: {peak / 2**20:.1f} MiB")

python diff_report.py service_v1

where                       growth           blocks   code
service_v1.py:24      22,053.5 KiB    +1,000 blocks   return header + body
service_v1.py:18      13,865.2 KiB  +202,000 blocks   return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
service_v1.py:28       7,823.6 KiB      +400 blocks   buffer = bytearray(40_000)
service_v1.py:38         108.7 KiB    +1,000 blocks   RESULT_CACHE[(customer, request_id)] = page
service_v1.py:21         100.9 KiB    +1,000 blocks   theme = self.theme()
service_v1.py:30          90.3 KiB    +1,000 blocks   raise ValueError(f"upload {request_id} is corrupt")
traced now: 60.8 MiB, peak: 60.8 MiB

2.3 Read the table

Each row is one line of code, and the three biggest rows tell the whole story:

• Line 24, return header + body: 22,053.5 KiB and 1,000 new blocks. One block per request, each about 22 KB: the report page. One thousand requests kept one thousand pages.
• Line 18, the dictionary comprehension: 13,865.2 KiB and 202,000 new blocks, which is 202 per request. That fits one hundred key strings and one hundred value strings plus the dictionary and its table of entries, and every one of them is still alive.
• Line 28, buffer = bytearray(40_000): 7,823.6 KiB and 400 blocks. Only the failing requests keep their buffer (200 of the 1,000), and each bytearray accounts for two blocks, the object and its data.

The three small rows below them, around 100 KiB each, are bookkeeping that travels with the big three: the tuple keys of the two caches and the exception objects with their tracebacks. These small rows vary by a few KiB and a few blocks from run to run; the big three do not move. The last line, traced now, is the total tracemalloc sees: 60.8 MiB.

One habit makes these tables easy to read: compare the blocks column with the number of requests. A line that gains almost exactly one block per request (or one per failing request) is a line where one object per request is being kept. Here that rule picks out all three leaks at once.

2.4 Gotchas

• No warm-up. If you take the baseline too early, a cache that is still filling up will look like the biggest leak in the program.
• Tracing started late. Anything allocated before start() is invisible. If the leak happens during import or start-up, start tracing from the command line instead (Step 8 shows how).
• A table with no big rows. That is a result too. It means nothing in the traced Python code grew by 20 KiB or more in that window, and Step 9 explains where else memory can go.

Step 3: Follow the call chain and ask one object where it came from

By default tracemalloc remembers only the line that allocated each block. In real programs that line is often a helper that is called from many places, such as a function that builds a string or parses JSON, and the diff would blame the helper rather than the feature that misuses it. The fix is to store more frames per block.

3.1 Group the growth by call chain

Passing a number to tracemalloc.start() stores that many frames. The documentation notes: “Storing more frames increases the memory and CPU overhead of the tracemalloc module.” You will measure that cost in Step 8, so for now just use ten frames and compare by "traceback" instead of by "lineno".

# traceback_view.py - group the growth by call chain instead of by line
import importlib
import sys
import tracemalloc

from snaps import source, take_snapshot, top_growth, where

service = importlib.import_module(sys.argv[1])

tracemalloc.start(10)  # keep up to 10 frames for every allocated block
for request_id in range(400):
    service.handle_request(request_id)
before = take_snapshot()

for request_id in range(400, 1400):
    service.handle_request(request_id)
after = take_snapshot()

top = top_growth(after, before, "traceback", 1)[0]
print(f"{top.size_diff / 1024:,.1f} KiB in {top.count_diff:,} blocks, allocated by this call chain (oldest call first):")
for frame in top.traceback:
    print(f"  {where(frame):<24}{source(frame)}")

python traceback_view.py service_v1

22,053.5 KiB in 1,000 blocks, allocated by this call chain (oldest call first):
  traceback_view.py:16    service.handle_request(request_id)
  service_v1.py:37        page = renderer.render(request_id)
  service_v1.py:24        return header + body

The chain is printed oldest call first. The first row is the loop in the script, the second is the call in handle_request, and the last row is the line that actually built the string. In your own service that top row would be your web framework’s request handler and the chain would show you which endpoint is responsible.

3.2 Ask a single object where it was allocated

Sometimes you already suspect an object and want to know where it came from. tracemalloc.get_object_traceback(obj) answers that. The documentation describes it: “Get the traceback where the Python object obj was allocated. Return a Traceback instance, or None if the tracemalloc module is not tracing memory allocations or did not trace the allocation of the object.” This script grabs one page out of the cache and asks.

# object_origin.py - ask a single object where it was allocated
import tracemalloc

import service_v1 as service
from snaps import source, where

tracemalloc.start(10)
for request_id in range(50):
    service.handle_request(request_id)

key, page = next(iter(service.RESULT_CACHE.items()))
print(f"cache entry {key} holds a {len(page):,}-character string, allocated here (oldest call first):")
for frame in tracemalloc.get_object_traceback(page):
    print(f"  {where(frame):<24}{source(frame)}")

python object_origin.py

cache entry ('customer-0', 0) holds a 22,541-character string, allocated here (oldest call first):
  object_origin.py:9      service.handle_request(request_id)
  service_v1.py:37        page = renderer.render(request_id)
  service_v1.py:24        return header + body

The cache entry for the first request holds a page that was built by the same return header + body line the diff blamed. That ties a suspect (the cache) to a line (24) with direct evidence. Remember the last clause of the documentation: if tracing was off when the object was created, you get None, not an answer.

Step 4: Find who is holding the memory

Here is the most important idea in the tutorial. tracemalloc tells you where memory was allocated. It does not tell you why it is still alive. Python can only free an object when nothing refers to it, so every leaked object has a holder, and the holder is the thing you must change. The allocation line is usually fine; the line that allocated the 22 KB page is doing its job. The bug is that something kept the page.

4.1 Look for containers that keep growing

Holders are almost always containers: dictionaries, lists, sets and queues. Two functions from the gc module help. gc.get_objects() “Returns a list of all objects tracked by the collector, excluding the list returned.” That covers most containers, which is what we want, but not plain strings or numbers, and there is one more limit, shown below. gc.get_referrers(obj) asks who points at an object, with a documented limit: “Return the list of objects that directly refer to any of objs. This function will only locate those containers which support garbage collection; extension types which do refer to other objects but do not support garbage collection will not be found.”

The limit is that the collector does not track every container. The documentation of gc.is_tracked says: “some type-specific optimizations can be present in order to suppress the garbage collector footprint of simple instances (e.g. dicts containing only atomic keys and values)”. This small script shows the effect:

# tracked_demo.py - the garbage collector skips some simple containers, so gc.get_objects() can miss them
import gc

print("dict of str to str       :", gc.is_tracked({"a": "b"}))
print("dict of str to list      :", gc.is_tracked({"a": []}))

cache = {}
cache["page"] = "text"
print("cache after a str entry  :", gc.is_tracked(cache))
cache["rows"] = [1, 2]
print("cache after a list entry :", gc.is_tracked(cache))

python tracked_demo.py

dict of str to str       : False
dict of str to list      : True
cache after a str entry  : False
cache after a list entry : True

A dictionary of strings to strings is invisible to gc.get_objects() until something the collector does track, such as a list, is stored in it. Many caches are plain string-to-string dictionaries, so if the scan below finds nothing, do not trust it: look at the module-level names directly (for example len(module.NAME)) and at the attributes of long-lived objects. In this service the caches have tuple keys and the scan finds them.

The script below sends 1,000 requests, records the length of every container, sends 500 more, and reports each container that gained at least 100 items. For a container that is a module-level name it prints the name; for any other it prints the types of the objects that refer to it. The scan is wrapped in a function so that the script’s own variables are not reported as holders. The last lines are two extra probes that you will use in Step 6.

# holders.py - tracemalloc says where memory was allocated; this script asks which containers keep growing
import collections
import gc
import importlib
import sys
import types
import weakref

service = importlib.import_module(sys.argv[1])
KINDS = (dict, list, set, collections.deque)


def lengths():
    return {id(obj): len(obj) for obj in gc.get_objects() if isinstance(obj, KINDS)}


def growing_containers(before, minimum=100):
    live = {id(obj): obj for obj in gc.get_objects() if isinstance(obj, KINDS)}
    rows = []
    for key in list(live):
        obj = live[key]
        grew = len(obj) - before.get(key, len(obj))
        if grew < minimum:
            continue
        names = [name for name, value in vars(service).items() if value is obj]
        holders = {type(r).__name__ for r in gc.get_referrers(obj) if r is not live and not isinstance(r, types.FrameType)}
        label = f"{service.__name__}.{names[0]}" if names else "held by " + ", ".join(sorted(holders))
        rows.append((grew, type(obj).__name__, len(obj), label))
    return sorted(rows, reverse=True)


for request_id in range(1000):
    service.handle_request(request_id)
before = lengths()
for request_id in range(1000, 1500):
    service.handle_request(request_id)

print("containers that gained 100 or more items during 500 more requests:")
for grew, kind, size, label in growing_containers(before) or [(0, "-", 0, "none")]:
    print(f"  {kind:<6}{grew:>+5} items ({size:,} now)   {label}")

theme = getattr(service.Renderer, "theme", None)
if hasattr(theme, "cache_info"):
    print("Renderer.theme cache:", theme.cache_info())

probe = service.Renderer("probe")
probe.render(0)
ref = weakref.ref(probe)
del probe
gc.collect()
print("Renderer freed after del and gc.collect()?", ref() is None)

python holders.py service_v1

containers that gained 100 or more items during 500 more requests:
  dict   +500 items (1,500 now)   service_v1.RESULT_CACHE
  dict   +500 items (1,500 now)   held by _lru_cache_wrapper
  list   +100 items (300 now)   service_v1.RECENT_ERRORS
Renderer.theme cache: CacheInfo(hits=0, misses=1500, maxsize=None, currsize=1500)
Renderer freed after del and gc.collect()? False

4.2 Match suspects to lines

There are three suspects, one for each big row in the diff, and the matching is the real analysis:

• service_v1.RESULT_CACHE gained 500 items for 500 requests: one entry per request. Step 3 already showed that its values are the 22 KB pages from line 24.
• An unnamed dictionary also gained 500 items, and the thing that refers to it is an _lru_cache_wrapper. That is the object functools.lru_cache puts around a function, so this dictionary is the cache inside an lru_cache. It has no module-level name, which is why it was easy to overlook. It matches the theme rows (line 18). The next probe shows it directly: Renderer.theme.cache_info() reports hits=0, misses=1500, maxsize=None, currsize=1500. A cache with zero hits, no size limit and a current size equal to the number of calls is a leak with a friendly name.
• service_v1.RECENT_ERRORS gained 100 items for 500 requests, one per failure. Exceptions are not 40 KB buffers, so it is not yet obvious how this list is connected to line 28. Step 7 explains it.

The final line is a weak-reference probe. A weakref.ref does not keep its target alive: the documentation explains that “when the only remaining references to a referent are weak references, garbage collection is free to destroy the referent and reuse its memory for something else.” So the script makes a Renderer, renders once, deletes the variable, runs a full collection and asks the weak reference whether the renderer is gone. The answer is False. After a del and a full collection a throwaway object should be gone, so something still holds it. You have three suspects and three leaks. The next steps fix them one at a time, because each fix is a chance to confirm with tracemalloc that the leak really went away.

Step 5: Fix leak 1, the dictionary that never forgets

Before bounding a cache, ask what the cache is for. Every request id in this service is unique, so no request will ever look up an earlier one, and the cache can never score a hit. In many real services the right fix is to delete such a cache. This tutorial keeps a bounded one because it shows the general pattern: a limit, and eviction of the oldest entry when the limit is reached.

Copy service_v1.py to service_v2.py and apply the change below. It is a unified diff: lines starting with a minus sign are removed, lines starting with a plus sign are added and the other lines are context. OrderedDict.popitem(last=False) removes the oldest entry, so the cache settles at 256 pages.

--- service_v1.py
+++ service_v2.py
@@ -1,6 +1,8 @@
-# service_v1.py - a small report service with three planted memory leaks
+# service_v2.py - leak 1 fixed: the result cache holds at most 256 pages
 import functools
+from collections import OrderedDict
 
-RESULT_CACHE = {}  # leak 1: one entry per request, never removed
+MAX_CACHED_PAGES = 256
+RESULT_CACHE = OrderedDict()
 RECENT_ERRORS = []  # leak 3: failed requests are kept "for debugging"
 
@@ -37,4 +39,6 @@
     page = renderer.render(request_id)
     RESULT_CACHE[(customer, request_id)] = page
+    if len(RESULT_CACHE) > MAX_CACHED_PAGES:
+        RESULT_CACHE.popitem(last=False)  # drop the oldest page
     try:
         parse_upload(request_id)

python diff_report.py service_v2

where                       growth           blocks   code
service_v2.py:20      13,865.2 KiB  +202,000 blocks   return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
service_v2.py:30       7,823.6 KiB      +400 blocks   buffer = bytearray(40_000)
service_v2.py:26       4,410.7 KiB      +200 blocks   return header + body
service_v2.py:23         100.9 KiB    +1,000 blocks   theme = self.theme()
service_v2.py:32          90.3 KiB    +1,000 blocks   raise ValueError(f"upload {request_id} is corrupt")
service_v2.py:38          78.1 KiB    +1,000 blocks   renderer = Renderer(customer)
traced now: 41.0 MiB, peak: 41.0 MiB

The page row (line 26) fell from 22,053.5 KiB to 4,410.7 KiB, and the total traced memory fell from 60.8 MiB to 41.0 MiB. The theme row is now the biggest. But the page row did not go to zero: 200 blocks are still growing, which is one fifth of the requests, exactly the failing ones. Something other than the cache is keeping the page for failing requests. Hold that thought until Step 7.

Step 6: Fix leak 2, the cache that holds every object

6.1 Confirm the cause

Run the holder scan on the new version. The big dictionary is still there, held by the lru_cache wrapper, and the probes say the same as before.

python holders.py service_v2

containers that gained 100 or more items during 500 more requests:
  dict   +500 items (1,500 now)   held by _lru_cache_wrapper
  list   +100 items (300 now)   service_v2.RECENT_ERRORS
Renderer.theme cache: CacheInfo(hits=0, misses=1500, maxsize=None, currsize=1500)
Renderer freed after del and gc.collect()? False

The documentation of functools.lru_cache explains what is happening in two sentences. “The cache keeps references to the arguments and return values until they age out of the cache or until the cache is cleared. If a method is cached, the self instance argument is included in the cache.” And with no limit, “the LRU feature is disabled and the cache can grow without bound”, which is what maxsize=None means. In service_v1.py the cached method is Renderer.theme(self), so the cache key is the Renderer object itself. A new Renderer is made for every request, so no key is ever repeated, no call is ever a hit, and every entry keeps its Renderer and its theme dictionary alive forever.

6.2 Cache by the thing that repeats

The intent was sensible: the theme depends only on the customer, and there are only fifty customers. The mistake was keying the cache by the instance. The fix is to move the cached function out of the class and key it by the customer name, with a size limit as a safety net. Copy service_v2.py to service_v3.py and apply this diff:

--- service_v2.py
+++ service_v3.py
@@ -1,3 +1,3 @@
-# service_v2.py - leak 1 fixed: the result cache holds at most 256 pages
+# service_v3.py - leaks 1 and 2 fixed: the theme is cached per customer, not per Renderer instance
 import functools
 from collections import OrderedDict
@@ -12,14 +12,15 @@
 
 
[email protected]_cache(maxsize=128)
+def load_theme(customer):
+    return {f"color-{n}": f"#{n:06x}-{customer}" for n in range(100)}
+
+
 class Renderer:
     def __init__(self, customer):
         self.customer = customer
 
-    @functools.lru_cache(maxsize=None)  # leak 2: the cache key includes self
-    def theme(self):
-        return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
-
     def render(self, request_id):
-        theme = self.theme()
+        theme = load_theme(self.customer)
         header = f"<h1>Report for {self.customer}, {len(theme)} colors</h1>"
         body = build_rows(request_id, 1500)

python diff_report.py service_v3

where                       growth           blocks   code
service_v3.py:31       7,823.6 KiB      +400 blocks   buffer = bytearray(40_000)
service_v3.py:27       4,410.7 KiB      +200 blocks   return header + body
service_v3.py:33          90.2 KiB      +997 blocks   raise ValueError(f"upload {request_id} is corrupt")
service_v3.py:45          54.7 KiB      +400 blocks   parse_upload(request_id)
traced now: 22.4 MiB, peak: 22.5 MiB

The theme rows are gone, and the traced total dropped again to 22.4 MiB. The cache still holds up to fifty themes, but that number is fixed by the number of customers, not by the number of requests. Run the holder scan once more:

python holders.py service_v3

containers that gained 100 or more items during 500 more requests:
  list   +100 items (300 now)   service_v3.RECENT_ERRORS
Renderer freed after del and gc.collect()? True

The cache dictionary no longer appears, and the weak-reference probe now answers True: a throwaway Renderer is freed as soon as it is deleted. Only RECENT_ERRORS is left.

Common mistake. Adding lru_cache to a method to speed it up without noticing that self becomes part of the key. If the instances are short-lived, the cache turns every one of them into a permanent resident.

Step 7: Fix leak 3, the exception that remembers its variables

Two rows survive in the Step 6 table: the 40,000-byte buffer (7,823.6 KiB) and the report page (4,410.7 KiB). Both come from failing requests, and RECENT_ERRORS is the only container still growing. That list holds exception objects, not buffers and pages. How can a list of small exceptions keep 60 KB of data per failure alive?

7.1 See what an exception keeps

Every exception carries its traceback, and a traceback links to the stack frames that were running when the error happened. A frame keeps its local variables. So an exception that you store also keeps alive the local variables of every function between the place where it was caught and the place where it was raised. The Python reference mentions this connection when it describes frame.clear(): the method “helps break reference cycles involving frame objects (for example when catching an exception and storing its traceback for later use).” This script runs five requests, takes the stored exception and walks its traceback.

# exc_probe.py - what does a stored exception keep alive?
import sys

import service_v1 as service

for request_id in range(1, 6):  # request 5 fails
    service.handle_request(request_id)

exc = service.RECENT_ERRORS[0]
print(f"stored exception: {exc!r}")
tb = exc.__traceback__
while tb is not None:
    frame = tb.tb_frame
    big = [f"{name} ({sys.getsizeof(value):,} bytes)" for name, value in frame.f_locals.items() if sys.getsizeof(value) > 1000]
    print(f"  frame {frame.f_code.co_name}: keeps alive {', '.join(big) or 'only small values'}")
    tb = tb.tb_next

python exc_probe.py

stored exception: ValueError('upload 5 is corrupt')
  frame handle_request: keeps alive page (22,582 bytes)
  frame parse_upload: keeps alive buffer (40,057 bytes)

The stored ValueError keeps the frame of parse_upload alive, and with it the 40,057-byte buffer. It also keeps the frame of handle_request, the function that caught it, alive, and with it the 22,582-byte page. That is why fixing the cache in Step 5 did not remove the page row completely: for every failed request a second holder, the traceback, still kept the page. One leak had two holders.

7.2 Compare four ways to remember a failure

You usually want to keep a record of recent failures, so the question is what to store. This script runs 100 failing requests four times, each time storing the failure differently, and reports how much traced memory is still held.

# exc_variants.py - four ways to remember a failure, and how much memory each one keeps alive
import gc
import traceback
import tracemalloc


def failing_call():
    buffer = bytearray(40_000)
    raise ValueError("corrupt upload")


def handler(remember, store):
    page = "x" * 20_000
    try:
        failing_call()
    except ValueError as exc:
        remember(store, exc)
    return len(page)


def keep_exception(store, exc):
    store.append(exc)


def keep_text(store, exc):
    store.append(f"{type(exc).__name__}: {exc}")


def keep_without_traceback(store, exc):
    store.append(exc.with_traceback(None))


def keep_after_clear_frames(store, exc):
    traceback.clear_frames(exc.__traceback__)
    store.append(exc)


VARIANTS = [
    ("store the exception", keep_exception),
    ("store the text only", keep_text),
    ("store with_traceback(None)", keep_without_traceback),
    ("clear_frames, then store", keep_after_clear_frames),
]

for label, remember in VARIANTS:
    store = []
    tracemalloc.start()
    for _ in range(100):
        handler(remember, store)
    gc.collect()
    held = tracemalloc.get_traced_memory()[0] / 1024
    tracemalloc.stop()
    print(f"{label:<28}{held:>10,.0f} KiB held after 100 failures")

python exc_variants.py

store the exception              5,934 KiB held after 100 failures
store the text only                  7 KiB held after 100 failures
store with_traceback(None)          14 KiB held after 100 failures
clear_frames, then store         2,017 KiB held after 100 failures

Reading the table from the top:

• Store the exception: 5,934 KiB for 100 failures, about 60 KiB each. This is what service_v1.py does.
• Store the text only: 7 KiB. A string holds no frames. This is the best choice for a debugging list or a log.
• Store exc.with_traceback(None): 14 KiB. You keep the exception object, which can be useful if code later inspects its type or arguments, but you drop the traceback and with it the frames.
• Call traceback.clear_frames, then store: 2,017 KiB, only a partial fix. The documentation says the function “Clears the local variables of all the stack frames in a traceback tb by calling the clear() method of each frame object.” But frame.clear() cannot clear a running function: “RuntimeError is raised if the frame is currently executing or suspended.” The standard library function swallows that error, with the comment “Ignore the exception raised if the frame is still executing.” In this test the handler that caught the exception is still running, so its page survived: the 100 pages of about 20 KiB each account for the 2,017 KiB, while the finished frame’s buffers were released.

7.3 Apply the fix

Store text, and bound the list so even text cannot grow forever. A deque with a maxlen drops the oldest entry automatically. Copy service_v3.py to service_v4.py and apply this diff:

--- service_v3.py
+++ service_v4.py
@@ -1,9 +1,9 @@
-# service_v3.py - leaks 1 and 2 fixed: the theme is cached per customer, not per Renderer instance
+# service_v4.py - all three leaks fixed
 import functools
-from collections import OrderedDict
+from collections import OrderedDict, deque
 
 MAX_CACHED_PAGES = 256
 RESULT_CACHE = OrderedDict()
-RECENT_ERRORS = []  # leak 3: failed requests are kept "for debugging"
+RECENT_ERRORS = deque(maxlen=100)  # the last 100 failures, as text
 
 
@@ -45,4 +45,4 @@
         parse_upload(request_id)
     except ValueError as exc:
-        RECENT_ERRORS.append(exc)
+        RECENT_ERRORS.append(f"{type(exc).__name__}: {exc}")
     return len(page)

After the three changes the complete file looks like this. If your copy matches, you can skip the intermediate files from here on:

# service_v4.py - all three leaks fixed
import functools
from collections import OrderedDict, deque

MAX_CACHED_PAGES = 256
RESULT_CACHE = OrderedDict()
RECENT_ERRORS = deque(maxlen=100)  # the last 100 failures, as text


def build_rows(request_id, count):
    return ",".join(f"row-{request_id:05d}-{n:04d}" for n in range(count))


@functools.lru_cache(maxsize=128)
def load_theme(customer):
    return {f"color-{n}": f"#{n:06x}-{customer}" for n in range(100)}


class Renderer:
    def __init__(self, customer):
        self.customer = customer

    def render(self, request_id):
        theme = load_theme(self.customer)
        header = f"<h1>Report for {self.customer}, {len(theme)} colors</h1>"
        body = build_rows(request_id, 1500)
        return header + body


def parse_upload(request_id):
    buffer = bytearray(40_000)
    if request_id % 5 == 0:
        raise ValueError(f"upload {request_id} is corrupt")
    return len(buffer)


def handle_request(request_id):
    customer = f"customer-{request_id % 50}"
    renderer = Renderer(customer)
    page = renderer.render(request_id)
    RESULT_CACHE[(customer, request_id)] = page
    if len(RESULT_CACHE) > MAX_CACHED_PAGES:
        RESULT_CACHE.popitem(last=False)  # drop the oldest page
    try:
        parse_upload(request_id)
    except ValueError as exc:
        RECENT_ERRORS.append(f"{type(exc).__name__}: {exc}")
    return len(page)

7.4 Verify the fix

Run the snapshot diff on the final version:

python diff_report.py service_v4

no line grew by 20 KiB or more
traced now: 6.3 MiB, peak: 6.4 MiB

No line grew by 20 KiB or more in 1,000 requests, and the traced total is 6.3 MiB instead of 60.8 MiB. Now repeat the measurement that started the tutorial, with the resident-memory script from Step 1:

python symptom.py service_v4

         start:    20.2 MiB resident
   1000 requests:    26.9 MiB resident
   2000 requests:    26.9 MiB resident
   3000 requests:    27.4 MiB resident
   4000 requests:    27.4 MiB resident
   5000 requests:    27.4 MiB resident

Resident memory rises by about 6.7 MiB during the first batch, while the bounded caches fill, and then stays flat: 26.9 MiB, then 27.4 MiB, against 246.5 MiB for the leaky version. The 0.5 MiB step between 26.9 and 27.4 MiB did not repeat in the later batches. The important comparison is that the curve is no longer proportional to the number of requests.

Step 8: Keep tracemalloc on in a running service without hurting it

Leaks that only appear in production need a tool that works in production. Two numbers decide whether that is practical: what tracing costs while the service runs, and what taking a snapshot costs.

8.1 Measure the cost of tracing

This pair of scripts serves 1,000 requests in a fresh process for each setting: tracing off, then 1, 10 and 25 frames per block. overhead_one.py does one measurement; overhead.py launches it four times and prints a table.

# overhead_one.py - serve 1,000 requests in a fresh process with tracemalloc at one setting (0 means off)
import sys
import time
import tracemalloc

import service_v1 as service

frames = int(sys.argv[1])
if frames:
    tracemalloc.start(frames)

start = time.perf_counter()
for request_id in range(1000):
    service.handle_request(request_id)
elapsed_ms = (time.perf_counter() - start) * 1000 / 1000

if frames:
    traced = tracemalloc.get_traced_memory()[0] / 2**20
    own = tracemalloc.get_tracemalloc_memory() / 2**20
    print(f"{frames:>10}{elapsed_ms:>14.3f}{traced:>14.1f}{own:>14.1f}")
else:
    print(f"{'off':>10}{elapsed_ms:>14.3f}{'-':>14}{'-':>14}")

# overhead.py - what does leaving tracemalloc on cost? One fresh process per setting.
import subprocess
import sys

print(f"{'frames':>10}{'ms/request':>14}{'traced MiB':>14}{'own MiB':>14}")
for frames in (0, 1, 10, 25):
    result = subprocess.run([sys.executable, "overhead_one.py", str(frames)], capture_output=True, text=True, check=True)
    print(result.stdout.rstrip())

python overhead.py

    frames    ms/request    traced MiB       own MiB
       off         0.309             -             -
         1         3.442          43.2          10.4
        10         8.622          43.2          10.4
        25         8.849          43.2          10.4

With tracing off the handler takes 0.309 ms per request. With one frame it takes 3.442 ms, about eleven times longer, and with ten or twenty-five frames roughly twenty-eight to twenty-nine times longer. Treat these as a worst case, not a prediction: this handler creates thousands of small strings per request, and tracemalloc pays a price for every allocation. A service that spends its time waiting for a database or a network will slow down far less, so measure your own. The docs agree with the direction: “Storing more frames increases the memory and CPU overhead of the tracemalloc module.”

The memory column shows the other cost. The module reports its own footprint through get_tracemalloc_memory(), which the documentation describes as “Get the memory usage in bytes of the tracemalloc module used to store traces of memory blocks.” Here it was 10.4 MiB for 43.2 MiB of traced memory, about a quarter, and it was the same at 1, 10 and 25 frames. That fits a program where many blocks share a few call chains, but I did not look inside the module to confirm why, and a program with many distinct call chains may behave differently.

If you need to trace from the very beginning of a process, for example a leak during start-up, set the frame count outside the code. Both of these print True 5, meaning tracing is on and five frames are stored:

python -X tracemalloc=5 -c "import tracemalloc; print(tracemalloc.is_tracing(), tracemalloc.get_traceback_limit())"
set PYTHONTRACEMALLOC=5
python -c "import tracemalloc; print(tracemalloc.is_tracing(), tracemalloc.get_traceback_limit())"

True 5

(On macOS and Linux use PYTHONTRACEMALLOC=5 python -c ... for the second form.)

8.2 Measure the cost of a snapshot

The second cost is paid each time you look. This script grows the leaky service to about 50,000, 100,000 and 200,000 live blocks and times three operations: taking a snapshot, filtering it with Snapshot.filter_traces and comparing two snapshots with top_growth.

# snapshot_cost.py - how long do snapshots take, and how much does filter_traces add?
import time
import tracemalloc

import service_v1 as service
from snaps import NOISE, take_snapshot, top_growth


def seconds(fn):
    start = time.perf_counter()
    result = fn()
    return result, time.perf_counter() - start


tracemalloc.start()
print(f"{'live blocks':>12}{'take_snapshot':>15}{'filter_traces':>15}{'compare_to':>12}")
sent = 0
for target in (250, 500, 1000):
    for request_id in range(sent, target):
        service.handle_request(request_id)
    sent = target
    first, t_take = seconds(take_snapshot)
    filters = [tracemalloc.Filter(False, name) for name in NOISE]
    _, t_filter = seconds(lambda: first.filter_traces(filters))
    for request_id in range(sent, sent + 20):
        service.handle_request(request_id)
    sent += 20
    second = take_snapshot()
    _, t_compare = seconds(lambda: top_growth(second, first))
    print(f"{len(first.traces):>12,}{t_take:>14.2f}s{t_filter:>14.2f}s{t_compare:>11.2f}s")

python snapshot_cost.py

 live blocks  take_snapshot  filter_traces  compare_to
      52,200          0.01s          1.33s       0.10s
     104,773          0.03s          3.16s       0.28s
     209,588          0.06s          6.42s       0.45s

Taking the snapshot is cheap: 0.06 seconds at 209,588 live blocks. Comparing two snapshots takes 0.45 seconds. Calling filter_traces takes 6.42 seconds, and its time grows with the number of live blocks. That is why snaps.py filters the small list of results instead of the raw traces. The documentation’s own examples use filter_traces, and for a small program that is fine; a long-running service with hundreds of thousands of live blocks is the case where it hurts.

8.3 A watcher thread

Putting those pieces together gives a small watcher. It starts tracing, and every interval seconds it compares a new snapshot with the previous one and logs the lines that grew. The log argument can be any function that takes a string, so in a real service you would pass something like logging.getLogger("memwatch").info.

# memwatch.py - log the biggest memory growth every few seconds from a background thread
import threading
import tracemalloc

from snaps import fmt, take_snapshot, top_growth


def start_watch(interval=60.0, top=5, nframe=1, log=print):
    """Start tracing and report the top growing lines every `interval` seconds. Call .set() on the result to stop."""
    tracemalloc.start(nframe)
    stop = threading.Event()

    def loop():
        previous = take_snapshot()
        while not stop.wait(interval):
            current = take_snapshot()
            log("memory growth since the last check:")
            for stat in top_growth(current, previous, "lineno", top):
                log("  " + fmt(stat))
            previous = current

    threading.Thread(target=loop, name="memwatch", daemon=True).start()
    return stop

This demo runs the leaky service for about three and a half seconds with a one second interval:

# demo_watch.py - run the leaky service for a few seconds with the watcher on
import time

import memwatch
import service_v1 as service

stop = memwatch.start_watch(interval=1.0, top=3, nframe=5)
deadline = time.monotonic() + 3.5
request_id = 0
while time.monotonic() < deadline:
    service.handle_request(request_id)
    request_id += 1
stop.set()
print(f"served {request_id:,} requests")

python demo_watch.py

memory growth since the last check:
  service_v1.py:24       2,999.3 KiB      +136 blocks   return header + body
  service_v1.py:18       1,898.8 KiB   +27,666 blocks   return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
  service_v1.py:28       1,095.3 KiB       +56 blocks   buffer = bytearray(40_000)
memory growth since the last check:
  service_v1.py:24       3,881.4 KiB      +176 blocks   return header + body
  service_v1.py:18       2,439.8 KiB   +35,552 blocks   return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
  service_v1.py:28       1,369.1 KiB       +70 blocks   buffer = bytearray(40_000)
served 416 requests

Each report names the same three lines as the original diff, and they keep appearing in every report. That is the signal to look for in a real log: a line that shows up in report after report is a strong leak suspect; a line that appears once is probably a cache filling. In production use an interval of minutes, not seconds, ignore the first reports while caches warm up, and remember that the watcher itself uses memory and CPU, as the tables above show.

Step 9: When tracemalloc and Task Manager disagree

You will meet two situations where the resident memory reported by the operating system and the numbers from tracemalloc do not agree. Neither is a bug in either tool, and knowing them saves hours of searching in the wrong place. This script demonstrates both.

# blindspots.py - two ways resident memory and tracemalloc can disagree
import ctypes
import ctypes.util
import gc
import os
import sys
import tracemalloc

import psutil

process = psutil.Process(os.getpid())


def resident_mib():
    return process.memory_info().rss / 2**20


print("1. Memory allocated by C code outside Python's allocators")
tracemalloc.start()
resident_before = resident_mib()
traced_before = tracemalloc.get_traced_memory()[0]

libc = ctypes.CDLL("ucrtbase" if sys.platform == "win32" else ctypes.util.find_library("c"))
libc.malloc.restype = ctypes.c_void_p
libc.malloc.argtypes = [ctypes.c_size_t]
libc.free.argtypes = [ctypes.c_void_p]
size = 64 * 2**20
pointer = libc.malloc(size)
ctypes.memset(pointer, 1, size)  # touch every page so the operating system really hands it over

resident_after = resident_mib()
traced_after = tracemalloc.get_traced_memory()[0]
print(f"   malloc of 64 MiB: resident {resident_after - resident_before:+.1f} MiB, tracemalloc {(traced_after - traced_before) / 2**20:+.2f} MiB")
libc.free(pointer)
tracemalloc.stop()

print("2. Python memory that was freed but not handed back")


def report(label):
    print(f"   {label:<30}{resident_mib():7.1f} MiB resident{sys.getallocatedblocks():>12,} blocks")


report("start")
keep, junk = [], []
for i in range(1_500_000):
    (keep if i % 25 == 0 else junk).append([i])
report("1.5 million small lists")
del junk
gc.collect()
report("freed 96 percent of them")
del keep
gc.collect()
report("freed the rest")

python blindspots.py

1. Memory allocated by C code outside Python's allocators
   malloc of 64 MiB: resident +64.1 MiB, tracemalloc +0.00 MiB
2. Python memory that was freed but not handed back
   start                            21.9 MiB resident      44,781 blocks
   1.5 million small lists         194.4 MiB resident   4,544,293 blocks
   freed 96 percent of them        183.4 MiB resident     224,540 blocks
   freed the rest                   25.0 MiB resident      44,551 blocks

9.1 Memory that Python never allocated

tracemalloc traces blocks allocated by Python, which is what its documentation says it does. Part 1 allocates 64 MiB directly through the C library’s malloc using ctypes, and writes to it so the operating system really provides the pages. Resident memory grew by 64.1 MiB and tracemalloc saw 0.00 MiB. Native libraries and C extensions can do the same thing without any ctypes in sight. If resident memory climbs but the traced total and the snapshot diff stay flat, the leak is not in Python objects, and the next tool you need is one that tracks native allocations.

9.2 Memory that was freed but not handed back

Part 2 builds 1.5 million small lists, keeps every 25th one, and frees the rest. It uses sys.getallocatedblocks(), which the documentation describes as “Return the number of memory blocks currently allocated by the interpreter, regardless of their size. This function is mainly useful for tracking and debugging memory leaks.” The count fell from 4,544,293 to 224,540 blocks, a drop of about 95 percent, yet resident memory only fell from 194.4 MiB to 183.4 MiB. It dropped to 25.0 MiB only when the survivors were freed too.

Python keeps small objects in larger areas called arenas. The C API documentation says its allocator is “optimized for small objects (smaller or equal to 512 bytes) with a short lifetime” and that arenas have “a fixed size of either 256 KiB on 32-bit platforms or 1 MiB on 64-bit platforms.” My reading of the measurement, not a statement from the documentation, is that an arena can only go back to the operating system when everything in it is free, and the survivors here were spread evenly through memory, so almost every arena had at least one occupant. The practical lesson is that resident memory that stays high after a spike is not proof of a leak.

9.3 A way to decide

• Resident memory climbs and the traced total climbs. A Python-level leak. Use the snapshot diff from Step 2.
• Resident memory climbs and the traced total is flat. Native memory, or fragmentation. Repeat the same workload several times: freed memory that is reused stays flat, a native leak keeps climbing.
• Resident memory jumps after a spike, then stays flat. Probably nothing to fix. A leak keeps rising on every repeat of the workload.

Step 10: Add a test that fails when a leak comes back

Fixing a leak once is not enough if the next refactor can bring it back unnoticed. The idea of this test follows from Step 2: call the code in several equal windows and compare the growth of each window. A cache that is filling up grows in the first window and then stops. A leak grows in every window.

10.1 The helper

growth_per_window calls a function many times and returns the traced growth, in KiB, for each window. assert_no_leak fails if the last window still grows by more than a small allowance. explain_growth is for the moment a test fails: it prints the lines that grew most.

# leakcheck.py - a reusable leak check for tests
import gc
import tracemalloc

from snaps import fmt, take_snapshot, top_growth


def growth_per_window(fn, windows=4, size=300):
    """Call fn(i) windows * size times; return how many KiB the traced memory grew in each window."""
    tracemalloc.start()
    try:
        grown = []
        i = 0
        for _ in range(windows):
            gc.collect()
            start = tracemalloc.get_traced_memory()[0]
            for _ in range(size):
                fn(i)
                i += 1
            gc.collect()
            grown.append((tracemalloc.get_traced_memory()[0] - start) / 1024)
        return grown
    finally:
        tracemalloc.stop()


def assert_no_leak(fn, windows=4, size=300, max_kib=64):
    """Fail if the last window still grows. A cache that is filling up stops growing; a leak does not."""
    grown = growth_per_window(fn, windows, size)
    if grown[-1] > max_kib:
        per_window = ", ".join(f"{g:,.0f}" for g in grown)
        raise AssertionError(f"memory still growing: {per_window} KiB per window of {size} calls")


def explain_growth(fn, size=300, limit=3):
    """Diagnosis helper: the lines that grew most over one window of calls."""
    tracemalloc.start()
    try:
        before = take_snapshot()
        for i in range(size):
            fn(i)
        gc.collect()
        return [fmt(stat) for stat in top_growth(take_snapshot(), before, "lineno", limit)]
    finally:
        tracemalloc.stop()

This short script shows what the windows look like for the leaky service and for the fixed one:

# windows_demo.py - growth per window of 300 calls: a leak keeps growing, a cache that fills up stops
import service_v1
import service_v4
from leakcheck import explain_growth, growth_per_window

for label, service in (("service_v1 (leaky)", service_v1), ("service_v4 (fixed)", service_v4)):
    grown = growth_per_window(service.handle_request)
    print(f"{label:<20}" + "".join(f"{g:>12,.0f}" for g in grown) + "  KiB per window")

print("top growth over one more window of the leaky service:")
for line in explain_growth(service_v1.handle_request):
    print("  " + line)

python windows_demo.py

service_v1 (leaky)        13,259      13,263      13,281      13,245  KiB per window
service_v4 (fixed)         6,400          23           0           0  KiB per window
top growth over one more window of the leaky service:
  service_v1.py:24       6,616.1 KiB      +300 blocks   return header + body
  service_v1.py:18       4,159.6 KiB   +60,600 blocks   return {f"color-{n}": f"#{n:06x}-{self.customer}" for n in range(100)}
  service_v1.py:28       2,347.1 KiB      +120 blocks   buffer = bytearray(40_000)

The leaky service grows by about 13,260 KiB in every window of 300 calls. The fixed service grows by 6,400 KiB in the first window while the page cache (256 pages) and the theme cache (50 customers) fill, then 23 KiB, then nothing. If you had compared only the first window against the last, the fixed service would have looked like a leak, which is the reason for looking at the last window.

10.2 The tests

There are three tests. The first checks the checker: the leaky service must make assert_no_leak fail, because a leak check that never fails proves nothing. The second says the fixed service passes. The third documents the cache-filling behaviour and clears the theme cache first so the test starts cold.

# test_leaks.py - the leak check, tested against the leaky service and the fixed one
import pytest

import service_v1
import service_v4
from leakcheck import assert_no_leak, growth_per_window


def test_leaky_service_is_caught():
    with pytest.raises(AssertionError, match="still growing"):
        assert_no_leak(service_v1.handle_request)


def test_fixed_service_is_flat():
    assert_no_leak(service_v4.handle_request)


def test_a_filling_cache_grows_first_and_then_stops():
    service_v4.load_theme.cache_clear()  # start with a cold cache
    grown = growth_per_window(service_v4.handle_request)
    assert grown[0] > 300  # the 50 customer themes are being cached
    assert grown[-1] < 64  # and then nothing more is kept

python -m pytest -q test_leaks.py

... [100%]
3 passed in 21.43s

All three pass. The test takes about twenty seconds because tracemalloc slows the code down, as Step 8 measured, so keep windows small and run leak tests in a separate, slower job instead of on every save.

10.3 Pitfalls in leak tests

• Warm-up. Looking at the last window, not the first, is what protects you from false alarms. A single threshold on total growth would fail on any service with a legitimate cache.
• Shared state. Module-level caches survive between tests in one process. Clear them (as the third test does) when the test depends on a cold start.
• The allowance. 64 KiB per window suits this service. Pick yours by running the check on a known-good version and leaving some margin.
• Garbage collection. The helper calls gc.collect() at window edges so that objects waiting in reference cycles are not counted as growth.

Confirm the whole thing works end to end

Run these four checks in order. If all four give the results below, you have reproduced a leak, located it, fixed three causes and locked the fix in with a test:

1. python symptom.py service_v1 climbs from about 20 MiB to about 246 MiB over 5,000 requests.
2. python symptom.py service_v4 rises to about 27 MiB in the first batch and stays there.
3. python diff_report.py service_v4 prints no line grew by 20 KiB or more.
4. python -m pytest -q test_leaks.py reports three tests passed.

Common mistakes to avoid

• Calling gc.collect() in the request path to try to fix a leak. Collection only frees objects that can no longer be reached, and a leaked object is by definition still reachable.
• Reading only the size column. The block count tells you how many objects are kept per request, which is the number that identifies the leak.
• Stopping at the allocation line. The line that allocated the memory is rarely the line to change. Find the holder.
• Fixing all leaks at once. After each fix, measure again; the second measurement is what exposed the exception holder here.
• Keeping exception objects in lists, caches or global state. Store text, or drop the traceback.
• Putting self in a cache key without meaning to.

Where to go next

• Save snapshots from production and analyze them elsewhere. The documentation describes Snapshot.dump as “Write the snapshot into a file. Use load() to reload the snapshot.”
• When you can see the leak in a diff but not the holder, run the Step 4 scan inside the live process, for example from a debug endpoint that only an administrator can reach.
• If the problem is time rather than memory, the sampling profiler in How to Find Slow Python Code With the Python 3.15 Tachyon Sampling Profiler answers it.
• If a cache is the right design but its entries expire together, How to Stop a Cache Stampede in Python With Single-Flight Loads, TTL Jitter, and Background Refresh covers the next problem you will meet.
• If the service is already failing under memory pressure, How to Add Backpressure and Load Shedding to a Python Service Before Overload Takes It Down shows how to refuse work gracefully.
• For another class of shared-state surprise in Python objects, see How to Deep Copy Python Objects and Avoid Shared-State Bugs.