Learning Hub

How to Run CPU-Bound Python on Every Core With InterpreterPoolExecutor and Subinterpreters

A hands-on tutorial that times threads, processes and Python 3.14+ subinterpreters on one CPU-bound job, then shows what can cross between interpreters, which popular packages refuse to load in one,...

October 24, 2026 40 Min Read
31

If your Python program spends its time calculating instead of waiting, it has probably never used more than one core of your computer. The usual advice is to switch from threads to processes, but processes are heavy: each one starts a fresh Python, and every argument and result has to be copied between them. Python 3.14 added a middle path to the standard library: subinterpreters, driven by the new concurrent.interpreters module and the InterpreterPoolExecutor class. Each worker gets its own interpreter with its own lock, so workers run on different cores inside a single process.

Table Of Content

In this tutorial you will time one CPU-bound job with threads, processes and interpreters, learn exactly what can and cannot be sent to a worker, check whether the packages you depend on load inside a subinterpreter, and build a parallel access-log analyzer with a test suite. Every number below was measured on one Windows 11 machine (an Intel Core Ultra 5 235 with 14 logical processors) with Python 3.15.1, and the main experiment was repeated on Python 3.14.9. Your numbers will differ. The shapes should not.

Here is the short version of what you will see. With 14 workers, threads ran at 0.75x the speed of a plain loop, interpreters at 10.48x and processes at 10.45x. Starting 14 interpreters took 0.056 seconds and added about 94 MiB, while 14 processes took 0.210 seconds and added about 182 MiB. A worker interpreter shares fate with your program (one call to os._exit ended the whole process), and of 25 popular packages whose compiled modules I tried, only 2 loaded inside a subinterpreter. None of that is a reason to avoid the feature. It is a reason to measure before you adopt it, which is what you will learn to do here.

The ideas you need first

Parallel Python comes with a handful of terms that are easy to mix up. Read these once and the rest of the tutorial will be much easier to follow.

CPU-bound and I/O-bound work

A CPU-bound job spends its time computing, for example parsing a large file or counting primes. An I/O-bound job spends its time waiting for a disk or a network. Interpreters help with the first kind. For the second kind, asyncio or plain threads are usually enough, and our TaskGroup tutorial covers that side.

Threads, processes and the GIL

A thread is a path of execution inside a program, and all threads in a program share the same memory. A process is a separate running program with its own memory. In the standard build of CPython, the global interpreter lock (the GIL) lets only one thread run Python code at a time inside one interpreter, so CPU-bound threads take turns instead of running side by side. You will measure that in Step 2.

Interpreters and subinterpreters

An interpreter is the execution context of the Python runtime. The documentation puts it this way: “It contains all of the state the runtime needs to execute a program.” Your program starts with one, the main interpreter. Any other interpreter created inside the same process is a subinterpreter. Since Python 3.12 each interpreter can have its own GIL, and the same page explains the consequence: “interpreters are sufficiently isolated that they do not share the GIL, which means combining threads with multiple interpreters enables full multi-core parallelism. (This has been the case since Python 3.12.)”

Copying, sharing and pickle

Because interpreters are isolated, a Python object cannot simply be handed from one to another. The usual route is a copy. Pickle is the standard library’s way of turning an object into bytes and back, and the same page says: “By default, most objects are copied with pickle when they are passed to another interpreter.” A few types do share memory instead, which you will test in Step 7.

Executors

The concurrent.futures module gives you pools of workers with two main methods: submit for one call and map for many. InterpreterPoolExecutor has the same interface. The executor documentation says: “It is a ThreadPoolExecutor subclass, which means each worker is running in its own thread. The difference here is that each worker has its own interpreter, and runs each task using that interpreter.”

What you need before you start

  • Python 3.14 or newer. InterpreterPoolExecutor and concurrent.interpreters were added in 3.14. I used 3.15.1 for everything and 3.14.9 for one repeat.
  • An operating system with a terminal. I used Windows 11 with Git Bash. On macOS or Linux, replace venv/Scripts/python.exe with venv/bin/python.
  • The uv installer (optional but used below) to fetch Python 3.14 and 3.15 without replacing your system Python, plus two packages: pytest 9.1.1 and psutil.
  • Some Python. You should be comfortable with functions, classes, reading a traceback, and the basic idea of submit and map.
  • A machine with several cores. The point of the exercise is to see speedups, so a single-core virtual machine will teach you the wrong lesson.

Set up a folder and an environment

These commands install two Python versions into uv’s own folder, create a virtual environment for 3.15, install the two packages, and make an empty working folder called lab. Every script in this tutorial goes into that one folder. The last command checks that the new module imports.

uv python install 3.15 3.14
uv venv --python 3.15 venv
uv pip install --python venv/Scripts/python.exe pytest==9.1.1 psutil
mkdir lab
cd lab
../venv/Scripts/python.exe -c "import sys; from concurrent import interpreters; print(sys.version.split()[0])"

You should see the version number of your interpreter:

3.15.1

What failure looks like on an older Python

If you ever see an ImportError on one of these names, you are running Python 3.13 or older. Here is the real message from Python 3.13.14:

python3.13 --version
python3.13 -c "from concurrent.futures import InterpreterPoolExecutor"
Python 3.13.14
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from concurrent.futures import InterpreterPoolExecutor
ImportError: cannot import name 'InterpreterPoolExecutor' from 'concurrent.futures' (C:\Python313\Lib\concurrent\futures\__init__.py)

Every script below also keeps its work under if __name__ == "__main__":. That guard is required for process pools on Windows, because each worker process imports your script again, and without the guard each of them would try to start its own pool.

Step 1: Build a CPU-bound job and time it without any pool

Before you parallelize anything, you need a job that really is CPU-bound and a baseline time to beat. The job below counts the primes below 1,500,000 by trial division in pure Python. I chose it because it needs almost no input and returns one small integer, so the experiment measures computing and not copying. The range is cut into 224 chunks so a pool has plenty of pieces to hand out.

1a. The worker-side code

Save this as jobs.py. It holds the functions that run inside workers. Keep its imports light, because every worker interpreter imports this module again (Step 8 shows why).

"""jobs.py: worker-side code for the CPU-bound experiments.

Keep the imports here light. Every worker interpreter imports this module again.
"""
import time


def nap(seconds):
    time.sleep(seconds)
    return seconds


def noop(x):
    return x


def count_primes(bounds):
    """Count the primes in [lo, hi) by trial division. Pure Python, no shared data."""
    lo, hi = bounds
    count = 0
    for n in range(max(lo, 2), hi):
        if n % 2 == 0:
            if n == 2:
                count += 1
            continue
        d = 3
        while d * d <= n:
            if n % d == 0:
                break
            d += 2
        else:
            count += 1
    return count


def total_primes(limit, pieces):
    """The whole job in one call: count the primes below `limit`."""
    return sum(count_primes(chunk) for chunk in make_chunks(limit, pieces))


def make_chunks(limit, pieces):
    """Split [0, limit) into about `pieces` (lo, hi) tuples."""
    step = -(-limit // pieces)
    return [(lo, min(lo + step, limit)) for lo in range(0, limit, step)]

count_primes takes one tuple of two integers and returns one integer. make_chunks turns a limit into a list of such tuples. nap and noop are tiny helpers used to warm pools up and to measure overhead.

1b. The main-side helpers

Save this as labkit.py. It builds pools by name, times things, and prints a banner so every output starts with the facts that matter.

"""labkit.py: main-side helpers shared by the step scripts."""
import os
import sys
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

try:  # InterpreterPoolExecutor exists from Python 3.14
    from concurrent.futures import InterpreterPoolExecutor
except ImportError:
    InterpreterPoolExecutor = None

import jobs

POOLS = {
    "threads": ThreadPoolExecutor,
    "interpreters": InterpreterPoolExecutor,
    "processes": ProcessPoolExecutor,
}


def banner():
    gil = getattr(sys, "_is_gil_enabled", lambda: True)()
    print(f"Python {sys.version.split()[0]}, GIL enabled: {gil}, logical CPUs: {os.cpu_count()}")


def start_pool(kind, workers):
    """Create a pool and force every worker to exist before returning.

    Returns (pool, startup_seconds). Each of the `workers` tasks sleeps 0.05 s, so
    no worker can finish early and steal a second task: all of them have to start.
    """
    t0 = time.perf_counter()
    pool = POOLS[kind](max_workers=workers)
    list(pool.map(jobs.nap, [0.05] * workers))
    return pool, time.perf_counter() - t0 - 0.05


def best_of(fn, repeat=3):
    """Run fn() `repeat` times and return (best_seconds, last_result)."""
    best = None
    for _ in range(repeat):
        t0 = time.perf_counter()
        result = fn()
        elapsed = time.perf_counter() - t0
        if best is None or elapsed < best:
            best = elapsed
    return best, result

Two details deserve attention. First, start_pool makes every worker run a 0.05 second nap before it returns, so all workers exist before any timing begins and the startup cost is measured separately. Second, best_of keeps the fastest of three runs, which filters out one-off interruptions.

1c. Run the baseline

Save this as step01_serial.py and run it. It times the plain loop, then checks the answer against a completely different algorithm, the sieve of Eratosthenes.

"""step01_serial.py: check the environment and time the serial baseline."""
from concurrent import interpreters

import jobs
import labkit

LIMIT, PIECES = 1_500_000, 224


def sieve_count(limit):
    """An independent answer from a different algorithm: the sieve of Eratosthenes."""
    flags = bytearray([1]) * limit
    flags[0:2] = bytes(2)
    for p in range(2, int(limit ** 0.5) + 1):
        if flags[p]:
            flags[p * p::p] = bytes(len(range(p * p, limit, p)))
    return sum(flags)


if __name__ == "__main__":
    labkit.banner()
    chunks = jobs.make_chunks(LIMIT, PIECES)
    secs, answer = labkit.best_of(lambda: sum(map(jobs.count_primes, chunks)))
    print(f"serial: {secs:.3f} s for {len(chunks)} chunks, primes below {LIMIT:,}: {answer:,}")
    print("the sieve of Eratosthenes agrees:", sieve_count(LIMIT) == answer)
    print("module imported:", interpreters.__name__)
    print("interpreters in this process:", [i.id for i in interpreters.list_all()])
../venv/Scripts/python.exe step01_serial.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
serial: 1.276 s for 224 chunks, primes below 1,500,000: 114,155
the sieve of Eratosthenes agrees: True
module imported: concurrent.interpreters
interpreters in this process: [0]

The banner confirms two things: this is a standard build where the GIL is on (sys._is_gil_enabled() returned True), and the machine reports 14 logical processors. The answer, 114,155 primes, is confirmed by the sieve, and the serial time (1.276 seconds here) is the number every later result will be compared against. Only one interpreter, number 0, exists so far.

Check before moving on: the sieve line must say True. If it does not, fix the job before measuring anything, because a fast wrong answer is worthless.

Step 2: See for yourself that threads do not help

This is the experiment that explains why the rest of the tutorial exists. It runs the same 224 chunks on a ThreadPoolExecutor with 4 workers.

"""step02_threads.py: do threads speed up a CPU-bound job? (Standard build, GIL on.)"""
import jobs
import labkit

LIMIT, PIECES, WORKERS = 1_500_000, 224, 4

if __name__ == "__main__":
    labkit.banner()
    chunks = jobs.make_chunks(LIMIT, PIECES)
    serial, answer = labkit.best_of(lambda: sum(map(jobs.count_primes, chunks)))
    print(f"serial          {serial:6.3f} s")
    pool, _ = labkit.start_pool("threads", WORKERS)
    with pool:
        secs, total = labkit.best_of(lambda: sum(pool.map(jobs.count_primes, chunks)))
    print(f"{WORKERS} threads       {secs:6.3f} s   speedup {serial / secs:4.2f}x   same answer: {total == answer}")
../venv/Scripts/python.exe step02_threads.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
serial           1.276 s
4 threads        1.903 s   speedup 0.67x   same answer: True

Four threads took 1.903 seconds against 1.276 for the plain loop, a speedup of 0.67x. The answer is still right, but the threads took turns holding the GIL and added overhead on top. More threads will not fix this, as the next steps show. Threads are still the right tool for waiting on networks and disks, where the lock is released while the thread waits.

Check before moving on: the answer must still say True, and the speedup should be about 1.0x or lower. That is the expected result on a standard build, and it is the baseline the next step improves on.

Step 3: Swap in InterpreterPoolExecutor

Now change one thing: the pool class. Save this as step03_interpreters.py.

"""step03_interpreters.py: the same job on InterpreterPoolExecutor."""
import jobs
import labkit

LIMIT, PIECES, WORKERS = 1_500_000, 224, 4

if __name__ == "__main__":
    labkit.banner()
    chunks = jobs.make_chunks(LIMIT, PIECES)
    serial, answer = labkit.best_of(lambda: sum(map(jobs.count_primes, chunks)))
    print(f"serial          {serial:6.3f} s")
    pool, startup = labkit.start_pool("interpreters", WORKERS)
    with pool:
        secs, total = labkit.best_of(lambda: sum(pool.map(jobs.count_primes, chunks)))
    print(f"{WORKERS} interpreters  {secs:6.3f} s   speedup {serial / secs:4.2f}x   same answer: {total == answer}")
../venv/Scripts/python.exe step03_interpreters.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
serial           1.282 s
4 interpreters   0.352 s   speedup 3.64x   same answer: True

Four interpreters finished the same job in 0.352 seconds, a speedup of 3.64x, with the same answer. Nothing else in the script changed, because InterpreterPoolExecutor implements the same Executor interface.

Check before moving on: the speedup should be clearly above 1.0x and the answer must still say True. If the speedup stays near 1.0x, confirm that your machine has more than one core and that the banner shows Python 3.14 or newer.

What happens under the hood

Each worker is a thread, as the documentation says, and when a worker starts it creates its own interpreter. You can read this in the standard library source (Lib/concurrent/futures/interpreter.py at tag v3.15.1): the worker context’s initialize method calls interpreters.create(), finalize calls close(), and each task runs through self.interp.call(do_call, ...). Because that interpreter has its own GIL, the four workers run on four cores at once. The with pool: block matters: it shuts the workers down and closes their interpreters.

Common mistake: submitting thousands of tiny tasks. Step 5 measures what each task costs, and the documentation notes that “With ThreadPoolExecutor and InterpreterPoolExecutor, chunksize has no effect.” To batch work you have to make bigger chunks yourself, as make_chunks does.

Step 4: Scale from 1 to 14 workers and compare with processes

One data point is an anecdote. This script runs threads, interpreters and processes at 1, 2, 4, 8 and 14 workers and prints one table. It re-times the serial loop at the start of every row so you can see whether the machine itself changes speed during the experiment.

"""step04_scaling.py: threads, interpreters and processes at 1 to 14 workers."""
import jobs
import labkit

LIMIT, PIECES = 1_500_000, 224
WORKERS = (1, 2, 4, 8, 14)
KINDS = ("threads", "interpreters", "processes")

if __name__ == "__main__":
    labkit.banner()
    chunks = jobs.make_chunks(LIMIT, PIECES)
    rows = []
    for n in WORKERS:
        serial, answer = labkit.best_of(lambda: sum(map(jobs.count_primes, chunks)))  # re-timed for every row
        times = []
        for kind in KINDS:
            pool, _ = labkit.start_pool(kind, n)
            with pool:
                secs, total = labkit.best_of(lambda: sum(pool.map(jobs.count_primes, chunks)))
            assert total == answer, (kind, n, total, answer)
            times.append(secs)
        rows.append((n, serial, times))
    reference = min(serial for _, serial, _ in rows)
    print(f"speedups use the fastest serial time seen, {reference:.3f} s; the serial column shows how it drifted")
    print(f"{'workers':>7} {'serial':>8} " + " ".join(f"{k:>16}" for k in KINDS))
    for n, serial, times in rows:
        cells = [f"{secs:6.3f}s {reference / secs:5.2f}x" for secs in times]
        print(f"{n:>7} {serial:>7.3f}s " + " ".join(f"{c:>16}" for c in cells))
../venv/Scripts/python.exe step04_scaling.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
speedups use the fastest serial time seen, 1.278 s; the serial column shows how it drifted
workers   serial          threads     interpreters        processes
      1   1.278s    1.805s  0.71x    1.603s  0.80x    1.296s  0.99x
      2   1.383s    2.022s  0.63x    0.748s  1.71x    0.665s  1.92x
      4   1.577s    1.688s  0.76x    0.381s  3.35x    0.342s  3.74x
      8   1.636s    1.703s  0.75x    0.192s  6.65x    0.184s  6.93x
     14   1.679s    1.707s  0.75x    0.122s 10.48x    0.122s 10.45x

How to read the table

Threads are flat. From 1 worker to 14, the time stays between 1.7 and 2.0 seconds. Adding threads buys nothing, which is the GIL at work.

Interpreters and processes track each other. At 14 workers the speedups are 10.48x and 10.45x, and at 4 workers 3.35x and 3.74x. On this machine 14 workers on 14 logical processors give a bit over 10x, so do not expect a perfect 14x from yours either.

The serial column drifted. The same single-threaded loop took 1.278 seconds in the first row and 1.679 seconds in the last, although nothing in the code changed. Fresh worker processes stayed fast in the one-worker row (0.99x), while threads (0.71x) and interpreters (0.80x), which run inside the main process, did not. I did not find the cause, so I make no claim about it. Because of it, all speedups in the table use the fastest serial time seen (1.278 seconds) as the reference, which makes the later rows conservative. Treat differences of 10 to 15 percent between backends as noise.

Check before moving on: the interpreter and process columns should climb as workers are added, and the thread column should stay flat. If every column is flat, the job is too short for your machine, so raise LIMIT in the script and run it again.

Does Python 3.14 behave the same?

Run the same script with the 3.14 environment (create it like the first one, with uv venv --python 3.14 venv14 and the same two packages). Here is what Python 3.14.9 printed:

Python 3.14.9, GIL enabled: True, logical CPUs: 14
speedups use the fastest serial time seen, 1.826 s; the serial column shows how it drifted
workers   serial          threads     interpreters        processes
      1   1.826s    2.273s  0.80x    2.227s  0.82x    1.839s  0.99x
      2   2.208s    2.269s  0.80x    0.999s  1.83x    0.940s  1.94x
      4   2.166s    2.251s  0.81x    0.540s  3.38x    0.484s  3.78x
      8   2.036s    2.667s  0.68x    0.269s  6.79x    0.262s  6.97x
     14   2.011s    2.314s  0.79x    0.177s 10.33x    0.168s 10.85x

The loop itself is slower on 3.14.9 (1.826 seconds against 1.278 on 3.15.1), but the ratios match: at 14 workers interpreters reached 10.33x, processes 10.85x, and threads 0.79x.

Step 5: Find out what a worker costs

A speedup table hides the price of getting there. This script measures three things for a pool of 14 workers: how long it takes to start, how much private memory it adds (using psutil’s unique set size for the main process and for each child process), and how many tiny tasks per second it can push through, which is a direct view of per-task overhead.

"""step05_startup.py: what does a pool of 14 workers cost to start and to keep?"""
import time

import psutil

import jobs
import labkit

WORKERS, TINY = 14, 2000


def private_bytes(proc):
    info = proc.memory_full_info()
    return getattr(info, "uss", info.rss)


if __name__ == "__main__":
    labkit.banner()
    me = psutil.Process()
    print(f"{'backend':<13} {'startup':>9} {'extra memory':>13} {'tiny tasks/s':>13}")
    for kind in ("threads", "interpreters", "processes"):
        before = private_bytes(me)
        pool, startup = labkit.start_pool(kind, WORKERS)
        with pool:
            kids = sum(private_bytes(c) for c in me.children(recursive=True))
            extra = private_bytes(me) - before + kids
            t0 = time.perf_counter()
            list(pool.map(jobs.noop, range(TINY)))
            rate = TINY / (time.perf_counter() - t0)
        print(f"{kind:<13} {startup:>8.3f}s {extra / 2**20:>10.1f} MiB {rate:>13,.0f}")
../venv/Scripts/python.exe step05_startup.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
backend         startup  extra memory  tiny tasks/s
threads          0.004s        0.5 MiB       218,355
interpreters     0.056s       94.4 MiB        44,284
processes        0.210s      182.4 MiB        10,820

Fourteen interpreters were ready in 0.056 seconds and added 94.4 MiB. Fourteen processes needed 0.210 seconds and added 182.4 MiB. For tiny tasks the pools handled about 44,000 (interpreters) and 10,800 (processes) per second, against 218,000 for threads. By my arithmetic that is roughly 23 microseconds of overhead per task for interpreters and 92 for processes. Our job’s chunks take about 5.7 milliseconds each (1.276 seconds over 224 chunks), so overhead is well under one percent. If your tasks last a few microseconds, batch them first.

Step 6: Learn what can cross between interpreters

Everything you send to a worker, and everything it sends back, has to cross an interpreter boundary. The documentation of Interpreter.call in the standard library source says what is supported: “Nearly all callables, args, kwargs, and return values are supported. All “shareable” objects are supported, as are “stateless” functions (meaning non-closures that do not use any globals). This method will fall back to pickle.” The best way to learn the rules is to probe them.

6a. The probe functions

Save this as probes.py. This is the first part. The next steps add more functions to the same file, and I mark those with a comment line.

"""probes.py: small worker-side functions used by steps 6 to 9."""
import os
import sys
import threading
import time
from concurrent import interpreters
from dataclasses import dataclass


@dataclass
class Point:
    x: int
    y: int


def type_name(obj):
    return type(obj).__name__


def make_point():
    return Point(1, 2)


def make_lock():
    return threading.Lock()


def make_generator():
    return (i for i in range(3))


def make_closure():
    factor = 5
    return lambda x: x * factor


def size_of(obj):
    return len(obj)


def make_bytes(n):
    return bytes(n)


def shout_in_place(view):
    """Turn lowercase ASCII letters into uppercase by writing straight into the caller's memory."""
    for i in range(len(view)):
        if 97 <= view[i] <= 122:
            view[i] -= 32
    return len(view)


def append_marker(items):
    items.append("worker was here")
    return len(items)

6b. The probe script

This script sends 17 kinds of object to a worker and 5 kinds back, and prints whether each crossing worked.

"""step06_types.py: what can cross the boundary between two interpreters?"""
import threading
from concurrent import interpreters
from concurrent.futures import InterpreterPoolExecutor
from decimal import Decimal
from functools import partial

import probes


class Cart:
    """Defined in the script itself, so it lives in __main__."""

    def __init__(self):
        self.items = ["tea"]


def make_adder(n):
    return lambda x: x + n


def run(pool, label, fn, *args, report=str):
    try:
        out = pool.submit(fn, *args).result()
        return f"{label:<22} ok      {report(out)}"
    except BaseException as exc:
        cause = type(exc.__cause__).__name__ if exc.__cause__ else "none"
        return f"{label:<22} FAILED  {type(exc).__name__} (cause: {cause})"


if __name__ == "__main__":
    handle = open(__file__)
    arguments = [
        ("int", 42), ("str", "tea"), ("bytes", b"tea"), ("tuple of ints", (1, 2, 3)),
        ("list", [1, 2, 3]), ("dict", {"a": 1}), ("set", {1, 2}), ("Decimal", Decimal("1.10")),
        ("Point (probes.py)", probes.Point(1, 2)), ("Cart (this script)", Cart()),
        ("lambda", lambda: 1), ("closure", make_adder(5)), ("generator", (i for i in range(3))),
        ("threading.Lock", threading.Lock()), ("open file", handle),
        ("memoryview", memoryview(bytearray(b"abc"))), ("interpreters.Queue", interpreters.create_queue()),
    ]
    results = [
        ("Point", probes.make_point), ("Lock", probes.make_lock), ("generator", probes.make_generator),
        ("closure", probes.make_closure), ("bytes(1024)", partial(probes.make_bytes, 1024)),
    ]
    with InterpreterPoolExecutor(2) as pool:
        print("Arguments: the worker reports the type it received")
        for label, obj in arguments:
            print(run(pool, label, probes.type_name, obj))
        print("\nReturn values: the worker builds an object and sends it back")
        for label, fn in results:
            print(run(pool, label, fn, report=lambda out: f"main received a {type(out).__name__}"))
    handle.close()
../venv/Scripts/python.exe step06_types.py
Arguments: the worker reports the type it received
int                    ok      int
str                    ok      str
bytes                  ok      bytes
tuple of ints          ok      tuple
list                   ok      list
dict                   ok      dict
set                    ok      set
Decimal                ok      Decimal
Point (probes.py)      ok      Point
Cart (this script)     ok      Cart
lambda                 ok      function
closure                FAILED  NotShareableError (cause: NotShareableError)
generator              FAILED  NotShareableError (cause: NotShareableError)
threading.Lock         FAILED  NotShareableError (cause: NotShareableError)
open file              FAILED  NotShareableError (cause: NotShareableError)
memoryview             ok      memoryview
interpreters.Queue     ok      Queue

Return values: the worker builds an object and sends it back
Point                  ok      main received a Point
Lock                   FAILED  NotShareableError (cause: Exception)
generator              FAILED  NotShareableError (cause: Exception)
closure                FAILED  NotShareableError (cause: Exception)
bytes(1024)            ok      main received a bytes

What the matrix teaches

  • Plain values cross. Integers, strings, bytes and tuples of them are shared or copied efficiently. Lists, dictionaries, sets, a Decimal and a dataclass from an importable module also cross, copied through pickle.
  • Closures, generators, locks and open files fail with NotShareableError. The documentation says this exception is “a subclass of TypeError” raised “when an object cannot be sent to another interpreter.”
  • A lambda crossed. Pickle cannot serialize a lambda, yet it worked, because a function with no closure and no globals is what the quoted docstring calls a stateless function, and those are copied by value before pickle is tried. Do not read this as a license to send lambdas, though. A function that uses a global is not stateless, and the next section explains what happens then.
  • memoryview and Queue crossed as themselves. The documentation names them as the types that “actually share mutable data between interpreters.”

6c. The main-script rule that surprises everyone

One row above looks wrong: Cart (this script) is a class defined in the script that submits the work, and it crossed. How did a worker get a class that only exists in your script? The next script prints a line whenever the top level of the script runs.

"""step10_main_rerun.py: workers re-run this script's top level, but not what sits under the main guard."""
import sys
from concurrent import interpreters
from concurrent.futures import InterpreterPoolExecutor

import probes

sys.stdout.reconfigure(line_buffering=True)  # show main and worker lines in the order they happen
print(f"  top level ran: __name__={__name__!r}, interpreter {interpreters.get_current().id}", flush=True)

FACTOR = 3


def scaled(x):
    return x * FACTOR  # uses a global, so it cannot be copied on its own


def dump(x):
    return json.dumps(x)  # json is imported only under the main guard below


if __name__ == "__main__":
    import json

    print("A: submit probes.type_name, a function from another module")
    with InterpreterPoolExecutor(2) as pool:
        print("  result:", pool.submit(probes.type_name, 1).result())
    print("B: submit scaled, a function defined in this script that uses a global")
    with InterpreterPoolExecutor(2) as pool:
        print("  result:", list(pool.map(scaled, range(4))))
    print("C: submit dump, which needs a name that exists only under the main guard")
    with InterpreterPoolExecutor(1) as pool:
        try:
            pool.submit(dump, {"a": 1}).result()
        except NameError as exc:
            print(f"  NameError: {exc}")
../venv/Scripts/python.exe step10_main_rerun.py
  top level ran: __name__='__main__', interpreter 0
A: submit probes.type_name, a function from another module
  result: int
B: submit scaled, a function defined in this script that uses a global
  top level ran: __name__='<fake __main__>', interpreter 3
  top level ran: __name__='<fake __main__>', interpreter 2
  result: [0, 3, 6, 9]
C: submit dump, which needs a name that exists only under the main guard
  top level ran: __name__='<fake __main__>', interpreter 4
  NameError: name 'json' is not defined

In part A, a function from another module did not trigger anything. In part B, the function scaled uses a global, so it could not be copied alone, and each worker ran this script’s top level again, with __name__ set to '<fake __main__>'. That is why the Cart class was available, and why the main guard still protects you: its body did not run in the workers. Part C shows the flip side. A name that exists only under the guard, here the json import, does not exist in the worker, so the call fails with NameError.

I could not find this behavior in the documentation pages I read, so treat it as observed on Python 3.15.1 and not as a promise. The practical rules are safe either way: put the functions and classes you send to workers in a normal module (like jobs.py), import what they need at the top of that module, and keep the top level of your main script cheap, because any expensive top-level work will repeat once per worker.

Step 7: Move data the cheap way and the shared way

Crossing the boundary costs time proportional to the data. This script sends 64 MiB of bytes, a list of two million integers and a 64 MiB memoryview to one worker of each kind, then asks the worker to build 64 MiB and send it back. It finishes with the two experiments that show what is shared and what is copied.

"""step07_data.py: what does it cost to move data, and what is really shared?"""
import os
import time
from concurrent.futures import InterpreterPoolExecutor

import labkit
import probes

MIB = 2**20


def send(pool, obj):
    """Seconds to send obj to the worker and get its length back."""
    t0 = time.perf_counter()
    try:
        pool.submit(probes.size_of, obj).result()
    except TypeError as exc:
        return f"TypeError: {str(exc)[:30]}"
    return f"{time.perf_counter() - t0:.3f} s"


def receive(pool):
    t0 = time.perf_counter()
    pool.submit(probes.make_bytes, 64 * MIB).result()
    return f"{time.perf_counter() - t0:.3f} s"


if __name__ == "__main__":
    labkit.banner()
    payload = os.urandom(64 * MIB)
    numbers = list(range(2_000_000))
    view = memoryview(bytearray(payload))
    pools = {kind: labkit.POOLS[kind](max_workers=1) for kind in ("interpreters", "processes")}
    for pool in pools.values():
        pool.submit(probes.size_of, b"warm up").result()  # start the worker before timing
    print(f"{'one-way trip of':<22} {'interpreters':>14} {'processes':>14}")
    for label, obj in [("bytes, 64 MiB", payload), ("list of 2M ints", numbers), ("memoryview, 64 MiB", view)]:
        print(f"{label:<22} {send(pools['interpreters'], obj):>14} {send(pools['processes'], obj):>14}")
    print(f"{'bytes(64 MiB) returned':<22} {receive(pools['interpreters']):>14} {receive(pools['processes']):>14}")
    for pool in pools.values():
        pool.shutdown()

    letters = bytearray(b"hello interpreters")
    with InterpreterPoolExecutor(1) as pool:
        pool.submit(probes.shout_in_place, memoryview(letters)).result()
    print("\nbytearray after a worker wrote through a memoryview:", bytes(letters))

    items = ["main"]
    with InterpreterPoolExecutor(1) as pool:
        inside = pool.submit(probes.append_marker, items).result()
    print(f"list inside the worker had {inside} items; the list in main is still {items}")
../venv/Scripts/python.exe step07_data.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
one-way trip of          interpreters      processes
bytes, 64 MiB                 0.021 s        0.110 s
list of 2M ints               0.139 s        0.105 s
memoryview, 64 MiB            0.000 s TypeError: cannot pickle memoryview objec
bytes(64 MiB) returned        0.028 s        0.140 s

bytearray after a worker wrote through a memoryview: b'HELLO INTERPRETERS'
list inside the worker had 2 items; the list in main is still ['main']

What the numbers say

  • Bytes are cheap. 64 MiB took 0.021 seconds to reach an interpreter and 0.110 seconds to reach a process, and 64 MiB came back in 0.028 seconds against 0.140. The documentation says “Nearly all of the immutable builtin objects are either directly shared or copied efficiently.”
  • A list of integers is not. It has to be pickled, and here interpreters (0.139 seconds) were slightly slower than processes (0.105). Interpreters are not automatically cheaper for every payload.
  • A memoryview costs nothing to send (0.000 seconds), and processes cannot send one at all: TypeError: cannot pickle memoryview objects.
  • The shared memory is really shared. The worker uppercased the letters through the view, and the bytearray in the main interpreter changed to HELLO INTERPRETERS. The list, by contrast, was copied: the worker saw 2 items and the main list still holds one.

Be careful with shared memory. The documentation warns: “Any data actually shared between interpreters loses the thread-safety provided by the GIL.” If two workers write to the same bytes, you are back to race conditions. Give each worker its own disjoint slice, and for most programs prefer the simplest pattern of all: send small descriptions of work (a file path and two offsets, as in Step 11) and let each worker read its own input.

Step 8: Remember that every interpreter has its own state

Module-level variables, imported modules and caches belong to one interpreter. That is the source of both the isolation and several surprises. First add the second part of probes.py (workers use it to count calls and to build a table in an initializer), and a small module that announces when it is imported.

# probes.py (continued)
_calls = 0


def tick(_=None):
    """Count calls in a module global; sleep so tasks spread over all workers."""
    global _calls
    _calls += 1
    time.sleep(0.05)
    return interpreters.get_current().id, _calls


TABLE = None
BUILD_SECONDS = 0.0


def build_table(size):
    """An initializer: runs once in every worker interpreter."""
    global TABLE, BUILD_SECONDS
    t0 = time.perf_counter()
    TABLE = [i * i % 1009 for i in range(size)]
    BUILD_SECONDS = time.perf_counter() - t0


def table_info(_=None):
    time.sleep(0.05)
    return interpreters.get_current().id, id(TABLE), len(TABLE), round(BUILD_SECONDS, 3)
"""import_probe.py: prints a line whenever the module body runs."""
from concurrent import interpreters

print(f"  import_probe body ran in interpreter {interpreters.get_current().id}", flush=True)


def hello(_=None):
    return "hello"

Now run a script that compares an interpreter pool with a thread pool.

"""step08_state.py: every worker interpreter has its own module state."""
import sys
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor

import probes

sys.stdout.reconfigure(line_buffering=True)  # show main and worker lines in the order they happen


def show_pool(title, pool_cls, workers=3):
    print(title)
    with pool_cls(workers, initializer=probes.build_table, initargs=(300_000,)) as pool:
        for round_no in (1, 2):
            ticks = sorted(pool.map(probes.tick, range(workers)))
            print(f"  round {round_no}: (interpreter id, calls so far) = {ticks}")
        infos = set(pool.map(probes.table_info, range(workers)))
        print(f"  distinct interpreters: {len({i[0] for i in infos})}, distinct TABLE objects: {len({i[1] for i in infos})}")


if __name__ == "__main__":
    show_pool("InterpreterPoolExecutor(3, initializer=build_table):", InterpreterPoolExecutor)
    show_pool("ThreadPoolExecutor(3, initializer=build_table):", ThreadPoolExecutor)

    print("\nImporting a module whose body prints:")
    import import_probe  # runs once in the main interpreter

    for title, pool_cls in [("interpreters", InterpreterPoolExecutor), ("threads", ThreadPoolExecutor)]:
        print(f"{title}: submit import_probe.hello to 3 workers")
        with pool_cls(3) as pool:
            list(pool.map(import_probe.hello, range(3)))
../venv/Scripts/python.exe step08_state.py
InterpreterPoolExecutor(3, initializer=build_table):
  round 1: (interpreter id, calls so far) = [(1, 1), (2, 1), (3, 1)]
  round 2: (interpreter id, calls so far) = [(1, 2), (2, 2), (3, 2)]
  distinct interpreters: 3, distinct TABLE objects: 3
ThreadPoolExecutor(3, initializer=build_table):
  round 1: (interpreter id, calls so far) = [(0, 3), (0, 3), (0, 3)]
  round 2: (interpreter id, calls so far) = [(0, 6), (0, 6), (0, 6)]
  distinct interpreters: 1, distinct TABLE objects: 1

Importing a module whose body prints:
  import_probe body ran in interpreter 0
interpreters: submit import_probe.hello to 3 workers
  import_probe body ran in interpreter 5
  import_probe body ran in interpreter 6
  import_probe body ran in interpreter 4
threads: submit import_probe.hello to 3 workers

What this shows

  • State persists inside a worker but is not shared. In the interpreter pool each worker counted its own calls: 1 then 2. In the thread pool the three workers shared one global, so the counter reached 3 then 6.
  • An initializer runs once per worker. Three interpreters built three separate TABLE objects, so a big lookup table costs its memory once per worker. The documentation notes that the executor “serializes the initializer and initargs using pickle”.
  • Imports run once per interpreter. The body of import_probe ran in the main interpreter (0) and again in workers 5, 6 and 4. With threads it never ran again. Heavy imports are therefore a per-worker startup cost. The lazy imports tutorial shows how to postpone them, though I did not measure that combination here.

Interpreter numbers depend on how many interpreters have existed before, so yours may differ and the lines may appear in a different order.

Step 9: Understand failures, exits and crashes

Two things can go wrong in a worker: it can raise an exception, or it can take the process down. Add the last part of probes.py and run the failure script.

# probes.py (continued)
class InputError(Exception):
    """A custom exception that lives in an importable module."""


def fail_value(x):
    raise ValueError(f"bad value {x}")


def fail_custom():
    raise InputError("custom failure")


def fail_sysexit():
    sys.exit(3)


def fail_osexit():
    os._exit(3)
"""step09_failures.py: what a failure looks like when the work ran in another interpreter."""
import os
import subprocess
import sys
import sysconfig
from concurrent.futures import InterpreterPoolExecutor

import probes

HERE = os.getcwd()

CHILD = """
import probes
from concurrent.futures import {pool_cls}

if __name__ == '__main__':
    with {pool_cls}(1) as pool:
        try:
            pool.submit(probes.{name}).result()
        except BaseException as exc:
            print('main caught', type(exc).__name__, exc)
    print('main is still alive')
"""


def show(label, fn, *args):
    try:
        fn(*args).result()
    except BaseException as exc:
        cause = exc.__cause__
        print(f"{label}: raised {type(exc).__name__}: {exc}")
        print(f"   __cause__ is {type(cause).__name__}; its text starts:")
        text = str(cause).replace(HERE, ".").replace(sysconfig.get_path("stdlib"), "<stdlib>")
        for line in text.splitlines()[:7]:
            print("      " + line)


if __name__ == "__main__":
    with InterpreterPoolExecutor(1) as pool:
        show("ValueError", pool.submit, probes.fail_value, 3)
        show("custom InputError", pool.submit, probes.fail_custom)
    for pool_cls, name in [("InterpreterPoolExecutor", "fail_sysexit"), ("InterpreterPoolExecutor", "fail_osexit"),
                           ("ProcessPoolExecutor", "fail_osexit")]:
        done = subprocess.run([sys.executable, "-c", CHILD.format(pool_cls=pool_cls, name=name)],
                              capture_output=True, text=True)
        print(f"\n{pool_cls}: the worker runs probes.{name}() -> process exit code {done.returncode}")
        for line in done.stdout.splitlines():
            print("   " + line)
        if not done.stdout.strip():
            print("   (no output at all)")
../venv/Scripts/python.exe step09_failures.py
ValueError: raised ValueError: bad value 3
   __cause__ is ExecutionFailed; its text starts:
      ValueError: bad value 3
      
      Uncaught in the interpreter:
      
      Traceback (most recent call last):
        File "<stdlib>\concurrent\futures\interpreter.py", line 11, in do_call
          return func(*args, **kwargs)
custom InputError: raised InputError: custom failure
   __cause__ is ExecutionFailed; its text starts:
      probes.InputError: custom failure
      
      Uncaught in the interpreter:
      
      Traceback (most recent call last):
        File "<stdlib>\concurrent\futures\interpreter.py", line 11, in do_call
          return func(*args, **kwargs)

InterpreterPoolExecutor: the worker runs probes.fail_sysexit() -> process exit code 0
   main caught SystemExit 3
   main is still alive

InterpreterPoolExecutor: the worker runs probes.fail_osexit() -> process exit code 3
   (no output at all)

ProcessPoolExecutor: the worker runs probes.fail_osexit() -> process exit code 0
   main caught BrokenProcessPool A process in the process pool was terminated abruptly while the future was running or pending.
   main is still alive

Reading the results

Exceptions keep their type. The caller got a real ValueError and a real InputError, not a generic wrapper. The pool documentation describes the mechanism: “the worker always tries to preserve the exception as-is. If that is successful then it also sets the __cause__ to a corresponding ExecutionFailed instance, which contains a summary of the original exception.” The __cause__ text above includes the worker’s traceback header, and I cut it to seven lines. I kept the custom exception class in an importable module, probes.py, which is the safe choice for exceptions that have to travel back.

sys.exit() is only an exception. The program in the first child received SystemExit 3, handled it, and carried on.

os._exit() ends everything. The same call inside a worker interpreter ended the whole program with exit code 3 and printed nothing, not even the last line. With a process pool the identical call produced BrokenProcessPool and the main program survived. A worker interpreter shares its fate with your program, and the documentation is blunt about the limits of isolation: “do not use multiple interpreters in security-sensitive situations, where they shouldn’t have access to each other’s data.” I did not test a native crash inside an extension module, but the same reasoning applies.

Step 10: Check which of your dependencies load

This is the step that decides whether interpreters are practical for your project. The documentation lists it among the key details: “not all PyPI packages support use in multiple interpreters yet”. A compiled extension module must opt in, and the extension isolation guide explains how: “To use per-module state, use multi-phase extension module initialization. This signals that your module supports multiple interpreters correctly.” Modules that have not done this refuse to load, so test your own list.

10a. The standard library

This script imports every public standard-library module in a brand new subinterpreter, one interpreter per module, and reports the failures.

"""survey_stdlib.py: which standard-library modules refuse to load in a fresh subinterpreter?"""
import importlib
import sys
from concurrent import interpreters

SKIP = {"antigravity", "this", "idlelib", "turtledemo", "tkinter", "turtle", "test", "pydoc_data"}

if __name__ == "__main__":
    tried = failed = 0
    for name in sorted(sys.stdlib_module_names):
        if name.startswith("_") or name in SKIP:
            continue
        try:
            importlib.import_module(name)
        except BaseException:
            continue  # not importable on this platform even in the main interpreter
        tried += 1
        interp = interpreters.create()
        try:
            interp.exec(f"import warnings; warnings.simplefilter('ignore'); import {name}")
        except interpreters.ExecutionFailed as exc:
            failed += 1
            print(f"  {name:<14} {exc.excinfo.type.__name__}: {exc.excinfo.msg}")
        finally:
            interp.close()
    print(f"{tried} public standard-library modules import in the main interpreter; {failed} fail in a fresh subinterpreter")

10b. Third-party packages

The probe imports one module in the main interpreter and then in a fresh subinterpreter, and prints the two results on one line. The survey script installs 28 popular packages from wheels into an empty environment and runs the probe on the compiled module of each, because the pure-Python wrapper of a package can import fine while its extension does not.

"""probe_import.py MODULE: does MODULE import in the main interpreter and in a fresh subinterpreter?

Prints one line: MAIN_RESULT|SUBINTERPRETER_RESULT
"""
import re
import sys
from concurrent import interpreters

name = sys.argv[1]
try:
    __import__(name)
    main_result = "ok"
except BaseException as exc:
    main_result = f"FAILED {type(exc).__name__}"

sub_result = "-"
if main_result == "ok":
    interp = interpreters.create()
    try:
        interp.exec(f"import {name}")
        sub_result = "ok"
    except interpreters.ExecutionFailed as exc:
        text = " ".join(exc.excinfo.msg.split())  # numpy's message spans many lines
        refused = re.search(r"module (\S+) does not support loading in subinterpreters", text)
        sub_result = f"refused: {refused[1]}" if refused else f"{exc.excinfo.type.__name__}: {text[:100]}"
    finally:
        interp.close()
print(f"{main_result}|{sub_result}")
"""survey_pkgs.py UV_EXE SURVEY_PYTHON: install popular packages from wheels and try each in a subinterpreter.

SURVEY_PYTHON is the python.exe of an empty virtual environment dedicated to this survey.
The module named for each package is the compiled part, not just the Python wrapper around it.
"""
import subprocess
import sys

PACKAGES = [
    ("numpy", "numpy"), ("pandas", "pandas"), ("pydantic-core", "pydantic_core._pydantic_core"),
    ("orjson", "orjson"), ("ujson", "ujson"), ("lxml", "lxml.etree"),
    ("cryptography", "cryptography.hazmat.bindings._rust"), ("bcrypt", "bcrypt._bcrypt"),
    ("pillow", "PIL._imaging"), ("regex", "regex._regex"), ("psutil", "psutil"), ("pyyaml", "yaml._yaml"),
    ("msgpack", "msgpack._cmsgpack"), ("cffi", "_cffi_backend"), ("markupsafe", "markupsafe._speedups"),
    ("charset-normalizer", "charset_normalizer.md"), ("multidict", "multidict._multidict"),
    ("yarl", "yarl._quoting_c"), ("grpcio", "grpc._cython.cygrpc"), ("protobuf", "google._upb._message"),
    ("zstandard", "zstandard.backend_c"), ("lz4", "lz4._version"), ("rpds-py", "rpds"), ("jiter", "jiter"),
    ("greenlet", "greenlet._greenlet"), ("websockets", "websockets.speedups"),
    ("httptools", "httptools.parser.parser"), ("sqlalchemy", "sqlalchemy.util._collections_cy"),
]


def run(cmd, timeout=240):
    return subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)


if __name__ == "__main__":
    uv, py = sys.argv[1], sys.argv[2]
    probe = __file__.replace("survey_pkgs.py", "probe_import.py")
    print(f"{'package':<19} {'version':<10} {'compiled module imported':<38} {'main':<7} subinterpreter")
    counts = {"no wheel": 0, "ok": 0, "refused": 0}
    for pip_name, module in PACKAGES:
        install = run([uv, "pip", "install", "--python", py, "--only-binary", ":all:", "-q", pip_name])
        if install.returncode != 0:
            counts["no wheel"] += 1
            print(f"{pip_name:<19} {'-':<10} {module:<38} {'-':<7} no wheel for Python 3.15")
            continue
        version = run([py, "-c", f"import importlib.metadata as m; print(m.version({pip_name!r}))"]).stdout.strip()
        probed = run([py, probe, module])
        line = probed.stdout.strip().splitlines()[-1] if probed.stdout.strip() else ""
        if "|" not in line:
            counts["refused"] += 1
            print(f"{pip_name:<19} {version:<10} {module:<38} {'-':<7} process died, exit code {probed.returncode}")
            continue
        main_result, sub_result = line.split("|", 1)
        counts["ok" if sub_result == "ok" else "refused"] += 1
        print(f"{pip_name:<19} {version:<10} {module:<38} {main_result:<7} {sub_result}")
    print(f"\nloads in a subinterpreter: {counts['ok']}, refused or failed: {counts['refused']}, no wheel yet: {counts['no wheel']}")
../venv/Scripts/python.exe survey_stdlib.py
uv venv --python 3.15 ../survey_venv
../venv/Scripts/python.exe survey_pkgs.py uv ../survey_venv/Scripts/python.exe
  faulthandler   ImportError: module faulthandler does not support loading in subinterpreters
  tracemalloc    ImportError: module _tracemalloc does not support loading in subinterpreters
174 public standard-library modules import in the main interpreter; 2 fail in a fresh subinterpreter
package             version    compiled module imported               main    subinterpreter
numpy               2.5.4      numpy                                  ok      refused: numpy._core._multiarray_umath
pandas              3.1.0      pandas                                 ok      ImportError: Unable to import required dependency numpy. Please see the traceback for details.
pydantic-core       2.50.1     pydantic_core._pydantic_core           ok      refused: pydantic_core._pydantic_core
orjson              3.13.0     orjson                                 ok      refused: orjson.orjson
ujson               6.0.0      ujson                                  ok      refused: ujson
lxml                6.1.3      lxml.etree                             ok      refused: lxml.etree
cryptography        50.0.2     cryptography.hazmat.bindings._rust     ok      refused: cryptography.hazmat.bindings._rust
bcrypt              5.0.0      bcrypt._bcrypt                         ok      refused: bcrypt._bcrypt
pillow              13.0.0     PIL._imaging                           ok      refused: PIL._imaging
regex               2026.9.29  regex._regex                           ok      refused: regex._regex
psutil              7.2.2      psutil                                 ok      refused: psutil._psutil_windows
pyyaml              -          yaml._yaml                             -       no wheel for Python 3.15
msgpack             1.2.3      msgpack._cmsgpack                      ok      refused: msgpack._cmsgpack
cffi                2.1.1      _cffi_backend                          ok      refused: _cffi_backend
markupsafe          3.0.4      markupsafe._speedups                   ok      ok
charset-normalizer  3.5.2      charset_normalizer.md                  ok      refused: charset_normalizer.cd
multidict           7.2.0      multidict._multidict                   ok      ok
yarl                1.25.1     yarl._quoting_c                        ok      refused: yarl._quoting_c
grpcio              1.84.1     grpc._cython.cygrpc                    ok      refused: grpc._cython.cygrpc
protobuf            7.36.2     google._upb._message                   ok      refused: google._upb._message
zstandard           -          zstandard.backend_c                    -       no wheel for Python 3.15
lz4                 -          lz4._version                           -       no wheel for Python 3.15
rpds-py             2026.9.1   rpds                                   ok      refused: rpds.rpds
jiter               0.18.0     jiter                                  ok      refused: jiter.jiter
greenlet            3.5.6      greenlet._greenlet                     ok      refused: greenlet._greenlet
websockets          17.3       websockets.speedups                    ok      refused: websockets.speedups
httptools           0.9.0      httptools.parser.parser                ok      refused: httptools.parser.parser
sqlalchemy          2.1.6      sqlalchemy.util._collections_cy        ok      refused: sqlalchemy.util._collections_cy

loads in a subinterpreter: 2, refused or failed: 23, no wheel yet: 3

What the surveys found

  • The standard library is almost clean. Of 174 public modules that import normally, only faulthandler and tracemalloc refuse to load in a subinterpreter. If you profile memory with the tracemalloc approach, do it in the main interpreter.
  • Third-party support is thin. Of 25 packages with a wheel for Python 3.15 on Windows, only the compiled modules of markupsafe and multidict loaded. NumPy, Pillow, lxml, cryptography, pydantic-core, orjson, regex, psutil and 14 others raised ImportError, and pandas failed because it needs NumPy. Three packages (PyYAML, zstandard, lz4) had no wheel for 3.15 yet at the time of the run.
  • The failure is clean. You get an ImportError naming the module, not a crash. NumPy wraps the same sentence inside a long troubleshooting banner, so read to the end of its message.

The survey is a snapshot of one platform on one day, so rerun it for your own dependency list before you commit to a design. When a package you need refuses to load, you have two options: keep that work in the main interpreter or a process pool, or ask the maintainers about multi-phase initialization using the guide above.

Step 11: Build the parallel log analyzer

Now apply everything to a realistic job. The analyzer reads a web-server access log, counts requests by status code and by first path segment, and totals bytes and latency. The design follows from what you learned: send small descriptions of work, let each worker read its own piece of the file, return small plain dictionaries, and merge them in the main interpreter.

11a. A repeatable log file

Save this as gen_logs.py. It writes a seeded synthetic log, so every run produces the identical file.

"""gen_logs.py PATH [LINES]: write a deterministic synthetic access log (seeded, so every run is identical).

Line format: TIMESTAMP IP METHOD PATH STATUS BYTES LATENCY_MS
"""
import random
import sys

ROUTES = ["/", "/login", "/logout", "/search", "/api/v1/items", "/api/v1/items/{n}", "/api/v1/cart",
          "/api/v1/orders", "/api/v1/orders/{n}", "/static/app.js", "/static/site.css", "/health", "/admin"]
ROUTE_WEIGHTS = [8, 3, 2, 10, 14, 20, 6, 5, 7, 12, 8, 3, 1]
STATUSES = [200, 301, 304, 400, 401, 404, 500, 503]
STATUS_WEIGHTS = [70, 3, 8, 3, 3, 8, 3, 2]
METHODS = ["GET", "GET", "GET", "GET", "GET", "GET", "GET", "GET", "POST", "POST", "PUT", "DELETE"]


def main(path, lines):
    rng = random.Random(7)
    written = 0
    with open(path, "w", newline="\n") as out:
        while written < lines:
            batch = min(100_000, lines - written)
            routes = rng.choices(ROUTES, ROUTE_WEIGHTS, k=batch)
            statuses = rng.choices(STATUSES, STATUS_WEIGHTS, k=batch)
            rows = []
            for i in range(batch):
                second = (written + i) // 14 % 86_400
                route = routes[i].replace("{n}", str(rng.randrange(1, 5000)))
                if rng.random() < 0.0005:
                    rows.append("garbled line without the expected fields")
                    continue
                rows.append(
                    f"2026-10-24T{second // 3600:02d}:{second // 60 % 60:02d}:{second % 60:02d}Z "
                    f"10.{rng.randrange(256)}.{rng.randrange(256)}.{rng.randrange(1, 255)} "
                    f"{rng.choice(METHODS)} {route} {statuses[i]} {rng.randrange(200, 90_000)} "
                    f"{int(rng.expovariate(1 / 140))}"
                )
            out.write("\n".join(rows) + "\n")
            written += batch
    return written


if __name__ == "__main__":
    n = main(sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 1_200_000)
    print(f"wrote {n:,} lines to {sys.argv[1]}")
../venv/Scripts/python.exe gen_logs.py access.log 1200000
wrote 1,200,000 lines to access.log

11b. The analyzer

Save this as logstats.py. Read the four functions in order:

  • split_offsets picks cut points in the file and moves each one forward to the start of the next line.
  • parse_range runs in a worker: it opens the file, reads its byte range, parses each line with a regular expression that is compiled once per interpreter, and returns plain dictionaries.
  • merge adds the partial results together in the main interpreter.
  • analyze ties them together and works with any executor, or with none for the serial baseline.
"""logstats.py: parallel access-log statistics.

Worker side: parse_range(path, start, end).  Main side: split_offsets, merge, analyze.
Every function that crosses to a worker takes and returns only plain values.
"""
import os
import re
from collections import Counter

# TIMESTAMP IP METHOD PATH STATUS BYTES LATENCY_MS.  Compiled once per interpreter, when the module is imported.
LINE = re.compile(rb"(\S+) (\S+) (\S+) (\S+) (\d{3}) (\d+) (\d+)")


def parse_range(path, start, end):
    """Parse every line that starts in [start, end). Both offsets must sit on line starts."""
    with open(path, "rb") as f:
        f.seek(start)
        data = f.read(end - start)
    status, route = Counter(), Counter()
    good = bad = total_bytes = total_ms = slow = 0
    for line in data.split(b"\n"):
        if not line:
            continue
        m = LINE.fullmatch(line)
        if m is None:
            bad += 1
            continue
        good += 1
        status[m[5]] += 1
        parts = m[4].split(b"/", 2)
        route[parts[1] if len(parts) > 1 else b""] += 1
        total_bytes += int(m[6])
        ms = int(m[7])
        total_ms += ms
        if ms >= 500:
            slow += 1
    return {
        "lines": good, "bad": bad, "bytes": total_bytes, "ms": total_ms, "slow": slow,
        "status": {k.decode(): v for k, v in status.items()},
        "route": {(k.decode() or "(root)"): v for k, v in route.items()},
    }


def split_offsets(path, pieces):
    """Return up to `pieces` (start, end) byte ranges that begin and end on line starts."""
    size = os.path.getsize(path)
    cuts = [0]
    with open(path, "rb") as f:
        for i in range(1, pieces):
            f.seek(size * i // pieces)
            f.readline()  # finish the line we landed in, so the cut moves to the next line start
            cuts.append(min(f.tell(), size))
    cuts.append(size)
    return [(a, b) for a, b in zip(cuts, cuts[1:]) if b > a]


def merge(parts):
    total = {"lines": 0, "bad": 0, "bytes": 0, "ms": 0, "slow": 0, "status": Counter(), "route": Counter()}
    for part in parts:
        for key in ("lines", "bad", "bytes", "ms", "slow"):
            total[key] += part[key]
        total["status"].update(part["status"])
        total["route"].update(part["route"])
    total["status"] = dict(sorted(total["status"].items()))
    total["route"] = dict(sorted(total["route"].items(), key=lambda kv: (-kv[1], kv[0])))
    return total


def analyze(path, pieces, pool=None):
    """Statistics for the whole file. `pool` is any executor; None runs serially in this thread."""
    ranges = split_offsets(path, pieces)
    mapper = map if pool is None else pool.map
    return merge(mapper(parse_range, [path] * len(ranges), [a for a, _ in ranges], [b for _, b in ranges]))

11c. Compare the backends

Save this as step11_logs.py. It runs the analyzer serially and then on each kind of pool, and checks every result against the serial one.

"""step11_logs.py: the finished tool on a generated log, serial versus the three pools."""
import os

import labkit
import logstats

LOG = "access.log"

if __name__ == "__main__":
    labkit.banner()
    print(f"{LOG}: {os.path.getsize(LOG) / 2**20:.1f} MiB")
    serial, expected = labkit.best_of(lambda: logstats.analyze(LOG, 1), repeat=2)
    print(f"serial                     {serial:6.2f} s   {expected['lines']:,} lines, {expected['bad']} unparsable")
    for kind in ("threads", "interpreters", "processes"):
        for n in (4, 14):
            pool, _ = labkit.start_pool(kind, n)
            with pool:
                secs, got = labkit.best_of(lambda: logstats.analyze(LOG, n * 4, pool), repeat=2)
            print(f"{n:>2} {kind:<22} {secs:6.2f} s   speedup {serial / secs:5.2f}x   same result as serial: {got == expected}")
    print("\nstatus codes:", expected["status"])
    print("top routes:  ", dict(list(expected["route"].items())[:4]))
    print(f"mean latency: {expected['ms'] / expected['lines']:.1f} ms, requests of 500 ms or more: {expected['slow']:,}")
../venv/Scripts/python.exe step11_logs.py
Python 3.15.1, GIL enabled: True, logical CPUs: 14
access.log: 75.5 MiB
serial                       0.87 s   1,199,365 lines, 635 unparsable
 4 threads                  0.84 s   speedup  1.04x   same result as serial: True
14 threads                  1.31 s   speedup  0.67x   same result as serial: True
 4 interpreters             0.29 s   speedup  3.04x   same result as serial: True
14 interpreters             0.09 s   speedup  9.41x   same result as serial: True
 4 processes                0.24 s   speedup  3.68x   same result as serial: True
14 processes                0.09 s   speedup 10.10x   same result as serial: True

status codes: {'200': 840295, '301': 35788, '304': 95467, '400': 36388, '401': 35955, '404': 95615, '500': 35811, '503': 24046}
top routes:   {'api': 628925, 'static': 243463, 'search': 121500, '(root)': 96712}
mean latency: 139.4 ms, requests of 500 ms or more: 33,899

All six parallel runs returned a result identical to the serial one, which is the check that matters most. With 14 workers, interpreters ran 9.41x faster than the serial pass and processes 10.10x, while threads ran at 0.67x. This job allocates a match object, several small byte strings and counter entries for every line, and it still scaled well. The earlier free-threaded Python tutorial measured only 2.87x for an allocation-heavy job on the same machine, but those were different jobs, so take it as a reason to test your own.

Step 12: Meet the newline trap

The one design decision that is easy to get wrong is where to cut the file. Cutting at raw byte offsets looks natural, because it makes equal pieces. This script shows what it costs.

"""step12_naive_split.py: what happens if you cut the file at raw byte offsets."""
import os

import logstats

LOG = "access.log"


def naive_ranges(path, pieces):
    size = os.path.getsize(path)
    cuts = [size * i // pieces for i in range(pieces + 1)]
    return list(zip(cuts, cuts[1:]))


def totals(ranges):
    return logstats.merge(logstats.parse_range(LOG, a, b) for a, b in ranges)


def what_each_cut_did(ranges):
    """Classify the line each raw cut lands in: on a line start, two broken halves, or a prefix that still parses."""
    data = open(LOG, "rb").read()
    kinds = {"on a line start": 0, "two unparsable halves": 0, "prefix parses with a shortened last number": 0,
             "suffix parses, cut fell in the timestamp": 0}
    for _, cut in ranges[:-1]:
        if data[cut - 1:cut] == b"\n":
            kinds["on a line start"] += 1
            continue
        line_start = data.rfind(b"\n", 0, cut) + 1
        line_end = data.find(b"\n", cut)
        prefix, suffix = data[line_start:cut], data[cut:line_end]
        parses = (logstats.LINE.fullmatch(prefix) is not None, logstats.LINE.fullmatch(suffix) is not None)
        if parses == (False, False):
            kinds["two unparsable halves"] += 1
        elif parses == (True, False):
            kinds["prefix parses with a shortened last number"] += 1
        else:
            kinds["suffix parses, cut fell in the timestamp"] += 1
    return kinds


if __name__ == "__main__":
    expected = logstats.analyze(LOG, 1)
    print(f"serial: {expected['lines']:,} good lines, {expected['bad']} unparsable, "
          f"{expected['bytes']:,} bytes, {expected['ms']:,} ms of latency")
    for pieces in (4, 56):
        ranges = naive_ranges(LOG, pieces)
        got = totals(ranges)
        print(f"{pieces:>3} raw byte ranges: good lines {got['lines'] - expected['lines']:+d}, "
              f"unparsable {got['bad'] - expected['bad']:+d}, bytes {got['bytes'] - expected['bytes']:+,}, "
              f"latency {got['ms'] - expected['ms']:+,} ms")
        print(f"    the {pieces - 1} cuts: {what_each_cut_did(ranges)}")
    got = totals(logstats.split_offsets(LOG, 56))
    print(f" 56 line-aligned ranges: identical to serial: {got == expected}")
../venv/Scripts/python.exe step12_naive_split.py
serial: 1,199,365 good lines, 635 unparsable, 54,096,120,652 bytes, 167,231,754 ms of latency
  4 raw byte ranges: good lines -2, unparsable +5, bytes -100,055, latency -258 ms
    the 3 cuts: {'on a line start': 0, 'two unparsable halves': 2, 'prefix parses with a shortened last number': 0, 'suffix parses, cut fell in the timestamp': 1}
 56 raw byte ranges: good lines -41, unparsable +93, bytes -1,796,587, latency -3,955 ms
    the 55 cuts: {'on a line start': 0, 'two unparsable halves': 41, 'prefix parses with a shortened last number': 5, 'suffix parses, cut fell in the timestamp': 9}
 56 line-aligned ranges: identical to serial: True

With 56 raw cuts, 41 good lines disappeared and 93 extra unparsable fragments appeared. Of the 55 cuts, 41 split a line into two unparsable halves, 9 fell in the timestamp so the rest of the line still parsed, and 5 left a prefix that still parsed with a shortened last number, which is the dangerous kind: nothing flags it. The totals are off by 0.003 percent (by my arithmetic: 41 of 1,199,365 lines, and 1,796,587 of 54,096,120,652 bytes), small enough that nobody would notice and wrong all the same. Line-aligned ranges from split_offsets reproduce the serial answer exactly.

Step 13: Test it, then try to break it

A parallel program that gives slightly wrong answers is the worst kind, so make the tests compare every pool with the serial result. Save this as test_logstats.py. The interpreter case is skipped on Python 3.13 and older, so the same file works in a CI job that still tests older versions.

"""test_logstats.py: the parallel result must equal the serial result, on every pool."""
import random
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

import pytest

import logstats

try:
    from concurrent.futures import InterpreterPoolExecutor
except ImportError:  # Python 3.13 and older
    InterpreterPoolExecutor = None

POOLS = [
    ThreadPoolExecutor,
    ProcessPoolExecutor,
    pytest.param(InterpreterPoolExecutor, marks=pytest.mark.skipif(
        InterpreterPoolExecutor is None, reason="InterpreterPoolExecutor needs Python 3.14 or newer")),
]


def make_lines(count, seed=1):
    rng = random.Random(seed)
    routes = ["/", "/login", "/api/v1/items/7", "/static/app.js", "/search"]
    return [
        f"2026-10-24T00:00:{i % 60:02d}Z 10.0.0.{rng.randrange(1, 255)} GET {rng.choice(routes)} "
        f"{rng.choice([200, 200, 200, 404, 500])} {rng.randrange(100, 9000)} {rng.randrange(1, 900)}"
        for i in range(count)
    ]


@pytest.fixture
def log(tmp_path):
    path = tmp_path / "access.log"
    path.write_text("\n".join(make_lines(300)) + "\n", newline="\n")
    return str(path)


def test_ranges_are_contiguous_and_start_on_line_starts(log):
    data = open(log, "rb").read()
    for pieces in range(1, 60):
        ranges = logstats.split_offsets(log, pieces)
        assert ranges[0][0] == 0 and ranges[-1][1] == len(data)
        assert all(a[1] == b[0] for a, b in zip(ranges, ranges[1:]))
        assert all(data[start - 1:start] == b"\n" for start, _ in ranges[1:])


@pytest.mark.parametrize("pieces", [1, 2, 3, 7, 50, 500])
def test_any_piece_count_gives_the_serial_totals(log, pieces):
    assert logstats.analyze(log, pieces) == logstats.analyze(log, 1)
    assert logstats.analyze(log, pieces)["lines"] == 300


def test_a_cut_that_lands_exactly_on_a_line_start_loses_nothing(tmp_path):
    line = "2026-10-24T00:00:00Z 10.0.0.1 GET /x 200 100 5"  # every line has the same width
    path = tmp_path / "even.log"
    path.write_text("\n".join([line] * 12) + "\n", newline="\n")
    for pieces in (2, 3, 4, 6, 12):  # each of these cuts at a multiple of whole lines
        assert logstats.analyze(str(path), pieces)["lines"] == 12


def test_last_line_without_a_newline_is_counted(tmp_path):
    path = tmp_path / "a.log"
    path.write_text("\n".join(make_lines(10)), newline="\n")
    assert logstats.analyze(str(path), 4)["lines"] == 10


def test_empty_file(tmp_path):
    path = tmp_path / "empty.log"
    path.write_bytes(b"")
    assert logstats.analyze(str(path), 4)["lines"] == 0


def test_unparsable_lines_are_counted_not_fatal(tmp_path):
    path = tmp_path / "b.log"
    lines = make_lines(20)
    lines[5] = "garbled"
    lines[13] = "also garbled"
    path.write_text("\n".join(lines) + "\n", newline="\n")
    got = logstats.analyze(str(path), 6)
    assert (got["lines"], got["bad"]) == (18, 2)


@pytest.mark.parametrize("pool_cls", POOLS)
def test_pools_agree_with_serial(log, pool_cls):
    with pool_cls(3) as pool:
        assert logstats.analyze(log, 12, pool) == logstats.analyze(log, 1)


@pytest.mark.parametrize("pool_cls", POOLS)
def test_a_missing_file_raises_the_original_exception_type(tmp_path, pool_cls):
    with pool_cls(2) as pool:
        with pytest.raises(FileNotFoundError):
            logstats.analyze(str(tmp_path / "nope.log"), 1, pool)  # split_offsets runs in the caller
        with pytest.raises(FileNotFoundError):
            list(pool.map(logstats.parse_range, [str(tmp_path / "nope.log")], [0], [10]))

Then save mutations.py. It copies logstats.py, breaks it in six different ways one at a time, and counts how many tests notice each break.

"""mutations.py: break logstats.py five ways and count the tests that notice."""
import os
import pathlib
import re
import shutil
import subprocess
import sys
import tempfile

HERE = pathlib.Path(__file__).resolve().parent
SOURCE = (HERE / "logstats.py").read_text(encoding="utf-8")

MUTANTS = {
    "cut at raw byte offsets": (
        "            f.readline()  # finish the line we landed in, so the cut moves to the next line start\n",
        "            pass\n",
    ),
    "cut one byte too far": ("cuts.append(min(f.tell(), size))", "cuts.append(min(f.tell() + 1, size))"),
    "last range lost": ("    cuts.append(size)\n", "    cuts.append(cuts[-1])\n"),
    "do not count unparsable lines": ("            bad += 1\n            continue", "            continue"),
    "merge skips the last part": ("    for part in parts:", "    for part in list(parts)[:-1]:"),
    "route counts overwritten, not added": (
        'total["route"].update(part["route"])', 'total["route"] = Counter(part["route"])'),
}


def failing_tests(workdir):
    env = {**os.environ, "COLUMNS": "170", "PYTHONDONTWRITEBYTECODE": "1"}
    done = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "test_logstats.py"],
                          cwd=workdir, capture_output=True, text=True, env=env)
    summary = done.stdout.strip().splitlines()[-1]
    failed = re.search(r"(\d+) failed", summary)
    return int(failed[1]) if failed else 0, summary


if __name__ == "__main__":
    caught = 0
    for name, (old, new) in MUTANTS.items():
        assert SOURCE.count(old) == 1, name
        with tempfile.TemporaryDirectory() as tmp:
            shutil.copy(HERE / "test_logstats.py", tmp)
            pathlib.Path(tmp, "logstats.py").write_text(SOURCE.replace(old, new), encoding="utf-8")
            failed, summary = failing_tests(tmp)
        caught += failed > 0
        print(f"{name:<38} {'caught' if failed else 'SURVIVED':<9} {summary}")
    print(f"{caught} of {len(MUTANTS)} mutants caught")
../venv/Scripts/python.exe -m pytest -q test_logstats.py
../venv/Scripts/python.exe mutations.py
................. [100%]
17 passed in 0.46s
cut at raw byte offsets                caught    11 failed, 6 passed in 0.54s
cut one byte too far                   caught    10 failed, 7 passed in 0.49s
last range lost                        caught    13 failed, 4 passed in 0.49s
do not count unparsable lines          caught    1 failed, 16 passed in 0.50s
merge skips the last part              caught    12 failed, 5 passed in 0.48s
route counts overwritten, not added    caught    8 failed, 9 passed in 0.50s
6 of 6 mutants caught

All 17 tests pass, and all six deliberate bugs were caught. One lesson from building this: my first version of the cut logic had an extra minus 1 in the seek position that I believed protected against cuts landing exactly on a line start. A mutant that removed it survived, and when I thought it through, the code was correct either way, because the cuts are computed in the main interpreter and only move forward. I deleted the pointless term and kept test_a_cut_that_lands_exactly_on_a_line_start_loses_nothing as an edge-case guard. A surviving mutant is a prompt to understand your code, not always a missing test.

Step 14: Decide between threads, interpreters, processes and free-threaded Python

Here is everything measured above in one table, for CPU-bound Python on a standard build, followed by the cases where each choice wins.

Question Threads Interpreters Processes
Speedup at 14 workers (Step 4) 0.75x 10.48x 10.45x
Start 14 workers (Step 5) 0.004 s 0.056 s 0.210 s
Extra private memory for 14 workers 0.5 MiB 94.4 MiB 182.4 MiB
Tiny tasks per second 218,355 44,284 10,820
Data between workers (Steps 6 and 7) everything shared copied, except memoryview and Queue pickled copies
Packages with compiled code (Step 10) all 2 of 25 loaded all
os._exit in a worker (Step 9) ends the program ends the program BrokenProcessPool, program survives
  • Use interpreters when the hot code is pure Python or uses only modules that load in a subinterpreter, tasks are chunky, workers need to start often or memory is tight, and the work needs bytes or small objects, not big graphs of objects.
  • Use processes when your dependencies refuse to load in a subinterpreter, when a worker may crash or call os._exit, or when you need an actual security boundary. The documentation says interpreters are not one.
  • Use threads or asyncio for work that mostly waits.
  • Consider a free-threaded build if you want ordinary threads that really run in parallel and share objects. Our free-threaded tutorial measured 9.74x for a pure-Python job there, and showed that the price is that races which the GIL used to hide become real. Interpreters take the opposite trade: nothing is shared unless you share it on purpose, which is also what the What’s New page means by “Think of multiple interpreters as threads but with opt-in sharing.”

Whichever you choose, find the hot function first with a profiler (the Tachyon profiler tutorial shows how), because parallelizing the wrong function is the most expensive mistake on the list.

Troubleshooting

ImportError: cannot import name 'InterpreterPoolExecutor'

You are on Python 3.13 or older. Run python --version and use a 3.14 or newer interpreter, as in the setup section.

NotShareableError when you submit or return something

The object cannot cross. Look it up in the Step 6 matrix: closures, generators, locks and open files never can. Send a path instead of an open file, and rebuild the object inside the worker.

ImportError: module X does not support loading in subinterpreters

The compiled module has not opted in. Use probe_import.py to confirm, then move that work to the main interpreter or a process pool (Step 10).

Lines printed once per worker, or a NameError for a name you know exists

The worker re-ran your script’s top level without the main guard (Step 6c). Move shared code into a module and keep the top level light.

Slower than the serial loop

Check the chunk size against the per-task cost (Step 5), check that you are not sending large lists or dictionaries (Step 7), and check the serial column drift from Step 4 by timing your serial baseline more than once.

Verify the whole thing end to end

When every file is saved, this block runs all the experiments, the tests and the mutation check in order. Every script should exit without a traceback, the sieve check should say True, the log analyzer results should all say same result as serial: True, pytest should report 17 passed, and the mutation script should report 6 of 6 mutants caught. Timings will differ on your machine and the speedup ratios should keep their shape.

for s in step01_serial step02_threads step03_interpreters step04_scaling step05_startup \
         step06_types step07_data step08_state step09_failures step10_main_rerun; do
  ../venv/Scripts/python.exe $s.py
done
../venv/Scripts/python.exe survey_stdlib.py
../venv/Scripts/python.exe gen_logs.py access.log 1200000
../venv/Scripts/python.exe step11_logs.py
../venv/Scripts/python.exe step12_naive_split.py
../venv/Scripts/python.exe -m pytest -q test_logstats.py
../venv/Scripts/python.exe mutations.py

Where to go next

  • Read PEP 734 (multiple interpreters in the standard library) and PEP 684 (a per-interpreter GIL) for the design reasoning.
  • Try concurrent.interpreters.create_queue() and Interpreter.call_in_thread() from the module documentation to build a pipeline of long-lived workers that pass messages.
  • Combine the pool with a bulkhead (see our bulkhead tutorial) so one slow job class cannot use every worker.
  • Re-run the Step 10 survey whenever you upgrade a dependency, because support will grow.

Sources

Tags:

ConcurrencyParallel ComputingPerformance EngineeringPythonPython 3.15Subinterpreters

Share

No Comment! Be the first one.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts