How to Set gRPC Deadlines and Retries in Python, Pass Them Down a Call Chain, and Stop Retries From Charging a Card Twice
Learn to give every gRPC call in Python a deadline, make a server stop work for callers who have left, pass the remaining time down a call chain, retry only safe failures with a service-config...
Every gRPC call you make is a small bet that another program will answer. This tutorial teaches you how to place that bet safely in Python. You will give every call a deadline (the moment you stop waiting), make servers stop working for callers who have left, pass the remaining time down a chain of services, retry only the failures that are safe to retry, and keep a retried payment from charging a customer twice. Each idea is shown on a small working lab that you can run on your own computer, with the real output printed next to the code.
Table Of Content
- What you will build
- Before you start
- What you need
- A few words we will use
- The files at a glance
- Step 1: Set up the lab and make your first call
- Create an environment and install the packages
- Describe the three services in a .proto file
- Generate the Python code
- Add two helper files
- Make the first call
- Step 2: See the default, then set a deadline
- What a deadline means on the wire
- How to choose the number
- Step 3: Make the server stop when the caller leaves
- The check that fixes it
- Step 4: Pass the deadline down a call chain
- Two small helpers
- An Orders service with four ways to call Pricing
- The trap: what “no deadline” looks like
- Why a reserve and a floor
- Step 5: Read status codes and stop leaking details
- What each line teaches
- Which codes should you retry?
- Close two holes with a server interceptor
- Step 6: Retry the right failures with a service-config policy
- A policy, and a helper that builds it
- Watch the policy work
- Three traps in policy files
- Server push-back
- What a policy cannot do
- Step 7: Keep retries from multiplying
- Retry throttling
- A rule of thumb
- Step 8: Make retried writes safe with an idempotency key
- A ledger that can remember keys
- Compare the two behaviours
- What a production version still needs
- Step 9: Make the safe path the default
- Step 10: If you use asyncio, the server behaves differently
- Step 11: Prove it with tests and planted bugs
- Common mistakes and how to spot them
- Where to go next
- Sources
gRPC is a way for one program to call a function that lives in another program, usually on another machine. Both sides agree on the shape of the calls in a .proto file, and gRPC does the networking. If you have never used it, you will still be fine, because we build everything from scratch. (For the background on how those .proto files can change safely, see our tutorial on evolving a Protobuf schema without breaking old clients.)
Two defaults make a gRPC system fragile, and both are easy to miss. The gRPC deadlines guide says: “By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever.” The retry guide says: “Retries are enabled by default, but there is no default retry policy.” In plain words: out of the box, one slow dependency can hold every caller for as long as it likes, and a flaky one is never retried. Once you add retries you meet a third problem, because a retry that repeats a write can charge a customer twice. This tutorial walks through all three, one measurable step at a time.
What you will build
You will build a tiny shop made of three services that all run inside one Python process: Pricing (looks up a price), Orders (calls Pricing on behalf of a customer) and Payments (writes a charge to a ledger). The lab can break each service on purpose: it can be slow, flaky, throw an unexpected exception, or lose its reply. For every failure you will first watch the client suffer, then apply the fix and measure again. At the end you will have:
- a deadline on every call, and a server that stops working when its caller has gone;
- a deadline that is passed down from Orders to Pricing, including the case where the caller set none;
- a retry policy that is built and checked by a helper, with retry throttling and server push-back;
- idempotency keys that make a retried charge safe;
- a client interceptor and a server interceptor that make the safe behaviour the default;
- 22 tests, and a check that plants 15 bugs on purpose to prove the tests notice them.
Before you start
What you need
You need Python 3.10 or newer (the PyPI page for grpcio 1.85.1 lists 3.10 as the minimum; I ran every step on Python 3.13.14), a terminal, and a folder where you can create about two dozen small files. I used Windows 11, so a few timings, such as how long a refused connection takes to fail, are specific to that machine, and I point those out where they matter. You should be comfortable with Python functions, classes and try and except. You do not need any gRPC experience, Docker, or a network: every server listens on 127.0.0.1 inside the same process as the client. Plan on about 40 minutes of reading and running.
The versions used are grpcio 1.85.1 (published on October 30, 2026), grpcio-tools 1.85.1 (it contains the compiler that turns a .proto file into Python) and pytest 9.2.0. Your timings will differ from mine by a few hundredths of a second, but the status names, counts and attempt numbers should match exactly.
A few words we will use
RPC means remote procedure call: a function call that runs in another process. A channel is the client’s connection to a server address. A stub is the generated client object with one method per RPC, and a servicer is the generated base class that a server extends.
A deadline is the point in time after which the caller stops waiting. The Python API asks for a timeout instead, which is a number of seconds from now. The two words describe the same limit, because, as the gRPC docs explain, a timeout can be converted to a deadline by adding it to the current time when the call starts.
A status code is how every call ends: OK, or one of sixteen error codes such as UNAVAILABLE or DEADLINE_EXCEEDED. Metadata is a set of key and value pairs sent alongside a call, much like HTTP headers. An interceptor is code that wraps every call on the client or the server. An operation is idempotent when doing it twice has the same effect as doing it once, which is the property that decides whether a retry is safe.
The files at a glance
Put every file in the same folder, next to each other. Each Python block below starts with a comment holding its file name, so you can copy it straight into a file with that name.
| File | What it holds | Step |
|---|---|---|
shop.proto |
the contract for the three services | 1 |
labkit.py, pricing.py |
helpers, and a pretend Pricing service with knobs for slow, flaky and broken | 1 |
deadlines.py, orders.py |
reading the caller’s deadline and passing it down to the next call | 4 |
server_guard.py |
a server interceptor for deadlines and error hygiene | 5 |
retries.py |
a retry-policy builder that refuses values gRPC would clamp or reject | 6 |
payments.py |
a Payments service with a ledger and idempotency keys | 8 |
client_guard.py |
a client interceptor that gives every call a deadline | 9 |
step01_first_call.py to step10_aio.py |
one experiment per step | 1 to 10 |
test_lab.py, mutants.py |
the tests, and the planted-bug check | 11 |
Step 1: Set up the lab and make your first call
We start by building the smallest working system: a Pricing server and a client that asks it for a price. Everything later builds on these pieces, so it is worth going slowly here.
Create an environment and install the packages
A virtual environment keeps this lab’s packages separate from the rest of your machine. Run the commands for your system, then the pip line.
# Windows (PowerShell)
python -m venv venv
venv\Scripts\Activate.ps1
# macOS or Linux
python3 -m venv venv
source venv/bin/activate
# both
python -m pip install grpcio==1.85.1 grpcio-tools==1.85.1 pytest==9.2.0
Check it worked by running python -c "import grpc; print(grpc.__version__)". It should print 1.85.1.
Describe the three services in a .proto file
gRPC starts from a contract. The file below declares three services and the messages they exchange. Orders.Place reuses the quote messages because, in our shop, placing an order starts by asking Pricing for a quote. Note the line package shop.v1;: it makes the full name of the first service shop.v1.Pricing, and that exact name matters in Step 6.
syntax = "proto3";
package shop.v1;
service Pricing {
rpc Quote (QuoteRequest) returns (QuoteReply);
}
service Orders {
rpc Place (QuoteRequest) returns (QuoteReply);
}
service Payments {
rpc Charge (ChargeRequest) returns (ChargeReply);
}
message QuoteRequest {
string item = 1;
int32 qty = 2;
}
message QuoteReply {
int64 cents = 1;
}
message ChargeRequest {
string order_id = 1;
int64 cents = 2;
}
message ChargeReply {
string charge_id = 1;
bool replayed = 2;
}
Generate the Python code
The compiler reads the contract and writes two Python files: shop_pb2.py holds the message classes, and shop_pb2_grpc.py holds the Stub classes for clients and the Servicer base classes for servers. You never edit them by hand.
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. shop.proto
Check it worked: your folder should now contain both generated files, and python -c "import shop_pb2_grpc" should print nothing and exit cleanly.
Add two helper files
labkit.py has three small helpers that every experiment shares: serve starts a server on a free port inside this process, closed_address returns an address where nothing is listening, and call_outcome runs a call and reports either OK or the status name, together with the seconds it took. The server runs its handlers in a pool of 16 worker threads, so each call that is being served occupies one thread until its handler returns.
# labkit.py
"""Small helpers every experiment shares: start a gRPC server in this process, find a closed port, run a call."""
import socket
import time
from concurrent import futures
import grpc
def serve(add_fn, servicer, *, workers=16, interceptors=(), **server_kwargs):
"""Start a server on a free 127.0.0.1 port. Returns (server, address)."""
server = grpc.server(
futures.ThreadPoolExecutor(max_workers=workers),
interceptors=interceptors,
**server_kwargs,
)
add_fn(servicer, server)
port = server.add_insecure_port("127.0.0.1:0")
server.start()
return server, f"127.0.0.1:{port}"
def closed_address():
"""An address where nothing is listening."""
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
return f"127.0.0.1:{s.getsockname()[1]}"
def call_outcome(fn, *args, **kwargs):
"""Run an RPC and return (text, seconds), where text is 'OK 120 cents' or the status name."""
start = time.monotonic()
try:
reply = fn(*args, **kwargs)
text = f"OK {reply.cents} cents"
except grpc.RpcError as err:
text = err.code().name
return text, time.monotonic() - start
pricing.py is the heart of the lab. It is a pretend Pricing service with knobs, so that we can make it slow, flaky or broken on demand. Each knob is used in a later step, and the table after the code says where.
# pricing.py
"""A pretend Pricing service with knobs: it can be slow, flaky or broken on demand, and it keeps notes on every call."""
import threading
import time
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
PRICES = {"apple": 120, "pear": 150}
class Pricing(pg.PricingServicer):
def __init__(
self,
steps=0, # pretend work: this many steps ...
step_seconds=0.4, # ... of this many seconds each
check_cancel=True, # stop between steps when the caller has gone away
fail_first=0, # the first N calls fail ...
fail_code=grpc.StatusCode.UNAVAILABLE, # ... with this status
explode=False, # raise an unexpected exception
pushback_ms=None, # ask clients to wait (or, with -1, to stop retrying) after a failure
):
self.steps, self.step_seconds, self.check_cancel = steps, step_seconds, check_cancel
self.fail_first, self.fail_code, self.explode, self.pushback_ms = fail_first, fail_code, explode, pushback_ms
self.lock = threading.Lock()
self.calls = [] # one dict per call that reached the handler
self.steps_run = 0 # work steps finished, across all calls
def Quote(self, request, context):
meta = dict(context.invocation_metadata())
with self.lock:
number = len(self.calls)
self.calls.append(
{
"at": time.monotonic(),
"previous_attempts": meta.get("grpc-previous-rpc-attempts"),
"raw_time_remaining": context.time_remaining(),
}
)
if self.explode:
raise ValueError("db password is hunter2")
if number < self.fail_first:
if self.pushback_ms is not None:
context.set_trailing_metadata((("grpc-retry-pushback-ms", str(self.pushback_ms)),))
context.abort(self.fail_code, "try again later")
if request.item not in PRICES:
context.abort(grpc.StatusCode.NOT_FOUND, f"no price for {request.item!r}")
for _ in range(self.steps):
if self.check_cancel and not context.is_active():
return pb.QuoteReply()
time.sleep(self.step_seconds)
with self.lock:
self.steps_run += 1
return pb.QuoteReply(cents=PRICES[request.item] * request.qty)
Read the Quote method once, because every handler in this tutorial has the same shape. gRPC calls it with the request message and a context object that stands for this one call. The handler can use the context to read the caller’s metadata and deadline, to ask whether the caller is still there, and to end the call with a status through context.abort(...). Whatever the handler returns becomes the reply.
| Knob | What it does | Used in |
|---|---|---|
steps, step_seconds |
pretend work: this many steps, each taking this many seconds | Steps 2 to 4, 9, 10 |
check_cancel |
whether the handler stops between steps when its caller has gone | Steps 3 and 4 |
fail_first, fail_code |
the first N calls fail with this status | Steps 6 and 7 |
explode |
raise an unexpected exception whose text contains a made-up secret | Step 5 |
pushback_ms |
ask clients to wait, or to stop retrying, after a failure | Step 6 |
Make the first call
This script starts a Pricing server, asks for three apples, and then asks for an item that does not exist. The second call fails on purpose so that you see what an error looks like to the caller.
# step01_first_call.py
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import serve
from pricing import Pricing
server, address = serve(pg.add_PricingServicer_to_server, Pricing())
stub = pg.PricingStub(grpc.insecure_channel(address))
reply = stub.Quote(pb.QuoteRequest(item="apple", qty=3))
print("apple x3 ->", reply.cents, "cents")
try:
stub.Quote(pb.QuoteRequest(item="durian", qty=1))
except grpc.RpcError as err:
print("durian ->", err.code().name, repr(err.details()))
server.stop(0)
Run it with python step01_first_call.py. You should see:
apple x3 -> 360 cents
durian -> NOT_FOUND "no price for 'durian'"
Two things are worth noticing. The first call returned a reply object, and its cents field holds 360 (three apples at 120 cents). The second call raised a grpc.RpcError. Every failed call raises this one exception type, and you read the outcome with err.code(), a grpc.StatusCode value, and err.details(), a human-readable message. Here NOT_FOUND and its message were chosen by our own handler with context.abort(...), which is how a server reports a failure on purpose.
Step 2: See the default, then set a deadline
Now we meet the first dangerous default. This script starts a Pricing server that needs 3.2 seconds to answer (8 steps of 0.4 seconds each) and calls it twice: once with no timeout, and once with timeout=1.0.
# step02_deadline.py
import time
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import serve
from pricing import Pricing
# 8 steps of 0.4 s each: this server needs 3.2 seconds to answer.
server, address = serve(pg.add_PricingServicer_to_server, Pricing(steps=8))
stub = pg.PricingStub(grpc.insecure_channel(address))
request = pb.QuoteRequest(item="apple", qty=1)
start = time.monotonic()
reply = stub.Quote(request) # no timeout given
print(f"no timeout: OK {reply.cents} cents after {time.monotonic() - start:.1f} s")
start = time.monotonic()
try:
stub.Quote(request, timeout=1.0)
except grpc.RpcError as err:
print(f"timeout=1.0: {err.code().name} {err.details()!r} after {time.monotonic() - start:.1f} s")
print("exception class:", type(err).__name__, "| is a grpc.RpcError:", isinstance(err, grpc.RpcError))
server.stop(0)
Run it with python step02_deadline.py:
no timeout: OK 120 cents after 3.2 s
timeout=1.0: DEADLINE_EXCEEDED 'Deadline Exceeded' after 1.0 s
exception class: _InactiveRpcError | is a grpc.RpcError: True
The call with no timeout simply waited the full 3.2 seconds. If the server had been stuck instead of slow, the client would have waited until something else broke. With timeout=1.0 the client gave up after 1.0 second and received the status DEADLINE_EXCEEDED. The exception class is _InactiveRpcError, a private subclass, so always catch the public grpc.RpcError instead.
What a deadline means on the wire
The timeout is not only a local timer. The client also tells the server how long it is willing to wait, using an HTTP/2 header called grpc-timeout. The protocol document shows one in its example request as grpc-timeout = 1S, a number with a unit letter. The server sees what is left of that budget through context.time_remaining(), which we use in Step 4.
How to choose the number
The gRPC guide suggests starting with an educated guess based on network latency and server processing time, then validating it with load testing. A practical rule is to look at how long the callee takes at its slow end (for example its 99th percentile), add a margin, and never let the number exceed what your own caller is willing to wait. The deadline covers the whole call, including connecting, and, as Step 6 shows, every retry attempt. The same thinking applies to plain HTTP clients, which we cover in the HTTP timeouts and deadlines tutorial.
Step 3: Make the server stop when the caller leaves
The client gave up after one second. Did the server stop working? Here is what the gRPC deadlines guide says: a server “deals with this situation by automatically cancelling a call (CANCELLED status) once a deadline set by the client has passed. Please note that the server application is responsible for stopping any activity it has spawned to service the RPC.” The cancellation guide adds that the library “in general does not have a mechanism to interrupt the application-provided server handler”. So the question is whether our handler notices. The script below runs the same slow call twice. The first server never checks for a departed caller (check_cancel=False), and the second does.
# step03_zombie_work.py
import time
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import serve
from pricing import Pricing
request = pb.QuoteRequest(item="apple", qty=1)
for check_cancel in (False, True):
pricing = Pricing(steps=8, check_cancel=check_cancel) # 8 steps of 0.4 s
server, address = serve(pg.add_PricingServicer_to_server, pricing)
stub = pg.PricingStub(grpc.insecure_channel(address))
try:
stub.Quote(request, timeout=1.0)
except grpc.RpcError as err:
print(f"check_cancel={check_cancel}: the client got {err.code().name} after 1 second")
time.sleep(3.5) # long enough for a handler that never stops to finish
print(f" steps the server ran for that one call: {pricing.steps_run} of 8")
server.stop(0)
Run it with python step03_zombie_work.py. It takes about 9 seconds:
check_cancel=False: the client got DEADLINE_EXCEEDED after 1 second
steps the server ran for that one call: 8 of 8
check_cancel=True: the client got DEADLINE_EXCEEDED after 1 second
steps the server ran for that one call: 3 of 8
In both cases the client got DEADLINE_EXCEEDED after one second, so from the client’s side the two servers look identical. But the first server kept doing all 8 steps of work for a caller that had left 2.2 seconds earlier. I call that zombie work. A server has a limited number of worker threads (our lab uses 16), and every handler that keeps running for nobody keeps one of them busy. During an incident that is the last thing you want.
The check that fixes it
The fix is in pricing.py, in the loop you copied in Step 1:
for _ in range(self.steps):
if self.check_cancel and not context.is_active():
return pb.QuoteReply()
time.sleep(self.step_seconds)
context.is_active() returns False once the call has ended for any reason: the deadline passed, the client cancelled, or the connection dropped. A long handler should ask this question between steps of work, then stop and release its resources. If your handler needs to clean up (close a file, release a lock), context.add_callback(fn) runs fn when the call ends.
The number 3 is not an accident. Each step takes 0.4 seconds and the deadline is 1.0 second, so the checks happen at 0.0, 0.4 and 0.8 seconds (all before the deadline, so three steps start) and again at 1.2 seconds (after the deadline, so the handler stops). The step size is chosen so that the deadline falls in the middle of a step, which keeps the count stable from run to run.
Step 4: Pass the deadline down a call chain
Real systems are chains: a client calls Orders, and Orders calls Pricing. If Orders ignores the deadline the client set, Pricing has no idea anyone stopped waiting. The gRPC guide describes the ideal: a server that also acts as a client “would want to honor the deadline set by the original client”, and it notes that some implementations propagate automatically: it “needs to be explicitly enabled (e.g. C++)” in some languages and “is enabled by default (e.g. Java and Go)” in others. The guide does not list Python, so we test it.
Two small helpers
The guide also explains why the number we pass on should be a timeout and not a clock time: gRPC “converts the deadline to a timeout from which the already elapsed time is already deducted”, which shields the system from clocks that disagree. So the helper in deadlines.py reads what is left of the caller’s budget and subtracts a small reserve, so that Orders still has time to turn Pricing’s failure into its own reply before the caller gives up. Read the comment at the top of the file now; the trap section below explains it.
# deadlines.py
"""Read the caller's deadline on the server side and pass what is left to the next call."""
# Python reports "no deadline" as a huge number of seconds, not as None (see step 4).
NO_DEADLINE = 365 * 24 * 3600
def time_left(context):
"""Seconds left before the caller gives up, or None when the caller set no deadline."""
left = context.time_remaining()
if left is None or left > NO_DEADLINE:
return None
return left
def child_timeout(context, reserve=0.05, fallback=None):
"""Timeout for an outgoing call made while serving `context`: what is left, minus a reserve for our own reply."""
left = time_left(context)
if left is None:
return fallback
return max(left - reserve, 0.001)
An Orders service with four ways to call Pricing
orders.py implements Orders.Place in four modes so that we can compare them. Mode none is the mistake: it calls Pricing without any deadline. Mode propagate passes on what is left. Mode cancel sets no deadline but cancels the inner call when the caller goes away, using context.add_callback. Mode raw is the idiom that looks obvious, timeout=context.time_remaining(), and it hides a trap that the section after this experiment shows.
# orders.py
"""Orders sits in front of Pricing, so every Orders call is also a call to Pricing."""
import time
import grpc
import shop_pb2_grpc as pg
from deadlines import child_timeout
class Orders(pg.OrdersServicer):
def __init__(self, pricing_address, mode="propagate", reserve=0.05, channel_options=()):
self.pricing = pg.PricingStub(grpc.insecure_channel(pricing_address, options=list(channel_options)))
self.mode, self.reserve = mode, reserve
self.busy_seconds = [] # how long each Place call kept a handler thread busy
def Place(self, request, context):
started = time.monotonic()
try:
if self.mode == "propagate": # pass on what is left of the caller's deadline
return self.pricing.Quote(request, timeout=child_timeout(context, self.reserve))
if self.mode == "raw": # the obvious idiom: hand over exactly what time_remaining() says
return self.pricing.Quote(request, timeout=context.time_remaining())
if self.mode == "cancel": # no deadline, but cancel the inner call when the caller leaves
call = self.pricing.Quote.future(request)
context.add_callback(call.cancel)
return call.result()
return self.pricing.Quote(request) # mode "none": the mistake
except grpc.RpcError as err:
context.abort(err.code(), err.details())
except grpc.FutureCancelledError:
context.abort(grpc.StatusCode.CANCELLED, "the caller went away")
finally:
self.busy_seconds.append(time.monotonic() - started)
The experiment calls Orders with a one-second deadline while Pricing needs 3.2 seconds. Pricing does check for a departed caller, so any extra work it does is the result of what Orders passed on.
# step04_chain.py
import time
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import serve
from orders import Orders
from pricing import Pricing
request = pb.QuoteRequest(item="apple", qty=1)
def run(mode):
pricing = Pricing(steps=8) # 8 steps of 0.4 s, and it does check for a caller that has gone
pricing_server, pricing_address = serve(pg.add_PricingServicer_to_server, pricing)
orders = Orders(pricing_address, mode=mode)
orders_server, orders_address = serve(pg.add_OrdersServicer_to_server, orders)
stub = pg.OrdersStub(grpc.insecure_channel(orders_address))
start = time.monotonic()
try:
stub.Place(request, timeout=1.0)
outcome = "OK"
except grpc.RpcError as err:
outcome = err.code().name
waited = time.monotonic() - start
time.sleep(3.5) # let any leftover work finish before we count it
seen = pricing.calls[0]["raw_time_remaining"]
seen = "none" if seen > 1e9 else f"{seen:.1f} s"
print(f"{mode:<10} client saw {outcome} after {waited:.1f} s | pricing saw a deadline of {seen:<5} "
f"| pricing ran {pricing.steps_run} of 8 steps | orders handler busy {orders.busy_seconds[0]:.1f} s")
pricing_server.stop(0)
orders_server.stop(0)
for mode in ("none", "propagate", "cancel"):
run(mode)
Run it with python step04_chain.py. It takes about 14 seconds:
none client saw DEADLINE_EXCEEDED after 1.0 s | pricing saw a deadline of none | pricing ran 8 of 8 steps | orders handler busy 3.2 s
propagate client saw DEADLINE_EXCEEDED after 1.0 s | pricing saw a deadline of 0.9 s | pricing ran 3 of 8 steps | orders handler busy 1.0 s
cancel client saw DEADLINE_EXCEEDED after 1.0 s | pricing saw a deadline of none | pricing ran 3 of 8 steps | orders handler busy 1.0 s
Read the three rows one at a time. In mode none, Pricing saw no deadline at all, ran all 8 steps, and the Orders handler stayed busy for 3.2 seconds, although the client had left after 1.0 second. The caller’s deadline simply stopped at Orders. In mode propagate, Pricing was told it had about 0.9 seconds, stopped after 3 steps, and the Orders handler was free after 1.0 second. In mode cancel, Pricing was given no deadline but was cancelled when the caller left, with the same result. In our run with grpcio 1.85.1, then, Python did not carry the deadline across the hop on its own; the handler had to do it.
The trap: what “no deadline” looks like
Why does deadlines.py not simply use timeout=context.time_remaining()? The Python API documentation says time_remaining() returns “None if no deadline was specified for the RPC”. Our run says otherwise, and the next script shows what happens when the caller sets no deadline at all.
# step04b_no_deadline.py
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import call_outcome, serve
from orders import Orders
from pricing import Pricing
request = pb.QuoteRequest(item="apple", qty=1)
# A caller that sets no deadline: what does the Pricing handler see?
pricing = Pricing()
server, address = serve(pg.add_PricingServicer_to_server, pricing)
stub = pg.PricingStub(grpc.insecure_channel(address))
stub.Quote(request)
print(f"time_remaining() with no deadline: {pricing.calls[0]['raw_time_remaining']:.3e} seconds")
text, seconds = call_outcome(stub.Quote, request, timeout=-1) # and a negative timeout?
print(f"timeout=-1: {text} after {seconds:.1f} s")
server.stop(0)
# The same caller, going through Orders with two ways of passing the deadline on.
for mode in ("raw", "propagate"):
pricing = Pricing()
pricing_server, pricing_address = serve(pg.add_PricingServicer_to_server, pricing)
orders = Orders(pricing_address, mode=mode)
orders_server, orders_address = serve(pg.add_OrdersServicer_to_server, orders)
stub = pg.OrdersStub(grpc.insecure_channel(orders_address))
text, seconds = call_outcome(stub.Place, request) # the client sets no timeout
print(f"orders mode {mode:<9}: {text} after {seconds:.1f} s, calls that reached Pricing: {len(pricing.calls)}")
pricing_server.stop(0)
orders_server.stop(0)
Run it with python step04b_no_deadline.py:
time_remaining() with no deadline: 9.223e+18 seconds
timeout=-1: DEADLINE_EXCEEDED after 0.0 s
orders mode raw : DEADLINE_EXCEEDED after 0.0 s, calls that reached Pricing: 0
orders mode propagate: OK 120 cents after 0.0 s, calls that reached Pricing: 1
With no deadline, time_remaining() returned about 9.2 quintillion seconds, not None. The installed source explains why: time_remaining in grpc/_server.py computes max(self._rpc_event.call_details.deadline - time.time(), 0), and a call with no deadline has a deadline set to the far future. Mode raw then handed that huge number to the inner call as a timeout, and the inner call failed at once with DEADLINE_EXCEEDED: not one call reached Pricing. A caller that sets no deadline would have every Orders call fail instantly. That is why time_left treats anything above a year (NO_DEADLINE) as “no deadline” and returns None, and why child_timeout falls back to a value you choose, which defaults to no timeout.
Why a reserve and a floor
The reserve of 0.05 seconds is the time Orders needs to receive Pricing’s failure and answer the caller. Without it, both calls expire together and the caller can only report that its own deadline passed. The floor of one millisecond stops the helper from ever returning zero or a negative number when the deadline is almost gone. The timeout=-1 line in the output above shows what a negative number does: the call fails at once with DEADLINE_EXCEEDED. Both numbers are choices, not rules: size the reserve to the time your own handler needs after the inner call returns.
Step 5: Read status codes and stop leaking details
Every failure ends in a status code, and the code decides what a caller should do next. This step produces the real failures one by one, so that you recognise them in your own logs. The script prints one line per failure: the situation, the status, how long it took, and the first characters of the message.
# step05_status_codes.py
import logging
import re
import threading
import time
from collections import Counter
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import closed_address, serve
from pricing import Pricing
from server_guard import ServerGuard
request = pb.QuoteRequest(item="apple", qty=1)
def show(label, call):
"""Run call() and print the status the client saw."""
start = time.monotonic()
try:
call()
code, details = "OK", ""
except grpc.RpcError as err:
code, details = err.code().name, err.details()
details = re.sub(r"127\.0\.0\.1:\d+", "ADDRESS", details) # random port
details = re.sub(r"reference [0-9a-f]{8}", "reference REF", details) # random reference
print(f"{label:<38} {code:<18} {time.monotonic() - start:3.1f} s {details[:62]}")
def start_pricing(servicer, **server_kwargs):
server, address = serve(pg.add_PricingServicer_to_server, servicer, **server_kwargs)
return pg.PricingStub(grpc.insecure_channel(address)), address, server
print("--- what the handler and the library do")
stub, address, server = start_pricing(Pricing(explode=True))
show("handler raises ValueError", lambda: stub.Quote(request, timeout=2))
server.stop(0)
stub, address, server = start_pricing(Pricing())
show("handler aborts with NOT_FOUND", lambda: stub.Quote(pb.QuoteRequest(item="durian", qty=1), timeout=2))
show("a method this server does not have", lambda: pg.OrdersStub(grpc.insecure_channel(address)).Place(request, timeout=2))
show("5 MB request (the limit is 4 MiB)", lambda: stub.Quote(pb.QuoteRequest(item="x" * 5_000_000, qty=1), timeout=5))
server.stop(0)
stub, address, server = start_pricing(Pricing(steps=3), maximum_concurrent_rpcs=2)
codes, details = [], set()
def call():
try:
stub.Quote(request, timeout=5)
codes.append("OK")
except grpc.RpcError as err:
codes.append(err.code().name)
details.add(err.details())
threads = [threading.Thread(target=call) for _ in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
print("5 calls at once, limit of 2:", dict(Counter(codes)), details)
server.stop(0)
print("--- what the network does")
closed = pg.PricingStub(grpc.insecure_channel(closed_address()))
show("nobody is listening", lambda: closed.Quote(request, timeout=5))
show("nobody is listening, wait_for_ready", lambda: closed.Quote(request, timeout=2, wait_for_ready=True))
stub, address, server = start_pricing(Pricing(steps=8))
show("server too slow, timeout=1", lambda: stub.Quote(request, timeout=1))
pending = stub.Quote.future(request)
time.sleep(0.3)
pending.cancel()
print(f"client cancels after 0.3 s: {pending.code().name} {pending.details()!r}")
server.stop(0)
print("--- the same failures behind ServerGuard")
records = []
class Capture(logging.Handler):
def emit(self, record):
records.append(record)
logging.getLogger("server_guard").addHandler(Capture())
logging.getLogger("server_guard").propagate = False
stub, address, server = start_pricing(Pricing(explode=True), interceptors=[ServerGuard()])
show("handler raises ValueError", lambda: stub.Quote(request, timeout=2))
print("full exception kept in the server log:", "hunter2" in str(records[0].exc_info[1]))
show("caller set no deadline", lambda: stub.Quote(request))
server.stop(0)
stub, address, server = start_pricing(Pricing(), interceptors=[ServerGuard()])
show("handler aborts with NOT_FOUND", lambda: stub.Quote(pb.QuoteRequest(item="durian", qty=1), timeout=2))
server.stop(0)
Run it with python step05_status_codes.py. It takes about 7 seconds. First look at the part above the third heading:
--- what the handler and the library do
handler raises ValueError UNKNOWN 0.0 s Exception calling application: db password is hunter2
handler aborts with NOT_FOUND NOT_FOUND 0.0 s no price for 'durian'
a method this server does not have UNIMPLEMENTED 0.0 s Method not found!
5 MB request (the limit is 4 MiB) RESOURCE_EXHAUSTED 0.1 s SERVER: Received message larger than max (5000007 vs. 4194304)
5 calls at once, limit of 2: {'RESOURCE_EXHAUSTED': 3, 'OK': 2} {'Concurrent RPC limit exceeded!'}
--- what the network does
nobody is listening UNAVAILABLE 2.0 s failed to connect to all addresses; last error: UNAVAILABLE: i
nobody is listening, wait_for_ready DEADLINE_EXCEEDED 2.0 s Deadline Exceeded
server too slow, timeout=1 DEADLINE_EXCEEDED 1.0 s Deadline Exceeded
client cancels after 0.3 s: CANCELLED 'Locally cancelled by application!'
What each line teaches
The handler raises an exception. The client received UNKNOWN with the message Exception calling application: db password is hunter2. The gRPC status code table lists this case: a server-side application that “throws an exception (or does something other than returning a Status code to terminate an RPC)” produces UNKNOWN. The message is the problem: whatever text was inside the exception travelled to the caller. In a real service that could be a SQL fragment, a file path or a password.
The handler aborts with NOT_FOUND. This is the healthy pattern: the handler chose the status and the message. A method the server does not have returns UNIMPLEMENTED. A 5 MB request returns RESOURCE_EXHAUSTED because the default receive limit is 4 MiB (4,194,304 bytes); the docs list “sent or received message was larger than configured limit” under this code.
Too many calls at once. We started the server with maximum_concurrent_rpcs=2 and sent 5 slow calls together. Two were accepted, and three were refused at once with RESOURCE_EXHAUSTED and the message Concurrent RPC limit exceeded!. A fast refusal is a form of load shedding: the caller learns immediately instead of waiting in a queue behind work that may expire anyway.
Nobody is listening. The client got UNAVAILABLE after 2.0 seconds on this Windows machine. The full message continues with the local address and the operating system’s own error text, which is another detail you do not want to show to your users. With wait_for_ready=True the call does not fail fast. It waits for the channel to connect and, with a 2-second deadline, ends as DEADLINE_EXCEEDED after 2.0 seconds. Use wait_for_ready only together with a deadline, otherwise you have recreated the wait-forever default.
Too slow, and cancelled. A slow server with timeout=1 gives DEADLINE_EXCEEDED after 1.0 second. A client that calls cancel() on a pending call gets CANCELLED with the message Locally cancelled by application!.
Which codes should you retry?
The gRPC status code page is careful here: “there is no fixed list of status codes on which it is appropriate to retry in all applications. As a result, individual applications must make their own determination as to which status codes should cause an RPC to be retried.” Its own guidance for the codes in the middle is concrete: use UNAVAILABLE “if the client can retry just the failing call”, use ABORTED if the client should retry at a higher level, and use FAILED_PRECONDITION “if the client should not retry until the system state has been explicitly fixed”. Using that as a base, this is the starting point I recommend; check each row against what your own services mean by it.
| Status | Retry? | Why |
|---|---|---|
UNAVAILABLE |
Yes, with backoff | the docs call it most likely transient, but “not always safe to retry non-idempotent operations” (Step 8) |
DEADLINE_EXCEEDED |
Rarely | your time budget is spent, and the docs warn that this error “may be returned even if the operation has completed successfully” |
RESOURCE_EXHAUSTED, ABORTED |
Case by case | depends on what the service means by them; honour server push-back when it sends it (Step 6) |
UNKNOWN, INTERNAL |
Usually no | a bug or broken invariant; the same request will probably fail the same way |
INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, FAILED_PRECONDITION, UNIMPLEMENTED |
No | the request itself, or the state it needs, has to change first |
Close two holes with a server interceptor
We found two holes: exception text reaches callers, and the server accepts calls that have no deadline. An interceptor closes both in one place, so that no handler has to remember. ServerGuard wraps every unary method. It refuses a call without a deadline with INVALID_ARGUMENT, and it turns an unexpected exception into INTERNAL with a generic message and a short random reference, while the full traceback goes to the server’s log under the same reference. One subtle detail: context.abort() works by raising an exception, so the guard checks context.code() to tell “the handler already chose a status” from “something unexpected blew up”.
# server_guard.py
"""A server interceptor: refuse calls that have no deadline and keep exception text away from callers."""
import logging
import uuid
import grpc
from deadlines import time_left
log = logging.getLogger("server_guard")
class ServerGuard(grpc.ServerInterceptor):
def __init__(self, require_deadline=True, hide_errors=True):
self.require_deadline, self.hide_errors = require_deadline, hide_errors
def intercept_service(self, continuation, handler_call_details):
handler = continuation(handler_call_details)
if handler is None or handler.unary_unary is None:
return handler # unknown method or a streaming method: leave it alone
inner = handler.unary_unary
method = handler_call_details.method
def guarded(request, context):
if self.require_deadline and time_left(context) is None:
context.abort(grpc.StatusCode.INVALID_ARGUMENT, "set a deadline on this call")
try:
return inner(request, context)
except Exception:
if not self.hide_errors or context.code() is not None:
raise # context.abort() already chose the status
ref = uuid.uuid4().hex[:8]
log.exception("unhandled error in %s (ref %s)", method, ref)
context.abort(grpc.StatusCode.INTERNAL, f"internal error, reference {ref}")
return grpc.unary_unary_rpc_method_handler(
guarded,
request_deserializer=handler.request_deserializer,
response_serializer=handler.response_serializer,
)
The last part of the same script ran the failures again behind the guard:
--- the same failures behind ServerGuard
handler raises ValueError INTERNAL 0.0 s internal error, reference REF
full exception kept in the server log: True
caller set no deadline INVALID_ARGUMENT 0.0 s set a deadline on this call
handler aborts with NOT_FOUND NOT_FOUND 0.0 s no price for 'durian'
The exception became INTERNAL with internal error, reference REF (the script replaces the random reference with the word REF so that the output is the same every time), and the original text hunter2 stayed in the server log. A call with no deadline was refused with INVALID_ARGUMENT, and the handler’s own NOT_FOUND passed through untouched. The guard only wraps unary methods and leaves streaming methods alone. When you add require_deadline to an existing system, set it to False first and turn it on once every client sends deadlines, or the guard will refuse the clients you have not fixed yet.
Step 6: Retry the right failures with a service-config policy
You can retry in two ways: write a loop in your own code (our tutorial on retrying API calls with exponential backoff and jitter does that), or let the gRPC channel do it from a retry policy. A policy lives in the channel’s service config, a JSON document that you pass as a channel option. It has a real advantage over a hand-written loop: the library replays the call itself, spreads the waits with jitter, stops at your deadline, understands server push-back, and can throttle itself. First, what happens with no policy at all?
The gRPC retry guide says it plainly: “Retries are enabled by default, but there is no default retry policy.” Only failures from low-level races that gRPC knows the server never saw are retried, which the guide calls transparent retry. So a server that answers UNAVAILABLE gets exactly one attempt unless you write a policy.
A policy, and a helper that builds it
A policy has five fields. maxAttempts “specifies the maximum number of RPC attempts, including the original request”, so 3 means one try and two retries. initialBackoff, maxBackoff and backoffMultiplier shape the waits: the first retry waits about initialBackoff, each later wait is multiplied, and none exceeds maxBackoff. The retry guide adds that “jitter of plus or minus 20%” is applied to each wait so that many clients do not retry in lockstep. retryableStatusCodes lists the only statuses that trigger a retry. The policy applies to a service, or to a method of it, by its full name.
Writing that JSON by hand is where mistakes happen, as the section on policy traps shows. So retries.py builds it from a few Python arguments, takes the service name from the generated descriptor so it can never be misspelled, and refuses values that gRPC would silently clamp or reject.
# retries.py
"""Build a gRPC retry policy (service config) and refuse values gRPC would clamp or reject."""
import json
import grpc
STATUS_NAMES = {code.name for code in grpc.StatusCode if code is not grpc.StatusCode.OK}
def retry_options(
service, # a protobuf ServiceDescriptor, so the name always matches the .proto file
*,
max_attempts=3,
codes=("UNAVAILABLE",),
initial_backoff=0.1,
max_backoff=1.0,
multiplier=2.0,
throttle=None, # (max_tokens, token_ratio)
):
"""Channel options that retry every method of `service` on the listed status codes."""
if not 2 <= max_attempts <= 5:
raise ValueError("max_attempts counts the first try, so it must be 2 to 5 (gRPC clamps larger values to 5)")
codes = list(codes)
unknown = [c for c in codes if c not in STATUS_NAMES]
if not codes or unknown:
raise ValueError(f"retryable codes must be non-empty status names, got {codes} (unknown: {unknown})")
if not 0 < initial_backoff <= max_backoff:
raise ValueError("need 0 < initial_backoff <= max_backoff")
if multiplier <= 0:
raise ValueError("multiplier must be greater than 0")
policy = {
"maxAttempts": max_attempts,
"initialBackoff": f"{initial_backoff:.3f}s",
"maxBackoff": f"{max_backoff:.3f}s",
"backoffMultiplier": multiplier,
"retryableStatusCodes": codes,
}
config = {"methodConfig": [{"name": [{"service": service.full_name}], "retryPolicy": policy}]}
if throttle is not None:
max_tokens, token_ratio = throttle
if not 0 < max_tokens <= 1000 or token_ratio <= 0:
raise ValueError("throttle needs 0 < max_tokens <= 1000 and token_ratio > 0")
config["retryThrottling"] = {"maxTokens": max_tokens, "tokenRatio": token_ratio}
return [("grpc.service_config", json.dumps(config))]
Watch the policy work
The script below calls a Pricing server in six situations and prints how many attempts reached the server. Each failing server answers UNAVAILABLE unless the line says otherwise.
# step06_retries.py
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import call_outcome, serve
from pricing import Pricing
from retries import retry_options
PRICING = pb.DESCRIPTOR.services_by_name["Pricing"]
request = pb.QuoteRequest(item="apple", qty=1)
def attempt(label, pricing, options=(), **call_kwargs):
server, address = serve(pg.add_PricingServicer_to_server, pricing)
stub = pg.PricingStub(grpc.insecure_channel(address, options=list(options)))
text, seconds = call_outcome(stub.Quote, request, **call_kwargs)
print(f"{label:<46} {text:<18} {seconds:3.1f} s attempts: {len(pricing.calls)}")
server.stop(0)
return pricing
policy = retry_options(PRICING, max_attempts=4)
p = attempt("fails twice, then works (4 attempts allowed)", Pricing(fail_first=2), policy)
print(" header the server saw on each attempt:", [c["previous_attempts"] for c in p.calls])
gaps = [b["at"] - a["at"] for a, b in zip(p.calls, p.calls[1:])]
print(" waits before attempts 2 and 3:", [f"{g:.2f} s" for g in gaps])
attempt("always fails with UNAVAILABLE", Pricing(fail_first=10**6), policy)
attempt("always fails with INVALID_ARGUMENT", Pricing(fail_first=10**6, fail_code=grpc.StatusCode.INVALID_ARGUMENT), policy)
attempt("always fails, deadline of 0.15 s", Pricing(fail_first=10**6), policy, timeout=0.15)
attempt("always fails, retries switched off", Pricing(fail_first=10**6), policy + [("grpc.enable_retries", 0)])
attempt("always fails, no policy at all", Pricing(fail_first=10**6))
Run it with python step06_retries.py:
fails twice, then works (4 attempts allowed) OK 120 cents 0.3 s attempts: 3
header the server saw on each attempt: [None, '1', '2']
waits before attempts 2 and 3: ['0.13 s', '0.18 s']
always fails with UNAVAILABLE UNAVAILABLE 0.6 s attempts: 4
always fails with INVALID_ARGUMENT INVALID_ARGUMENT 0.0 s attempts: 1
always fails, deadline of 0.15 s DEADLINE_EXCEEDED 0.2 s attempts: 2
always fails, retries switched off UNAVAILABLE 0.0 s attempts: 1
always fails, no policy at all UNAVAILABLE 0.0 s attempts: 1
The first block is the success story. The server failed twice and then worked, the client saw a plain OK, and it never knew about the two failures. The server could tell, though: gRPC sets a metadata entry named grpc-previous-rpc-attempts on a retried attempt, with the number of earlier attempts, so the three attempts carried no entry, then '1', then '2'. The waits before attempts 2 and 3 were close to the 0.1 and 0.2 seconds the policy asks for (the library randomises each wait by plus or minus 20 percent, and the printed numbers include a few milliseconds of overhead).
The other rows show the limits. A server that always fails got 4 attempts, the maximum we allowed. An INVALID_ARGUMENT failure got 1 attempt because that code is not in the list: retrying a request the server called invalid only repeats the rejection. With a deadline of 0.15 seconds the call ended as DEADLINE_EXCEEDED after 2 attempts, because a third attempt would have started after the deadline. The retry design document states the rule: “gRPC’s call deadline applies across all attempts for a given RPC.” With grpc.enable_retries set to 0 or with no policy, there was 1 attempt, the same as in the default case.
Three traps in policy files
This second script feeds gRPC hand-written policies of the kind people paste from blog posts. It prints the number of attempts and the outcome.
# step06b_policy_traps.py
import json
import re
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import call_outcome, serve
from pricing import Pricing
from retries import retry_options
PRICING = pb.DESCRIPTOR.services_by_name["Pricing"]
request = pb.QuoteRequest(item="apple", qty=1)
def raw_policy(service, **overrides):
"""A hand-written service config, the way people paste one from a blog post."""
policy = {"maxAttempts": 4, "initialBackoff": "0.1s", "maxBackoff": "1s", "backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]}
policy.update(overrides)
config = {"methodConfig": [{"name": [{"service": service}], "retryPolicy": policy}]}
return [("grpc.service_config", json.dumps(config))]
def attempt(label, pricing, options):
server, address = serve(pg.add_PricingServicer_to_server, pricing)
stub = pg.PricingStub(grpc.insecure_channel(address, options=options))
try:
stub.Quote(request)
text = "OK"
except grpc.RpcError as err:
text = err.code().name + " " + err.details().split("service config: ")[-1][:100]
print(f"{label:<28} attempts: {len(pricing.calls)} {text}")
server.stop(0)
attempt("maxAttempts 10", Pricing(fail_first=10**6), raw_policy("shop.v1.Pricing", maxAttempts=10))
attempt("service name typo", Pricing(fail_first=10**6), raw_policy("shop.Pricing"))
attempt("maxAttempts 1", Pricing(fail_first=10**6), raw_policy("shop.v1.Pricing", maxAttempts=1))
attempt("code name typo", Pricing(fail_first=10**6), raw_policy("shop.v1.Pricing", retryableStatusCodes=["UNAVALIABLE"]))
for label, pushback in (("server asks for 600 ms", 600), ("server says stop (-1)", -1)):
pricing = Pricing(fail_first=2, pushback_ms=pushback)
server, address = serve(pg.add_PricingServicer_to_server, pricing)
stub = pg.PricingStub(grpc.insecure_channel(address, options=retry_options(PRICING, max_attempts=4)))
text, seconds = call_outcome(stub.Quote, request)
gaps = [f"{b['at'] - a['at']:.1f} s" for a, b in zip(pricing.calls, pricing.calls[1:])]
print(f"{label:<28} attempts: {len(pricing.calls)} {text}, waits between attempts: {gaps}")
server.stop(0)
Run it with python step06b_policy_traps.py:
maxAttempts 10 attempts: 5 UNAVAILABLE try again later
service name typo attempts: 1 UNAVAILABLE try again later
maxAttempts 1 attempts: 0 INVALID_ARGUMENT [field:methodConfig[0].retryPolicy.maxAttempts error:must be at least 2]
code name typo attempts: 0 INVALID_ARGUMENT [field:methodConfig[0].retryPolicy.retryableStatusCodes error:must be non-empty; field:methodConfig[
server asks for 600 ms attempts: 3 OK 120 cents, waits between attempts: ['0.6 s', '0.6 s']
server says stop (-1) attempts: 1 UNAVAILABLE, waits between attempts: []
Too many attempts is clamped. Asking for maxAttempts 10 gave 5 attempts. The design document says values “greater than 5 are treated as 5 without being considered a validation error”, and gRPC also wrote a line to the console saying it had clamped the value, which is easy to miss in a busy log. A misspelled service name is ignored. The policy named shop.Pricing, but the service is shop.v1.Pricing, so it matched nothing and the call got 1 attempt, with no error anywhere. An invalid policy breaks every call. With maxAttempts 1, or with a misspelled status name (which gRPC drops, leaving an empty list), no call even reached the server: each failed at once with INVALID_ARGUMENT and a message that starts with the offending field. That failure is loud, but it can take down a whole channel after a harmless-looking edit. The retry_options helper rejects the first and third kinds of mistake before the channel exists, and building the name from the descriptor removes the second.
Server push-back
The last two lines of the output show a feature that protects an overloaded server. A server can attach a trailing metadata entry named grpc-retry-pushback-ms to a failure. The design document explains that when a client gets such a response and a retry is appropriate, “it will retry after exactly that delay”, and that a negative value is “seen as the server asking the client not to retry at all”. In our run a push-back of 600 milliseconds made both waits 0.6 seconds instead of roughly 0.1 and 0.2, and a push-back of -1 stopped the retries after the first attempt. The server sets it in pricing.py with context.set_trailing_metadata just before it aborts. A service that is shedding load can use this to tell clients to slow down or stop.
What a policy cannot do
Two limits are worth remembering. First, “clients cannot override retry policy set by the service config”, so a policy applies to every call to that service on the channel; if one method must never be retried, leave it out of the policy by naming only the methods you want (a name entry in the service config can carry a method as well as a service; our helper names the whole service, so extend it if you need this), or call it through a channel without one. Second, retries stop once the response headers arrive: “Once the response header is received, the RPC is committed. No further retries will be attempted, and gRPC hands over the RPC to the application.” A call that fails after its response headers have arrived, for example a streaming call that breaks halfway, is therefore not retried by the channel.
Step 7: Keep retries from multiplying
Retries are most dangerous when a system is already failing. Suppose the client retries 3 times, and Orders retries its calls to Pricing 3 times. If Pricing is down, one client call becomes 3 Orders attempts, and each of those becomes 3 Pricing attempts. The script below measures how many calls reach a Pricing server that always answers UNAVAILABLE, for different settings of the two layers.
# step07_amplification.py
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import call_outcome, serve
from orders import Orders
from pricing import Pricing
from retries import retry_options
PRICING = pb.DESCRIPTOR.services_by_name["Pricing"]
ORDERS = pb.DESCRIPTOR.services_by_name["Orders"]
request = pb.QuoteRequest(item="apple", qty=1)
def pricing_hits(client_attempts, orders_attempts, calls=1, throttle=None):
"""Client -> Orders -> Pricing, with Pricing always answering UNAVAILABLE."""
pricing = Pricing(fail_first=10**6)
pricing_server, pricing_address = serve(pg.add_PricingServicer_to_server, pricing)
inner = retry_options(PRICING, max_attempts=orders_attempts, throttle=throttle) if orders_attempts > 1 else []
orders = Orders(pricing_address, channel_options=inner)
orders_server, orders_address = serve(pg.add_OrdersServicer_to_server, orders)
outer = retry_options(ORDERS, max_attempts=client_attempts, throttle=throttle) if client_attempts > 1 else []
stub = pg.OrdersStub(grpc.insecure_channel(orders_address, options=outer))
for _ in range(calls):
call_outcome(stub.Place, request, timeout=20)
pricing_server.stop(0)
orders_server.stop(0)
return len(pricing.calls)
print("Pricing calls caused by ONE client call")
for client_attempts, orders_attempts in ((1, 1), (3, 1), (1, 3), (3, 3), (5, 5)):
hits = pricing_hits(client_attempts, orders_attempts)
print(f" client tries {client_attempts}, Orders tries {orders_attempts}: {hits}")
print("Pricing calls caused by 10 client calls, both layers trying 3 times")
print(" no throttling: ", pricing_hits(3, 3, calls=10))
print(" with throttling:", pricing_hits(3, 3, calls=10, throttle=(10, 0.1)))
Run it with python step07_amplification.py. It takes about 16 seconds:
Pricing calls caused by ONE client call
client tries 1, Orders tries 1: 1
client tries 3, Orders tries 1: 3
client tries 1, Orders tries 3: 3
client tries 3, Orders tries 3: 9
client tries 5, Orders tries 5: 25
Pricing calls caused by 10 client calls, both layers trying 3 times
no throttling: 90
with throttling: 16
The multiplication is exact: 3 tries at each layer produce 9 calls for every client call, and 5 tries at each layer produce 25. Deadlines limit how long all this takes, but they do not limit how many requests are sent. When the dependency is struggling, that is the extra traffic that keeps it from recovering.
Retry throttling
The retry policy has a second block for exactly this case. In the helper it is the throttle argument, which writes retryThrottling with maxTokens 10 and tokenRatio 0.1 (the values from the gRPC guide). The guide describes the mechanism: for each server the client tracks a token_count; “failed RPCs decrement the count by 1, successful RPCs increment it by tokenRatio. If the token_count falls below half of maxTokens, retries are paused until the count recovers.” In plain words, a client that sees failure after failure stops retrying and sends each request once, and it starts retrying again as successes refill the bucket. In our run, 10 client calls caused 90 Pricing calls without throttling and 16 with throttling on both channels.
A rule of thumb
Retry at one layer, normally the one closest to the failing dependency, and give the other layers a single attempt. Add throttling to every channel that retries. Count the attempts of all layers multiplied together before you pick the numbers. The pattern of adding a circuit breaker or a bulkhead on top is covered in our tutorials on the circuit breaker and the bulkhead.
Step 8: Make retried writes safe with an idempotency key
So far every retried call was a read, which is harmless to repeat. A charge is not. The gRPC status code page says it directly about UNAVAILABLE: “it is not always safe to retry non-idempotent operations”. The retry design document explains that a non-idempotent method “should not be retried unless the error codes guarantee the original RPC was not processed”, and that “gRPC retries do not provide a mechanism to specifically mark a method as idempotent”. The decision is yours. The classic failure is this: the server does the work, then the reply is lost. The client sees UNAVAILABLE, retries, and the work is done a second time.
A ledger that can remember keys
The fix is an idempotency key: a random value that the client creates once per logical operation (one per order, not one per attempt) and sends with every attempt. The server stores the key together with the result, and when the same key arrives again it returns the stored result instead of doing the work again. payments.py implements the server side with SQLite. Two tables hold the ledger and the keys, and one transaction writes the charge and its key together, so there is no moment where one exists without the other. A lock makes threads take turns on the single connection.
The lose_reply_times knob is a stand-in for a real failure. After the charge is committed, the handler aborts with UNAVAILABLE, which is what the client would see if the server crashed, or a proxy reset the connection, between the commit and the reply. The knob is simulated, but the situation it models is one of the most common causes of duplicate writes.
# payments.py
"""A Payments service whose Charge call writes to a ledger. Optionally it remembers idempotency keys."""
import sqlite3
import threading
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
class Payments(pg.PaymentsServicer):
def __init__(self, remember_keys, lose_reply_times=0):
self.db = sqlite3.connect(":memory:", check_same_thread=False, isolation_level=None)
self.db.executescript(
"""
CREATE TABLE charges (id INTEGER PRIMARY KEY, order_id TEXT NOT NULL, cents INTEGER NOT NULL);
CREATE TABLE idempotency (key TEXT PRIMARY KEY, charge_id INTEGER NOT NULL REFERENCES charges(id));
"""
)
self.lock = threading.Lock()
self.remember_keys = remember_keys
self.lose_reply_times = lose_reply_times # stand-in for a crash or proxy reset after the commit
self.attempts = 0
def Charge(self, request, context):
key = dict(context.invocation_metadata()).get("idempotency-key") if self.remember_keys else None
with self.lock:
self.attempts += 1
self.db.execute("BEGIN IMMEDIATE")
try:
row = self.db.execute("SELECT charge_id FROM idempotency WHERE key = ?", (key,)).fetchone() if key else None
if row:
charge_id, replayed = row[0], True
else:
charge_id, replayed = self.db.execute(
"INSERT INTO charges (order_id, cents) VALUES (?, ?)", (request.order_id, request.cents)
).lastrowid, False
if key:
self.db.execute("INSERT INTO idempotency VALUES (?, ?)", (key, charge_id))
self.db.execute("COMMIT")
except Exception:
self.db.execute("ROLLBACK")
raise
lose = self.lose_reply_times > 0
self.lose_reply_times -= 1 if lose else 0
if lose:
context.abort(grpc.StatusCode.UNAVAILABLE, "connection reset before the reply left the server")
return pb.ChargeReply(charge_id=f"ch_{charge_id}", replayed=replayed)
def ledger(self):
with self.lock:
return self.db.execute("SELECT id, order_id, cents FROM charges ORDER BY id").fetchall()
Compare the two behaviours
The script charges order A100 for 1,200 cents through a channel that retries UNAVAILABLE up to 3 times. The first run sends no key. The second run sends one key in the call’s metadata and has a server that remembers it.
# step08_idempotency.py
import uuid
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from labkit import serve
from payments import Payments
from retries import retry_options
PAYMENTS = pb.DESCRIPTOR.services_by_name["Payments"]
charge = pb.ChargeRequest(order_id="A100", cents=1200)
def run(label, remember_keys, send_key):
payments = Payments(remember_keys=remember_keys, lose_reply_times=1) # the first reply is lost
server, address = serve(pg.add_PaymentsServicer_to_server, payments)
channel = grpc.insecure_channel(address, options=retry_options(PAYMENTS, max_attempts=3))
stub = pg.PaymentsStub(channel)
metadata = (("idempotency-key", str(uuid.uuid4())),) if send_key else ()
reply = stub.Charge(charge, timeout=5, metadata=metadata)
ledger = payments.ledger()
print(label)
print(f" attempts that reached the server: {payments.attempts}")
print(f" the client was told: {reply.charge_id}, replayed={reply.replayed}")
print(f" ledger: {ledger}")
print(f" customer charged {sum(cents for _, _, cents in ledger)} cents for a 1200 cent order")
server.stop(0)
run("no key, retries on", remember_keys=False, send_key=False)
run("key sent, server remembers it", remember_keys=True, send_key=True)
Run it with python step08_idempotency.py:
no key, retries on
attempts that reached the server: 2
the client was told: ch_2, replayed=False
ledger: [(1, 'A100', 1200), (2, 'A100', 1200)]
customer charged 2400 cents for a 1200 cent order
key sent, server remembers it
attempts that reached the server: 2
the client was told: ch_1, replayed=True
ledger: [(1, 'A100', 1200)]
customer charged 1200 cents for a 1200 cent order
Without a key, the retry wrote a second row: the customer was charged 2,400 cents for a 1,200-cent order, and the client was never told. With a key, the retried attempt found the key, wrote nothing, and returned the first charge (ch_1) with replayed=True. The same metadata argument is replayed by the library on every attempt, which is why the second attempt carried the key.
What a production version still needs
This lab shows the idea; a real service needs a few more rules. The unique constraint on the key is what protects you across several server processes, because a lock only covers the threads of one process. Keys should expire after a sensible time. The client should store the key next to the order, so that a retry after a client restart reuses it. And a server should reject the same key arriving with a different request, which our lab does not check. Our tutorial on preventing duplicate charges with idempotency keys covers those details for an HTTP API, and they carry over directly. Remember too that DEADLINE_EXCEEDED can hide a success, so a charge that ends that way needs the same key on the follow-up.
Step 9: Make the safe path the default
Everything so far depends on people remembering to set a timeout. A client interceptor removes that dependency: every call through a channel gets a deadline, and none may wait longer than a maximum. The interceptor does not change the ClientCallDetails it receives. It builds a new details object with the changed timeout and passes that on.
# client_guard.py
"""A client interceptor: every call gets a deadline, and no call may wait longer than a maximum."""
import collections
import grpc
class _Details(
collections.namedtuple("_Details", "method timeout metadata credentials wait_for_ready compression"),
grpc.ClientCallDetails,
):
pass
class DefaultDeadline(grpc.UnaryUnaryClientInterceptor):
def __init__(self, default, maximum):
self.default, self.maximum = default, maximum
def intercept_unary_unary(self, continuation, client_call_details, request):
timeout = client_call_details.timeout
timeout = self.default if timeout is None else min(timeout, self.maximum)
details = _Details(
client_call_details.method,
timeout,
client_call_details.metadata,
client_call_details.credentials,
client_call_details.wait_for_ready,
client_call_details.compression,
)
return continuation(details, request)
The experiment wraps a channel with a default of 1.0 second and a maximum of 2.0 seconds and calls the 3.2-second server three times:
# step09_client_guard.py
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
from client_guard import DefaultDeadline
from labkit import call_outcome, serve
from pricing import Pricing
server, address = serve(pg.add_PricingServicer_to_server, Pricing(steps=8)) # 3.2 s of work
channel = grpc.intercept_channel(grpc.insecure_channel(address), DefaultDeadline(default=1.0, maximum=2.0))
stub = pg.PricingStub(channel)
request = pb.QuoteRequest(item="apple", qty=1)
for label, kwargs in (("no timeout given", {}), ("timeout=30", {"timeout": 30}), ("timeout=0.5", {"timeout": 0.5})):
text, seconds = call_outcome(stub.Quote, request, **kwargs)
print(f"{label:<17} -> {text} after {seconds:.1f} s")
server.stop(0)
Run it with python step09_client_guard.py. It takes about 4 seconds:
no timeout given -> DEADLINE_EXCEEDED after 1.0 s
timeout=30 -> DEADLINE_EXCEEDED after 2.0 s
timeout=0.5 -> DEADLINE_EXCEEDED after 0.5 s
The call with no timeout was given the default and failed after 1.0 second. The call that asked for 30 seconds was capped at 2.0. The call that asked for 0.5 seconds kept its own shorter limit. In your own code, create channels in one function and wrap them there, so that nobody can forget:
def make_channel(address, *options):
channel = grpc.insecure_channel(address, options=list(options))
return grpc.intercept_channel(channel, DefaultDeadline(default=2.0, maximum=10.0))
Pair it with ServerGuard from Step 5 on every server. The two interceptors in this lab cover unary calls only. The gRPC Python documentation lists separate interceptor classes for the streaming call shapes, such as UnaryStreamClientInterceptor, if you need them.
Step 10: If you use asyncio, the server behaves differently
Many Python services use grpc.aio instead of threads. The cancellation story is better there, and a last experiment shows how. The handler below has no is_active() check anywhere. It only awaits asyncio.sleep between steps, and it records whether it was cancelled.
# step10_aio.py
import asyncio
import time
import grpc
import shop_pb2 as pb
import shop_pb2_grpc as pg
class AioPricing(pg.PricingServicer):
def __init__(self):
self.steps_run = 0
self.cancelled = False
async def Quote(self, request, context):
try:
for _ in range(8): # 8 steps of 0.4 s, and no is_active() check anywhere
await asyncio.sleep(0.4)
self.steps_run += 1
except asyncio.CancelledError:
self.cancelled = True
raise
return pb.QuoteReply(cents=120)
async def main():
pricing = AioPricing()
server = grpc.aio.server()
pg.add_PricingServicer_to_server(pricing, server)
port = server.add_insecure_port("127.0.0.1:0")
await server.start()
async with grpc.aio.insecure_channel(f"127.0.0.1:{port}") as channel:
start = time.monotonic()
try:
await pg.PricingStub(channel).Quote(pb.QuoteRequest(item="apple", qty=1), timeout=1.0)
except grpc.aio.AioRpcError as err:
print(f"client: {err.code().name} after {time.monotonic() - start:.1f} s ({type(err).__name__})")
await asyncio.sleep(3.5)
print(f"handler: ran {pricing.steps_run} of 8 steps, CancelledError raised inside it: {pricing.cancelled}")
await server.stop(0)
asyncio.run(main())
Run it with python step10_aio.py. It takes about 5 seconds:
client: DEADLINE_EXCEEDED after 1.0 s (AioRpcError)
handler: ran 2 of 8 steps, CancelledError raised inside it: True
When the deadline passed, the asyncio server cancelled the handler’s task, so a CancelledError arrived inside the handler at its next await, and it had run 2 of its 8 steps. The threaded server in Step 3 needed an explicit check to get the same result. Two cautions apply. If your async handler cleans up in an except block, it must re-raise CancelledError, as the lab does, or the cancellation is swallowed. And I did not test deadline propagation on grpc.aio in this lab, so keep passing the remaining time to outgoing calls explicitly, as in Step 4.
Step 11: Prove it with tests and planted bugs
A lab that only worked once is not much use. test_lab.py turns every claim above into an automated test. It reads a caller’s deadline from a fake context, checks that the deadline arrives at Pricing, that a caller with no deadline still works, that Pricing stops when its caller gives up, that both interceptors behave, that retries do what the policy says, and that a retried charge is written once with a key and twice without. The sleep times in the tests are chosen so that each deadline falls in the middle of a step, as in Step 3, to keep them stable.
# test_lab.py
import time
import uuid
import grpc
import pytest
import shop_pb2 as pb
import shop_pb2_grpc as pg
from client_guard import DefaultDeadline
from deadlines import NO_DEADLINE, child_timeout, time_left
from labkit import call_outcome, serve
from orders import Orders
from payments import Payments
from pricing import Pricing
from retries import retry_options
from server_guard import ServerGuard
REQUEST = pb.QuoteRequest(item="apple", qty=1)
PRICING = pb.DESCRIPTOR.services_by_name["Pricing"]
PAYMENTS = pb.DESCRIPTOR.services_by_name["Payments"]
class FakeContext:
def __init__(self, left):
self.left = left
def time_remaining(self):
return self.left
@pytest.fixture
def start():
"""start(add_fn, servicer, **kw) -> address; every server is stopped after the test."""
started = []
def _start(add_fn, servicer, **kwargs):
server, address = serve(add_fn, servicer, **kwargs)
started.append(server)
return address
yield _start
for server in started:
server.stop(0)
def pricing_stub(start, pricing, options=(), channel_wrapper=None, **server_kwargs):
address = start(pg.add_PricingServicer_to_server, pricing, **server_kwargs)
channel = grpc.insecure_channel(address, options=list(options))
if channel_wrapper:
channel = channel_wrapper(channel)
return pg.PricingStub(channel)
# ---- reading the caller's deadline
def test_no_deadline_is_reported_as_none():
assert time_left(FakeContext(None)) is None
assert time_left(FakeContext(NO_DEADLINE * 1000)) is None # what Python really returns
assert time_left(FakeContext(1.5)) == 1.5
def test_child_timeout_keeps_a_reserve_and_a_floor():
assert child_timeout(FakeContext(1.0), reserve=0.05) == pytest.approx(0.95)
assert child_timeout(FakeContext(0.01), reserve=0.05) == 0.001
assert child_timeout(FakeContext(None), fallback=2.0) == 2.0
assert child_timeout(FakeContext(None)) is None
# ---- deadlines through a chain
def test_orders_passes_a_smaller_deadline_to_pricing(start):
pricing = Pricing()
orders = Orders(start(pg.add_PricingServicer_to_server, pricing))
stub = pg.OrdersStub(grpc.insecure_channel(start(pg.add_OrdersServicer_to_server, orders)))
text, _ = call_outcome(stub.Place, REQUEST, timeout=1.0)
assert text == "OK 120 cents"
seen = pricing.calls[0]["raw_time_remaining"]
assert 0.5 < seen < 0.96
def test_a_caller_without_a_deadline_still_works(start):
pricing = Pricing()
orders = Orders(start(pg.add_PricingServicer_to_server, pricing))
stub = pg.OrdersStub(grpc.insecure_channel(start(pg.add_OrdersServicer_to_server, orders)))
text, _ = call_outcome(stub.Place, REQUEST) # no timeout
assert text == "OK 120 cents"
assert pricing.calls[0]["raw_time_remaining"] > NO_DEADLINE
@pytest.mark.parametrize("mode", ["propagate", "cancel"])
def test_pricing_stops_when_the_caller_gives_up(start, mode):
pricing = Pricing(steps=8, step_seconds=0.2)
orders = Orders(start(pg.add_PricingServicer_to_server, pricing), mode=mode)
stub = pg.OrdersStub(grpc.insecure_channel(start(pg.add_OrdersServicer_to_server, orders)))
text, _ = call_outcome(stub.Place, REQUEST, timeout=0.5)
assert text in ("DEADLINE_EXCEEDED", "CANCELLED")
time.sleep(2.0)
assert pricing.steps_run <= 4 # without the check it would be 8
def test_a_handler_that_never_checks_runs_to_the_end(start):
pricing = Pricing(steps=5, step_seconds=0.1, check_cancel=False)
stub = pricing_stub(start, pricing)
text, _ = call_outcome(stub.Quote, REQUEST, timeout=0.2)
assert text == "DEADLINE_EXCEEDED"
time.sleep(0.8)
assert pricing.steps_run == 5
# ---- guards
def test_default_deadline_adds_and_caps(start):
stub = pricing_stub(start, Pricing(steps=10, step_seconds=0.3), channel_wrapper=lambda ch: grpc.intercept_channel(ch, DefaultDeadline(default=0.4, maximum=0.8)))
text, seconds = call_outcome(stub.Quote, REQUEST)
assert text == "DEADLINE_EXCEEDED" and 0.3 < seconds < 0.7
text, seconds = call_outcome(stub.Quote, REQUEST, timeout=30)
assert text == "DEADLINE_EXCEEDED" and 0.7 < seconds < 1.2
text, seconds = call_outcome(stub.Quote, REQUEST, timeout=0.2)
assert text == "DEADLINE_EXCEEDED" and seconds < 0.4
def test_server_guard_refuses_calls_without_a_deadline(start):
stub = pricing_stub(start, Pricing(), interceptors=[ServerGuard()])
assert call_outcome(stub.Quote, REQUEST)[0] == "INVALID_ARGUMENT"
assert call_outcome(stub.Quote, REQUEST, timeout=2)[0] == "OK 120 cents"
def test_server_guard_hides_exception_text(start):
stub = pricing_stub(start, Pricing(explode=True), interceptors=[ServerGuard()])
with pytest.raises(grpc.RpcError) as caught:
stub.Quote(REQUEST, timeout=2)
assert caught.value.code() == grpc.StatusCode.INTERNAL
assert "hunter2" not in caught.value.details()
assert caught.value.details().startswith("internal error, reference ")
def test_server_guard_keeps_the_status_the_handler_chose(start):
stub = pricing_stub(start, Pricing(), interceptors=[ServerGuard()])
with pytest.raises(grpc.RpcError) as caught:
stub.Quote(pb.QuoteRequest(item="durian", qty=1), timeout=2)
assert caught.value.code() == grpc.StatusCode.NOT_FOUND
# ---- retries
def test_retry_policy_recovers_from_two_failures(start):
pricing = Pricing(fail_first=2)
stub = pricing_stub(start, pricing, retry_options(PRICING, max_attempts=3))
assert call_outcome(stub.Quote, REQUEST)[0] == "OK 120 cents"
assert [c["previous_attempts"] for c in pricing.calls] == [None, "1", "2"]
def test_retry_policy_ignores_codes_it_was_not_given(start):
pricing = Pricing(fail_first=10, fail_code=grpc.StatusCode.INVALID_ARGUMENT)
stub = pricing_stub(start, pricing, retry_options(PRICING, max_attempts=4))
assert call_outcome(stub.Quote, REQUEST)[0] == "INVALID_ARGUMENT"
assert len(pricing.calls) == 1
@pytest.mark.parametrize(
"kwargs",
[{"max_attempts": 1}, {"max_attempts": 6}, {"codes": ["UNAVALIABLE"]}, {"codes": []}, {"initial_backoff": 2.0, "max_backoff": 1.0}, {"throttle": (0, 0.1)}],
)
def test_retry_options_refuse_bad_values(kwargs):
with pytest.raises(ValueError):
retry_options(PRICING, **kwargs)
def test_retry_options_use_the_name_from_the_proto_file():
options = dict(retry_options(PRICING))
assert '"service": "shop.v1.Pricing"' in options["grpc.service_config"]
# ---- writes
def charge_through_a_lost_reply(start, remember_keys, send_key):
payments = Payments(remember_keys=remember_keys, lose_reply_times=1)
address = start(pg.add_PaymentsServicer_to_server, payments)
stub = pg.PaymentsStub(grpc.insecure_channel(address, options=retry_options(PAYMENTS, max_attempts=3)))
metadata = (("idempotency-key", str(uuid.uuid4())),) if send_key else ()
reply = stub.Charge(pb.ChargeRequest(order_id="A100", cents=1200), timeout=5, metadata=metadata)
return payments, reply
def test_a_retried_charge_without_a_key_charges_twice(start):
payments, reply = charge_through_a_lost_reply(start, remember_keys=False, send_key=False)
assert len(payments.ledger()) == 2 and not reply.replayed
def test_a_retried_charge_with_a_key_charges_once(start):
payments, reply = charge_through_a_lost_reply(start, remember_keys=True, send_key=True)
assert len(payments.ledger()) == 1 and reply.replayed
Run the suite with python -m pytest -q:
...................... [100%]
22 passed in 8.32s
Passing tests only prove that the code agrees with the tests. To check that the tests would notice a mistake, mutants.py plants one bug at a time in a scratch copy of the lab (for example, “trust the huge no-deadline number”, “never store the key” or “leak exception text”) and runs the suite. A planted bug that survives means a gap in the tests.
# mutants.py
"""Plant one bug at a time in a scratch copy of the lab and check that the test suite notices."""
import pathlib
import shutil
import subprocess
import sys
import tempfile
HERE = pathlib.Path(__file__).resolve().parent
FILES = ["shop_pb2.py", "shop_pb2_grpc.py", "labkit.py", "deadlines.py", "pricing.py", "orders.py",
"payments.py", "retries.py", "server_guard.py", "client_guard.py", "test_lab.py"]
MUTANTS = [
("deadlines.py", "if left is None or left > NO_DEADLINE:", "if left is None:", "trust the huge no-deadline number"),
("deadlines.py", "max(left - reserve, 0.001)", "max(left, 0.001)", "forget the reserve"),
("orders.py", "timeout=child_timeout(context, self.reserve)", "timeout=None", "do not pass the deadline on"),
("orders.py", "context.add_callback(call.cancel)", "context.add_callback(lambda: None)", "do not cancel the inner call"),
("pricing.py", "if self.check_cancel and not context.is_active():", "if False:", "never check for a gone caller"),
("client_guard.py", "min(timeout, self.maximum)", "max(timeout, self.maximum)", "cap with max instead of min"),
("client_guard.py", "self.default if timeout is None else", "timeout if timeout is None else", "no default deadline"),
("server_guard.py", "if self.require_deadline and time_left(context) is None:", "if False:", "accept calls with no deadline"),
("server_guard.py", "if not self.hide_errors or context.code() is not None:", "if True:", "leak exception text"),
("server_guard.py", "if not self.hide_errors or context.code() is not None:", "if not self.hide_errors:", "turn every abort into INTERNAL"),
("retries.py", "if not 2 <= max_attempts <= 5:", "if False:", "allow any maxAttempts"),
("retries.py", "unknown = [c for c in codes if c not in STATUS_NAMES]", "unknown = []", "accept misspelled codes"),
("payments.py", 'dict(context.invocation_metadata()).get("idempotency-key")', 'dict(context.invocation_metadata()).get("idempotency_key")', "read the wrong header name"),
("payments.py", "INSERT INTO idempotency VALUES (?, ?)", "SELECT ?, ?", "never store the key"),
("payments.py", "if key else None\n if row:", "if False else None\n if row:", "never look the key up"),
]
def run_suite(folder):
p = subprocess.run([sys.executable, "-m", "pytest", "-x", "-q", "-p", "no:cacheprovider"], cwd=folder,
capture_output=True, text=True, encoding="utf-8", errors="replace")
last = [line for line in p.stdout.strip().splitlines() if line.strip()][-1]
return p.returncode, last
def main():
caught = 0
for index, (name, old, new, label) in enumerate(MUTANTS, 1):
with tempfile.TemporaryDirectory() as tmp:
for f in FILES:
shutil.copy(HERE / f, tmp)
path = pathlib.Path(tmp) / name
text = path.read_text(encoding="utf-8")
assert text.count(old) == 1, (name, old, text.count(old))
path.write_text(text.replace(old, new), encoding="utf-8", newline="\n")
code, last = run_suite(tmp)
caught += code != 0
print(f"mutant {index:2d} {name:<16} {label:<36} {'caught' if code else 'SURVIVED'} ({last})")
print(f"{caught} of {len(MUTANTS)} planted bugs caught")
return 0 if caught == len(MUTANTS) else 1
if __name__ == "__main__":
sys.exit(main())
Run it with python mutants.py. It takes about two minutes:
mutant 1 deadlines.py trust the huge no-deadline number caught (1 failed in 0.17s)
mutant 2 deadlines.py forget the reserve caught (1 failed, 1 passed in 0.17s)
mutant 3 orders.py do not pass the deadline on caught (1 failed, 2 passed in 0.24s)
mutant 4 orders.py do not cancel the inner call caught (1 failed, 5 passed in 5.35s)
mutant 5 pricing.py never check for a gone caller caught (1 failed, 4 passed in 2.74s)
mutant 6 client_guard.py cap with max instead of min caught (1 failed, 7 passed in 9.82s)
mutant 7 client_guard.py no default deadline caught (1 failed, 7 passed in 9.38s)
mutant 8 server_guard.py accept calls with no deadline caught (1 failed, 8 passed in 7.86s)
mutant 9 server_guard.py leak exception text caught (1 failed, 9 passed in 7.89s)
mutant 10 server_guard.py turn every abort into INTERNAL caught (1 failed, 10 passed in 7.87s)
mutant 11 retries.py allow any maxAttempts caught (1 failed, 13 passed in 8.20s)
mutant 12 retries.py accept misspelled codes caught (1 failed, 15 passed in 8.28s)
mutant 13 payments.py read the wrong header name caught (1 failed, 21 passed in 8.48s)
mutant 14 payments.py never store the key caught (1 failed, 21 passed in 8.52s)
mutant 15 payments.py never look the key up caught (1 failed, 21 passed in 8.51s)
15 of 15 planted bugs caught
All 15 planted bugs were caught. That includes the bug behind the trap in Step 4, the one that trusts the huge number, which a unit test with a fake context catches in a fraction of a second.
Common mistakes and how to spot them
No deadline anywhere. The default is to wait effectively forever. Search your code for stub calls without timeout=, or add the interceptor from Step 9 so that the question cannot come up.
A deadline on the client only. The caller leaves, the server keeps working. Look for long handlers that never call context.is_active() (threads) or that swallow CancelledError (asyncio).
Passing the raw time_remaining() to the next call. It works until a caller sets no deadline, and then every inner call fails at once. Use a helper that maps the huge number to “no deadline”, as time_left does.
Retrying at every layer. Three layers with 3 attempts each means 27 calls at the bottom. Retry in one place and add throttling.
Retrying writes without a key. If a call changes state and its retry policy lists UNAVAILABLE, give it an idempotency key or leave it out of the policy.
Retrying on UNKNOWN or INTERNAL because “errors are errors”. Those usually mean a bug. Repeating the call repeats the bug and adds load.
Showing err.details() to end users. We saw it hold a secret, a local address and an operating system message. Log it on the server, show a generic message and a reference.
A hand-written service config. A typo in a service name is silently ignored, and an invalid value fails every call. Build it in code, and test it, as retries.py and its tests do.
Treating DEADLINE_EXCEEDED as “it did not happen”. For a write, it might have. Check or retry with the same key.
Where to go next
You now have a working pattern for deadlines, cancellation, status codes, retries and idempotent writes, tested on real calls. Good next steps are streaming calls (deadlines and cancellation work differently for them, and the interceptors here do not cover them), the gRPC hedging guide that the retry page links, and metrics: the retry guide lists OpenTelemetry counters such as grpc.client.attempt.started that tell you how many attempts your retries really send. To alert on the result, see our tutorial on burn-rate alerts for an SLO. To roll out servers without dropping calls, see the one on graceful shutdown and rolling restarts.
Sources
The gRPC documentation pages used above: Deadlines, Retry, Status Codes, Cancellation, the gRFC A6 client retries design, the status codes and their use in gRPC reference, the gRPC over HTTP/2 protocol and the gRPC Python API reference. Everything that is described as measured above comes from running the files in this tutorial on Windows 11 with Python 3.13.14, grpcio 1.85.1 and pytest 9.2.0.







No Comment! Be the first one.