How to Keep a gRPC Stream From Hanging Forever in Python With Keepalive, the Ping Timeout That Really Ends the Wait, a Watchdog, and a Reader That Resumes
Stage a dead link on your own computer, measure how long a gRPC stream client really waits, then fix it with keepalive (including the ping timeout setting that actually ended the wait in grpcio...
A server-streaming gRPC call is a promise that messages will keep arriving: a price feed, a log tail, the tokens of a model’s answer. The dangerous part is the silence. If the other side dies in a way that sends nothing back (a power cut, a pulled cable, a firewall that forgot your connection), your client simply waits, and nothing in the default setup tells it that the stream is dead. This tutorial teaches you to make that failure loud and recoverable. You will stage the failure on your own computer and measure how long a client really waits, turn on keepalive and find out which setting actually ends the wait, make the server agree with the client, catch a stream whose connection is healthy but whose producer is stuck, and finish with a reader that notices trouble, resumes where it left off, and never repeats or skips a message.
Table Of Content
- What you will build
- Before you start
- What you need
- A few words we will use
- Step 1: Set up the lab and a network you can break
- Create the project
- Describe the stream in a .proto file
- Build the tick server, the breakable link and the reader
- Check that it works
- Step 2: Watch a stream hang
- Step 3: Turn on keepalive, and find the setting that really ends the wait
- Why the documented timeout did not apply
- Check that it worked
- Step 4: Choose the timeout against real network blips
- Step 5: Make the server agree, or keepalive turns into an error
- What to set in production
- Step 6: Keepalive proves the pipe, not the producer
- Step 7: The server needs the same protection
- Step 8: Streams end on purpose, through deadlines and connection age
- Step 9: Put it together in a reader that resumes
- Resuming from the last message
- Deciding what to retry
- Counting attempts and waiting between them
- Never repeating, never skipping
- Step 10: Test the reader, and prove the tests can fail
- Plant bugs on purpose
- Check the whole thing works end to end
- Common mistakes and how to spot them
- Where to go next
- Sources
gRPC is a way for one program to call a function that lives in another program, usually on another machine. Most calls send one request and get one reply. A server-streaming call sends one request and gets many replies over time, and it may stay open for hours. That long life is the whole problem: a call that lasts hours has hours in which something can break without anyone saying so. Our tutorial on gRPC deadlines and retries covers ordinary one-reply calls; this one is its sequel for streams.
What you will build
You will build a small “tick” service that streams numbered messages, and a tiny network forwarder that you can switch to “silent” to imitate a dead host. On top of that you will build, test and break a reader. By the end you will have:
- a measurement of how long a client waits on a dead link with default settings, and with each keepalive setting;
- a client configuration that detects a dead link in about a second, plus a way to check that your gRPC build really honours it;
- a server configuration that agrees with the client, so keepalive pings are not punished with a
too_many_pingserror; - a watchdog that cancels a stream whose connection is alive but whose producer has stopped;
- a reader function that resumes a stream after a failure without duplicates or gaps;
- 19 tests, and a check that plants 13 bugs on purpose to prove the tests notice them.
Before you start
What you need
You need Python 3.10 or newer (I ran everything on Python 3.13.14), a terminal, and a folder for about twenty small files. I used Windows 11, so a few messages and timings are specific to that machine, and I point those out. You should be comfortable with Python functions, classes and threads at a basic level. You do not need any gRPC experience, Docker or a second machine: the server, the forwarder and the client all run on 127.0.0.1. Plan on about 45 minutes of reading and running.
The versions used are grpcio 1.85.1, grpcio-tools 1.85.1 (it contains the compiler that turns a .proto file into Python), protobuf 7.36.2 (installed as a dependency) and pytest 9.2.0. Your timings will differ from mine by a few hundredths of a second. The status names, the counts and the order of events should match.
A few words we will use
A channel is the client’s connection to a server address, and a stub is the generated client object with one method per call. gRPC runs on HTTP/2, which carries many calls over one TCP connection. HTTP/2 has a small control message called a PING: the receiver must answer it with a matching acknowledgement. A GOAWAY is the message a server sends to say that it is closing the connection and that no new calls should start on it.
A half-open connection is one where one side is gone but the other side has not been told. Keepalive is the practice of sending a PING now and then and treating a missing acknowledgement as a dead connection. A deadline is the moment after which a caller stops waiting for a call, and a status code is how every call ends, for example OK, UNAVAILABLE or DEADLINE_EXCEEDED. A stall, in this lab, means bytes stop moving on a link while the connection itself stays open.
Step 1: Set up the lab and a network you can break
The idea of the lab is simple. We cannot unplug a real cable inside one computer, so we put a small program between the client and the server that copies bytes in both directions. A switch on that program can make it hold the bytes instead of passing them on, while both TCP connections stay open. To the client that looks exactly like a host that died silently.
Create the project
Make a new folder, open a terminal in it, create a virtual environment and install the packages. The first two commands depend on your system, so they are comments in the block below; the last two are the same everywhere. The compiler line reads feed.proto, which you create in the next part, so run it after saving that file.
# Windows (PowerShell): python -m venv venv ; venv\Scripts\Activate.ps1
# macOS or Linux: python3 -m venv venv && source venv/bin/activate
python -m pip install grpcio==1.85.1 grpcio-tools==1.85.1 pytest==9.2.0
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. feed.proto
Describe the stream in a .proto file
A .proto file declares the calls and the message shapes. Ours has one call, Watch, that returns stream Tick: the word stream is what makes it server-streaming. The request says where to resume (from_seq), where to stop (until_seq) and how many seconds to wait between ticks (interval). Keeping where to resume in the request matters later: it is what lets a reader pick up a stream after a failure.
syntax = "proto3";
package feed.v1;
service Feed {
// Server streaming: Tick(seq) for seq = from_seq + 1 .. until_seq, one every `interval` seconds.
rpc Watch (WatchRequest) returns (stream Tick);
}
message WatchRequest {
int64 from_seq = 1;
int64 until_seq = 2;
double interval = 3;
}
message Tick {
int64 seq = 1;
}
Build the tick server, the breakable link and the reader
Save the next file as kit.py. It has three parts. FeedServicer is the server: it sends Tick(seq) every interval seconds, and it counts how many handlers are running so we can see whether the server notices that a client has gone. context.add_callback(gone.set) registers a function that gRPC calls when the call ends for any reason, and the handler waits on that event instead of sleeping, so it wakes up the moment the call is over.
Netsplit is the breakable link. It listens on its own port, connects to the real server, and copies bytes both ways through two queues. stall() closes a gate in front of the writers, so bytes pile up in the queues but nothing is delivered, and both TCP connections stay open. heal() opens the gate again. strand() leaves the gate closed forever for the connections that exist now, but gives new connections a fresh open gate, which imitates traffic moving to a new path. reset() closes every socket with a TCP reset, which imitates a process that died and let the operating system clean up. The forwarder also records every chunk it receives, with a time, so we can see exactly what the client sent.
consume() reads one stream until it ends, fails, or a time cap passes. If the cap passes, it cancels the call itself and reports STILL OPEN, so that a hung stream shows up as a result instead of freezing the lab.
# kit.py
"""Shared pieces for the gRPC streaming lab: a tick server, a TCP link that can go silent, and a stream reader."""
import queue
import socket
import struct
import threading
import time
from concurrent import futures
import grpc
import feed_pb2
import feed_pb2_grpc
class FeedServicer(feed_pb2_grpc.FeedServicer):
"""Watch streams Tick(seq) for seq = from_seq + 1 .. until_seq, one every `interval` seconds."""
def __init__(self, stall_at=0, stall_secs=0.0):
self.stall_at = stall_at # after sending this sequence number the producer goes quiet
self.stall_secs = stall_secs
self.active = 0 # handlers running right now
self.ended = [] # (time.monotonic() when the handler ended, last seq) for every handler
self._lock = threading.Lock()
def Watch(self, request, context):
gone = threading.Event()
context.add_callback(gone.set) # runs when the RPC ends for any reason
seq = request.from_seq
with self._lock:
self.active += 1
try:
while seq < request.until_seq:
if gone.wait(request.interval):
break
seq += 1
yield feed_pb2.Tick(seq=seq)
if seq == self.stall_at and gone.wait(self.stall_secs):
break
finally:
with self._lock:
self.active -= 1
self.ended.append((time.monotonic(), seq))
def start_server(servicer, options=()):
server = grpc.server(futures.ThreadPoolExecutor(max_workers=16), options=list(options))
feed_pb2_grpc.add_FeedServicer_to_server(servicer, server)
port = server.add_insecure_port("127.0.0.1:0")
server.start()
return server, port
class Netsplit:
"""A TCP forwarder with a switch, so a network failure can be staged on one machine.
stall(): bytes stop moving but every connection stays open (a dead host, a firewall that dropped its state)
heal(): held bytes move again
strand(): connections that exist now stay stalled forever; new connections work (traffic moved to a new path)
reset(): every connection gets a TCP reset (a process that died and closed its sockets)
"""
def __init__(self, target_port):
self.target = ("127.0.0.1", target_port)
self.gate = threading.Event()
self.gate.set()
self.seen = [] # (seconds since start, direction, nbytes) for every chunk the proxy received
self._t0 = time.monotonic()
self._socks = []
self._lock = threading.Lock()
self._listener = socket.socket()
self._listener.bind(("127.0.0.1", 0))
self._listener.listen(16)
self.port = self._listener.getsockname()[1]
threading.Thread(target=self._accept, daemon=True).start()
def now(self):
return round(time.monotonic() - self._t0, 2)
def stall(self):
self.gate.clear()
def heal(self):
self.gate.set()
def strand(self):
old = self.gate
old.clear()
self.gate = threading.Event()
self.gate.set()
def reset(self):
with self._lock:
socks, self._socks = self._socks, []
for s in socks:
try:
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close()
except OSError:
pass
def close(self):
try:
self._listener.close()
except OSError:
pass
self.reset()
def _accept(self):
while True:
try:
client, _ = self._listener.accept()
upstream = socket.create_connection(self.target)
except OSError:
return
with self._lock:
self._socks += [client, upstream]
gate = self.gate
for src, dst, name in ((client, upstream, "to_server"), (upstream, client, "to_client")):
q = queue.Queue()
threading.Thread(target=self._read, args=(src, q, name), daemon=True).start()
threading.Thread(target=self._write, args=(dst, q, gate), daemon=True).start()
def _read(self, src, q, name):
try:
while True:
data = src.recv(65536)
if not data:
break
self.seen.append((self.now(), name, len(data)))
q.put(data)
except OSError:
pass
q.put(None)
def _write(self, dst, q, gate):
try:
while True:
data = q.get()
if data is None:
break
gate.wait() # a stalled link holds the bytes here
dst.sendall(data)
dst.shutdown(socket.SHUT_WR)
except OSError:
pass
def consume(target, *, until_seq, interval, from_seq=0, options=(), timeout=None, cap=20.0, channel=None):
"""Read one Watch stream until it ends, fails, or `cap` seconds pass (then we cancel it ourselves)."""
own = channel is None
if own:
channel = grpc.insecure_channel(target, options=list(options))
stub = feed_pb2_grpc.FeedStub(channel)
call = stub.Watch(feed_pb2.WatchRequest(from_seq=from_seq, until_seq=until_seq, interval=interval), timeout=timeout)
gave_up = threading.Event()
def give_up():
gave_up.set()
call.cancel()
timer = threading.Timer(cap, give_up)
timer.daemon = True
timer.start()
start = time.monotonic()
ticks = []
status, details = "OK", ""
try:
for tick in call:
ticks.append((tick.seq, round(time.monotonic() - start, 2)))
except grpc.RpcError as err:
status, details = err.code().name, err.details() or ""
timer.cancel()
if own:
channel.close()
return {
"status": "STILL OPEN" if gave_up.is_set() else status,
"details": details,
"seconds": round(time.monotonic() - start, 2),
"last": ticks[-1][0] if ticks else 0,
"count": len(ticks),
"ticks": ticks,
}
def show(label, r):
text = f"{label}: {r['status']} at {r['seconds']:.1f} s, {r['count']} ticks, last seq {r['last']}"
if r["details"]:
text += f", details: {r['details']}"
print(text)
Check that it works
Run python -c "import kit". It should print nothing. An ImportError for feed_pb2 means the compiler line has not run yet; an error that mentions grpc means the virtual environment is not active.
Step 2: Watch a stream hang
Before you fix a problem, measure it. The next script starts a server, puts the forwarder in front of it, and reads a stream that wants to run for a thousand ticks. Two seconds in, it breaks the link in two different ways. In case A the link is reset: this is what happens when a server process dies, because the operating system closes its sockets and tells the other side. In case B the link goes silent: this is a host that lost power, a cable that was pulled, or a firewall that dropped its record of your connection. The client waits at most 15 seconds, then cancels so that the script can finish.
# step01_dead_peer.py
"""A stream client with no keepalive cannot tell a dead peer from a quiet one."""
import threading
from kit import FeedServicer, Netsplit, consume, show, start_server
def run(label, how):
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
threading.Timer(2.0, getattr(link, how)).start() # the failure starts 2 seconds in
result = consume(f"127.0.0.1:{link.port}", until_seq=1000, interval=0.25, cap=15.0)
show(label, result)
print(f" server handlers still running when the client stopped waiting: {servicer.active}")
server.stop(0)
link.close()
print("A. the server process dies and the operating system resets its connections")
run("client", "reset")
print("B. the link goes silent: a dead host, or a firewall that dropped its state")
run("client", "stall")
Run it with python step01_dead_peer.py. You should see this (the long operating-system text in case A will read differently on Linux or macOS):
A. the server process dies and the operating system resets its connections
client: UNAVAILABLE at 2.0 s, 7 ticks, last seq 7, details: Stream removed (WSAGetOverlappedResult: Connection reset (An existing connection was forcibly closed by the remote host.
-- 10054))
server handlers still running when the client stopped waiting: 0
B. the link goes silent: a dead host, or a firewall that dropped its state
client: STILL OPEN at 15.0 s, 7 ticks, last seq 7, details: Locally cancelled by application!
server handlers still running when the client stopped waiting: 1
Case A is the easy failure: the client got UNAVAILABLE at once, 2.0 seconds in, with seven ticks received. Case B is the dangerous one. The client received the same seven ticks and then simply stopped, and it was still waiting at 15.0 seconds when we cancelled it. A real client would have waited until something else intervened. The last line shows the other half of the problem: the server still had a handler running for a client that was no longer listening.
Why does the client not notice? TCP, the layer under HTTP/2, finds out that a peer is gone only when you send data and no acknowledgement comes back. A client that only reads sends nothing, so TCP has nothing to complain about. The fix has to be to make the client send something on purpose, and that is what an HTTP/2 PING is for.
Step 3: Turn on keepalive, and find the setting that really ends the wait
The gRPC keepalive guide describes the idea in one sentence: “The keepalive ping is a way to check if a channel is currently working by sending HTTP2 pings over the transport.” It names three client settings that matter for this step. In Python you pass them as channel options, which are a list of name and value pairs given when you create the channel:
grpc.keepalive_time_ms: how long the connection must be quiet before the client sends a PING. The default for a client is “disabled”.grpc.keepalive_timeout_ms: how long to wait for the acknowledgement. The guide says that if the sender “does not receive an acknowledgment within this time, it will close the connection”.grpc.keepalive_permit_without_calls: whether to ping when no call is running. Our stream is a running call, so it does not matter here.
Our next script sets a one-second keepalive time and a half-second keepalive timeout. If the documentation is the whole story, a dead link should end the stream about a second and a half after the last sign of life. The script runs three experiments at once so that you do not wait three times. Experiment A uses only the documented settings and is allowed to run for 80 seconds. Experiment B adds one more option, grpc.http2.ping_timeout_ms, which I will explain in a moment. Experiment C is B again with the link dying later, at 6 seconds.
# step02_keepalive.py
"""Keepalive turns a silent dead link into UNAVAILABLE, but which setting bounds the wait is not the one the docs name."""
import threading
from concurrent.futures import ThreadPoolExecutor
from kit import FeedServicer, Netsplit, consume, start_server
DOCUMENTED = [("grpc.keepalive_time_ms", 1000), ("grpc.keepalive_timeout_ms", 500)]
WITH_PING_TIMEOUT = DOCUMENTED + [("grpc.http2.ping_timeout_ms", 500)]
def run(label, options, stall_at, cap):
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
cut = {}
def cut_link():
cut["at"] = link.now()
link.stall()
threading.Timer(stall_at, cut_link).start()
r = consume(f"127.0.0.1:{link.port}", until_seq=100000, interval=0.25, options=options, cap=cap)
pings = [t for (t, d, n) in link.seen if d == "to_server" and n in (17, 43) and t > cut["at"]]
server.stop(0)
link.close()
after = r["seconds"] - stall_at
return (
f"{label}\n {r['status']} at {r['seconds']:.1f} s ({after:.1f} s after the link went silent), "
f"{r['count']} ticks received, details: {r['details']}\n"
f" client chunks of PING size after the link went silent, in seconds: {pings}"
)
jobs = [
("A. keepalive_time 1 s, keepalive_timeout 0.5 s (the documented settings), link dies at 2 s", DOCUMENTED, 2.0, 80.0),
("B. plus grpc.http2.ping_timeout_ms=500, link dies at 2 s", WITH_PING_TIMEOUT, 2.0, 20.0),
("C. same as B, link dies at 6 s", WITH_PING_TIMEOUT, 6.0, 20.0),
]
with ThreadPoolExecutor(max_workers=len(jobs)) as pool:
for text in pool.map(lambda j: run(*j), jobs):
print(text, flush=True)
Run it (it takes about a minute, because of experiment A) and compare your output with mine:
A. keepalive_time 1 s, keepalive_timeout 0.5 s (the documented settings), link dies at 2 s
UNAVAILABLE at 62.3 s (60.3 s after the link went silent), 7 ticks received, details: Stream removed (ping timeout)
client chunks of PING size after the link went silent, in seconds: [2.29]
B. plus grpc.http2.ping_timeout_ms=500, link dies at 2 s
UNAVAILABLE at 3.3 s (1.3 s after the link went silent), 7 ticks received, details: Stream removed (ping timeout)
client chunks of PING size after the link went silent, in seconds: [2.83]
C. same as B, link dies at 6 s
UNAVAILABLE at 7.0 s (1.0 s after the link went silent), 23 ticks received, details: Stream removed (ping timeout)
client chunks of PING size after the link went silent, in seconds: [6.52]
Experiment A is the surprise. With a half-second keepalive timeout the stream ended 60.3 seconds after the link went silent, not 1.5, and the error text says why: “Stream removed (ping timeout)”. Something other than the documented timeout decided when to give up. Experiments B and C, which add grpc.http2.ping_timeout_ms at 500 milliseconds, ended 1.3 and 1.0 seconds after the link went silent, and the second one shows that it does not matter whether the link dies early or late in the stream.
Why the documented timeout did not apply
I did not want to guess, so I asked gRPC core to log its own ping decisions and read the source for the exact release I was running. The next script runs the stall scenario once with the documented settings only, switches on gRPC’s tracing (GRPC_TRACE=http2_ping,http_keepalive,http), and prints the client’s lines that were logged after the link went silent.
# step02b_trace.py
"""Ask gRPC core to log its ping decisions, then print the client lines that explain step 2."""
import os
import re
import subprocess
import sys
import threading
if "--child" in sys.argv:
os.environ["GRPC_VERBOSITY"] = "DEBUG"
os.environ["GRPC_TRACE"] = "http2_ping,http_keepalive,http"
from kit import FeedServicer, Netsplit, consume, start_server
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
def cut():
print('MARK link stalled', file=sys.stderr, flush=True)
link.stall()
threading.Timer(2.0, cut).start()
options = [("grpc.keepalive_time_ms", 1000), ("grpc.keepalive_timeout_ms", 500)]
consume(f"127.0.0.1:{link.port}", until_seq=1000, interval=0.25, options=options, cap=6.0)
server.stop(0)
link.close()
sys.exit(0)
done = subprocess.run([sys.executable, __file__, "--child"], capture_output=True, text=True, env=dict(os.environ, PYTHONUTF8="1"))
wanted = ("KEEPALIVE_PING", "Set ping timeout timer", "Set keepalive ping timeout timer", "Ping delayed", " sent [ipv4")
seen = set()
lines = done.stderr.splitlines()
after = lines[[i for i, l in enumerate(lines) if l.startswith('MARK')][0] + 1:]
for line in after:
if "CLIENT" not in line or not any(w in line for w in wanted):
continue
text = line.split("] ", 1)[1] if "] " in line else line
text = re.sub(r"W:[0-9A-F]+ ", "", text)
text = re.sub(r"CLIENT\[[0-9A-F]+\]", "CLIENT[...]", text)
text = re.sub(r"\[ipv4:127\.0\.0\.1:\d+\]", "[ipv4:127.0.0.1:PORT]", text)
text = re.sub(r"ping id \d+", "ping id N", text)
text = re.sub(r"Ping \d+ sent", "Ping N sent", text)
text = re.sub(r"last_ping_sent_time_: @\d+ms", "last_ping_sent_time_: @Tms", text)
if text not in seen:
seen.add(text)
print(text)
print("log lines after the stall that mention a keepalive ping timeout timer:", sum("Set keepalive ping timeout timer" in l for l in after))
CLIENT [ipv4:127.0.0.1:PORT] state IDLE -> WRITING [KEEPALIVE_PING]
CLIENT[...]: Ping N sent [ipv4:127.0.0.1:PORT]: max_pings_without_data: 2, pings_before_data_required: 0, last_ping_sent_time_: @Tms
CLIENT[...]: Set ping timeout timer of 60000ms for ping id N
log lines after the stall that mention a keepalive ping timeout timer: 0
The client did send a keepalive ping (KEEPALIVE_PING), and it armed a ping timeout of 60,000 milliseconds. No line anywhere mentions a keepalive ping timeout timer, which is the timer that would carry the 500 milliseconds I configured. The source explains the pattern. In gRPC core v1.85.1, chttp2_transport.cc reads the transport settings and gives ping_timeout a default of one minute whenever a keepalive time is set. In writing.cc, the code that arms the keepalive timeout sits inside an if whose conditions are t->keepalive_incoming_data_wanted && t->keepalive_timeout < t->ping_timeout && t->keepalive_ping_timeout_handle != kInvalid. In the two files I searched, the only place that stores a new timer in keepalive_ping_timeout_handle is inside that same block (the other uses cancel the timer and reset the handle), so a first timer can never be armed. That looks like an inverted comparison, but it is my reading of the code and not a confirmed diagnosis: I did not look for an upstream issue report, and I tested only Python’s grpcio on Windows. What I can say as a measurement is that the same comparison appears in writing.cc at each of the ten release tags I checked, from v1.66.0 to v1.85.1, and on the main branch, and that the documented timeout never ended a wait in this build. In a side experiment that is not one of the lab files I also tried 3,000 and 5,000 milliseconds, with and without switching off the bandwidth-delay probe (grpc.http2.bdp_probe), and none of those variations ended the wait within 22 seconds.
The setting that does bound the wait is grpc.http2.ping_timeout_ms. The string appears in the compiled gRPC library that pip installed, but it is not in the public list of channel arguments (channel_arg_names.h), so treat it as an implementation detail that could change. The safe habit is to set both options to the same value, so that a future release that honours the documented one behaves the same, and to repeat the stall test of this step after every gRPC upgrade. If experiment A on your machine ends in about 1.5 seconds instead of 60, your build already honours the documented timeout, and you can drop the extra option.
Check that it worked
Your own pass mark is simple: with the link dying at 2 seconds and both timeouts at 500 milliseconds, the client must fail with UNAVAILABLE within a couple of seconds of the stall. You can also look at the “chunks of PING size” line. A bare PING frame is 17 bytes on the wire (a 9-byte frame header and an 8-byte payload), and a 43-byte chunk is a PING packed together with other small frames. In every experiment the forwarder saw exactly one such chunk from the client after the stall, which is what one keepalive ping looks like.
Step 4: Choose the timeout against real network blips
A timeout that is too short kills healthy streams during an ordinary hiccup, and one that is too long leaves you waiting. To see the trade, the next script runs six experiments at once. In each, the link goes silent at 2 seconds and comes back after a blip of 0.5, 2 or 6 seconds, and the timeout (set on both options from Step 3) is either 500 or 3,000 milliseconds. The stream wants 40 ticks, so a survivor shows 40.
# step03_blips.py
"""A short network blip should not kill a healthy stream, and a long one should. The timeout decides which."""
import threading
from concurrent.futures import ThreadPoolExecutor
from kit import FeedServicer, Netsplit, consume, start_server
def run(timeout_ms, blip):
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
options = [
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", timeout_ms),
("grpc.http2.ping_timeout_ms", timeout_ms),
("grpc.http2.max_pings_without_data", 0),
]
threading.Timer(2.0, link.stall).start()
threading.Timer(2.0 + blip, link.heal).start()
r = consume(f"127.0.0.1:{link.port}", until_seq=40, interval=0.25, options=options, cap=20.0)
server.stop(0)
link.close()
outcome = "stream survived" if r["status"] == "OK" else f"{r['status']} at {r['seconds']:.1f} s"
return timeout_ms, blip, outcome, r["count"]
jobs = [(t, b) for t in (500, 3000) for b in (0.5, 2.0, 6.0)]
with ThreadPoolExecutor(max_workers=len(jobs)) as pool:
results = list(pool.map(lambda args: run(*args), jobs))
print("keepalive time is 1 s in every row; the link stalls at 2 s and heals after the blip")
print(f"{'timeout':>9} {'blip':>7} outcome (ticks received of 40)")
for timeout_ms, blip, outcome, count in results:
print(f"{timeout_ms:>7} ms {blip:>5.1f} s {outcome} ({count})")
keepalive time is 1 s in every row; the link stalls at 2 s and heals after the blip
timeout blip outcome (ticks received of 40)
500 ms 0.5 s stream survived (40)
500 ms 2.0 s UNAVAILABLE at 3.4 s (7)
500 ms 6.0 s UNAVAILABLE at 3.4 s (7)
3000 ms 0.5 s stream survived (40)
3000 ms 2.0 s stream survived (40)
3000 ms 6.0 s UNAVAILABLE at 5.2 s (7)
Read the table by columns. With a 500 millisecond timeout the stream survived the half-second blip and died in both longer ones, each time at 3.4 seconds: the ping went out about 2.9 seconds in, and nothing answered within the timeout. With a 3 second timeout the stream also survived the 2 second blip, because the held bytes were delivered when the link came back, and the PING was answered late but before the timer ran out. Only the 6 second blip killed it, at 5.2 seconds.
So the timeout is the longest silence you are willing to ride out before you pay for a reconnect. The gRFC that defines client keepalive (A8) says an application should make the timeout a multiple of the round-trip time so that lost packets and retransmits do not trip it, and that it may need to be higher for pauses such as garbage collection. Measure your own network’s worst ordinary silence, and set the timeout comfortably above it. If a failed stream costs you only a quick resume, as it will at the end of this tutorial, lean short; if reconnecting is expensive, lean long.
Step 5: Make the server agree, or keepalive turns into an error
Everything so far was one-sided: the client sent PINGs and the server answered. But a server also has opinions about how often it will accept them. The keepalive guide puts it this way: “If a client sends pings more often than the server is willing to accept, the connection will be terminated with a GOAWAY frame”. A server counts each ping that comes too soon as a strike, and closes the connection when the strikes pass a limit. The two server settings are grpc.http2.min_ping_interval_without_data_ms (the shortest gap the server accepts between pings, 300,000 milliseconds, which is 5 minutes, by default) and grpc.http2.max_ping_strikes (the number of bad pings it tolerates, 2 by default).
The client has a limit of its own. The guide says: “When there isn’t any data/header being sent on the transport, gRPC clients restrict the number of pings to 2 by default. Setting GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA to 0 will remove this limit.” In Python the option is grpc.http2.max_pings_without_data. A default client therefore sends only two pings on an idle stream, which keeps it well inside what a default server accepts. That is why the defaults never produce an error, and, as you will see, also why they never detect anything. The next script uses an idle stream (the server sends one tick every 30 seconds, so almost no data flows) to show how the limits interact in five cases.
# step04_too_many_pings.py
"""Pings on an idle stream: the client throttles itself, and the server counts strikes."""
import threading
from concurrent.futures import ThreadPoolExecutor
from kit import FeedServicer, Netsplit, consume, start_server
BASE = [
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", 500),
("grpc.http2.ping_timeout_ms", 500),
]
UNLIMITED = BASE + [("grpc.http2.max_pings_without_data", 0)]
PERMISSIVE = [("grpc.http2.min_ping_interval_without_data_ms", 500), ("grpc.http2.max_ping_strikes", 0)]
# An idle stream: the server sends one tick every 30 seconds, so almost no data crosses the link.
IDLE = dict(until_seq=1000, interval=30.0)
def run(label, client_options, server_options, dies_at=None, cap=12.0):
servicer = FeedServicer()
server, port = start_server(servicer, server_options)
link = Netsplit(port)
if dies_at:
threading.Timer(dies_at, link.stall).start()
r = consume(f"127.0.0.1:{link.port}", options=client_options, cap=cap, **IDLE)
pings = [t for (t, d, n) in link.seen if d == "to_server" and n == 17 and t > 0.5]
server.stop(0)
link.close()
return (
f"{label}\n {r['status']} at {r['seconds']:.1f} s, {r['count']} ticks, details: {r['details']}\n"
f" PING frames the client sent, in seconds: {pings}"
)
jobs = [
("A. client max_pings_without_data=0, default server", UNLIMITED, []),
("B. client max_pings_without_data=0, server accepts a ping every 500 ms and counts no strikes", UNLIMITED, PERMISSIVE),
("C. client with the default max_pings_without_data (2), default server", BASE, []),
("D. default client, permissive server, link dies at 6 s", BASE, PERMISSIVE, 6.0, 20.0),
("E. max_pings_without_data=0, permissive server, link dies at 6 s", UNLIMITED, PERMISSIVE, 6.0, 20.0),
]
with ThreadPoolExecutor(max_workers=len(jobs)) as pool:
for text in pool.map(lambda j: run(*j), jobs):
print(text, flush=True)
A. client max_pings_without_data=0, default server
UNAVAILABLE at 5.1 s, 0 ticks, details: Too many pings
PING frames the client sent, in seconds: [1.03, 2.04, 3.05, 4.06, 5.07]
B. client max_pings_without_data=0, server accepts a ping every 500 ms and counts no strikes
STILL OPEN at 12.0 s, 0 ticks, details: Locally cancelled by application!
PING frames the client sent, in seconds: [1.03, 2.04, 3.05, 4.06, 5.07, 6.08, 7.09, 8.1, 9.12, 10.13, 11.14]
C. client with the default max_pings_without_data (2), default server
STILL OPEN at 12.0 s, 0 ticks, details: Locally cancelled by application!
PING frames the client sent, in seconds: [1.03, 2.04]
D. default client, permissive server, link dies at 6 s
STILL OPEN at 20.0 s, 0 ticks, details: Locally cancelled by application!
PING frames the client sent, in seconds: [1.03, 2.04]
E. max_pings_without_data=0, permissive server, link dies at 6 s
UNAVAILABLE at 6.6 s, 0 ticks, details: Stream removed (ping timeout)
PING frames the client sent, in seconds: [1.03, 2.04, 3.05, 4.06, 5.07, 6.08]
Case A is the failure the guide warns about. The client pinged once a second with no limit, and the default server answered the fifth ping with GOAWAY: the stream ended at 5.1 seconds with UNAVAILABLE and the details “Too many pings”. gRPC also printed an error line on the client, which you should search your logs for: Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings". Current keepalive time (before throttling): 1000ms. The words “before throttling” matter, because the gRFC says a client that receives this GOAWAY should log it and double the keepalive time it uses for new connections on that channel. A client that keeps hitting this error will therefore quietly become slower at noticing dead links.
Case B shows the fix on the server: it accepts a ping every 500 milliseconds and counts no strikes, and the stream stayed open for the full 12 seconds while the client pinged eleven times. Case C is the default client against the default server: it sent two pings (at 1.03 and 2.04 seconds) and then stopped, and the stream stayed open, which looks like success. Case D shows why it is not. With the same default client and a friendly server, the link died at 6 seconds and the client never noticed: it was still waiting at 20 seconds, because it had used up its two pings. Case E removes the client’s limit and detects the same failure at 6.6 seconds.
One honest caveat: in Step 3 the stream was busy, with a tick every quarter of a second, and the default client limit did not get in the way there. I do not know the exact rule that decides which frames reset the count, so the lab only proves what it shows: on an idle stream the default limit hides a dead link.
What to set in production
On the client, set grpc.http2.max_pings_without_data to 0 for streams that may be idle. On the server, set grpc.http2.min_ping_interval_without_data_ms to a value your clients’ keepalive time can always exceed. My recommendation, which I did not measure, is half of the client’s keepalive time (for example 5,000 for clients that ping every 10 seconds) and to leave the strike limit at its default. The lab sets strikes to 0 only to keep the experiments short; on a server that faces the public internet you want some limit, so that one client cannot ping you without end.
Step 6: Keepalive proves the pipe, not the producer
Keepalive answers one question: can this connection still carry a PING? It cannot tell you that the program on the other end is still producing messages. If the server handler hangs on a lock or a slow database, the connection is perfectly healthy and the client waits for ever. The cure is a watchdog: a timer that is restarted every time a message arrives and cancels the call if it ever runs out. Save this as watchdog.py:
# watchdog.py
"""Cancel a call that has stopped producing messages, even though its connection is healthy."""
import threading
class IdleWatchdog:
"""Cancels `call` when poke() has not been called for `idle` seconds."""
def __init__(self, call, idle):
self.call = call
self.idle = idle
self.fired = False
self._lock = threading.Lock()
self._stopped = False
self._timer = None
self.poke()
def poke(self):
with self._lock:
if self._stopped:
return
if self._timer:
self._timer.cancel()
self._timer = threading.Timer(self.idle, self._fire)
self._timer.daemon = True
self._timer.start()
def _fire(self):
self.fired = True
self.call.cancel()
def stop(self):
with self._lock:
self._stopped = True
if self._timer:
self._timer.cancel()
poke() is called on every message and restarts the timer. If the timer fires, _fire sets fired to true and cancels the call, which makes the client’s loop end with a CANCELLED error. The flag matters: it lets the caller tell a cancel caused by a quiet stream apart from a cancel that someone else made. The lock makes poke() and stop() safe to call from different threads, and stop() keeps a late poke() from starting a new timer after you are done.
The next script makes the producer hang after tick 3 while the connection stays healthy. In part A the client has keepalive on but no watchdog. In part B it has a watchdog set to 1.5 seconds.
# step05_stuck_producer.py
"""Keepalive proves the connection is alive, not that the producer is making progress."""
import threading
import time
import grpc
import feed_pb2
import feed_pb2_grpc
from kit import FeedServicer, Netsplit, consume, show, start_server
from watchdog import IdleWatchdog
OPTIONS = [
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", 500),
("grpc.http2.ping_timeout_ms", 500),
("grpc.http2.max_pings_without_data", 0),
]
PERMISSIVE = [("grpc.http2.min_ping_interval_without_data_ms", 500), ("grpc.http2.max_ping_strikes", 0)]
def start(stall_secs):
servicer = FeedServicer(stall_at=3, stall_secs=stall_secs) # the producer hangs after tick 3
server, port = start_server(servicer, PERMISSIVE) # the server tolerates the client's pings
link = Netsplit(port)
return servicer, server, link
# A. healthy link, stuck producer, keepalive on: nothing ever fails
servicer, server, link = start(60.0)
r = consume(f"127.0.0.1:{link.port}", until_seq=1000, interval=0.25, options=OPTIONS, cap=8.0)
show("A. keepalive on, producer stuck after tick 3", r)
pings = [t for (t, d, n) in link.seen if d == "to_server" and n == 17]
print(f" PING-sized chunks from the client: {len(pings)}, last at {pings[-1] if pings else None} s")
server.stop(0)
link.close()
# B. the same stream with an idle watchdog: no tick for 1.5 s means the stream is stuck
servicer, server, link = start(60.0)
channel = grpc.insecure_channel(f"127.0.0.1:{link.port}", options=OPTIONS)
call = feed_pb2_grpc.FeedStub(channel).Watch(feed_pb2.WatchRequest(from_seq=0, until_seq=1000, interval=0.25))
dog = IdleWatchdog(call, idle=1.5)
start_t = time.monotonic()
seqs = []
try:
for tick in call:
dog.poke()
seqs.append(tick.seq)
except grpc.RpcError as err:
print(
f"B. idle watchdog 1.5 s: {err.code().name} at {time.monotonic() - start_t:.1f} s after {len(seqs)} ticks "
f"(watchdog fired: {dog.fired})"
)
dog.stop()
channel.close()
server.stop(0)
link.close()
A. keepalive on, producer stuck after tick 3: STILL OPEN at 8.0 s, 3 ticks, last seq 3, details: Locally cancelled by application!
PING-sized chunks from the client: 7, last at 7.84 s
B. idle watchdog 1.5 s: CANCELLED at 2.3 s after 3 ticks (watchdog fired: True)
Part A is the lesson: keepalive was working (the client kept sending pings, seven PING-sized chunks in eight seconds, and the connection was never declared dead), and the client still received nothing after tick 3. No error ever came. Part B cancelled the stream 2.3 seconds in, which is the time of tick 3 plus the 1.5 second watchdog, and fired is True. Choose the watchdog limit from your producer’s real rhythm: it must be longer than the longest gap between messages that you consider normal. If your producer can be quiet for minutes, have it send a small heartbeat message every so often, so that silence always means trouble.
Step 7: The server needs the same protection
The mirror image of the first problem is on the server. When a client vanishes silently, the server’s handler keeps producing for nobody, holding a thread, memory and whatever the handler has open. Step 2 already showed one such handler. The next script cuts the link at 2 seconds and then looks at the server 12 seconds after the cut. Part A uses a server with default settings. Part B gives the server keepalive options, so that the server pings its clients.
# step06_server_side.py
"""The server has the mirror-image problem: it keeps producing for a client that is gone."""
import threading
import time
from kit import FeedServicer, Netsplit, consume, start_server
CLIENT = [
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", 500),
("grpc.http2.ping_timeout_ms", 500),
("grpc.http2.max_pings_without_data", 0),
]
def run(label, server_options):
servicer = FeedServicer()
server, port = start_server(servicer, server_options)
link = Netsplit(port)
cut = {}
def cut_link():
cut["at"] = time.monotonic()
link.stall()
threading.Timer(2.0, cut_link).start()
# The client side gives up quickly on its own (its keepalive is on); the question is the server.
consume(f"127.0.0.1:{link.port}", until_seq=100000, interval=0.25, options=CLIENT, cap=12.0)
time.sleep(max(0.0, cut["at"] + 12.0 - time.monotonic())) # look at the server 12 s after the cut
if servicer.ended:
delay = servicer.ended[0][0] - cut["at"]
print(f"{label}: handler ended {delay:.1f} s after the link went silent")
else:
print(f"{label}: handler still running 12.0 s after the link went silent (active handlers: {servicer.active})")
server.stop(0)
link.close()
run("A. server with default settings", [])
run(
"B. server keepalive 1 s / 0.5 s",
[
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", 500),
("grpc.http2.ping_timeout_ms", 500),
("grpc.http2.max_pings_without_data", 0),
],
)
A. server with default settings: handler still running 12.0 s after the link went silent (active handlers: 1)
B. server keepalive 1 s / 0.5 s: handler ended 1.4 s after the link went silent
Without server keepalive the handler was still running 12 seconds after the link went silent, and it would have carried on until it finished, however long that is. With it, the handler ended 1.4 seconds after the cut, because the server pinged, got no answer, closed the connection, and gRPC ran the add_callback function that wakes the handler. That wake-up is the reason our handler waits on an event instead of sleeping: a handler that sleeps through the whole interval notices the end of the call late, and one that never checks context.is_active() never notices at all. Our deadlines and retries tutorial has the long version of that lesson for ordinary calls.
Step 8: Streams end on purpose, through deadlines and connection age
Not every end is a failure. Two settings end streams deliberately, and a reader has to expect both. The first is the call deadline. In Python you pass timeout= in seconds, and it covers the whole call, not each message. The second is the server’s maximum connection age: a server can be told to retire every connection after a while, so that clients reconnect and a load balancer can spread them over new servers. The gRFC that defines it (A9) says the server sends a GOAWAY and lets outstanding calls finish, and “the maximum time the connection will be kept alive for outstanding RPCs to complete” is a separate setting called the grace period. It also says a random jitter of 10 percent either way is added to the age. The next script tries both.
# step07_lifetimes.py
"""A stream has a total deadline and the connection under it may be retired by the server."""
from kit import FeedServicer, Netsplit, consume, show, start_server
OPTIONS = [
("grpc.keepalive_time_ms", 1000),
("grpc.keepalive_timeout_ms", 500),
("grpc.http2.ping_timeout_ms", 500),
("grpc.http2.max_pings_without_data", 0),
]
# A. the deadline covers the whole stream, not each message
servicer = FeedServicer()
server, port = start_server(servicer)
r = consume(f"127.0.0.1:{port}", until_seq=1000, interval=0.5, timeout=3.0, cap=10.0)
show("A. timeout=3 on a stream that wants to run for ever", r)
print(f" last tick arrived at {r['ticks'][-1][1]} s; the rest of the 3 s was spent waiting for tick {r['last'] + 1}")
server.stop(0)
# B. the server retires connections: max age 2 s, then 1 s of grace for streams that are still running
for grace_ms in (1000, 10_000):
servicer = FeedServicer()
server, port = start_server(
servicer,
[("grpc.max_connection_age_ms", 2000), ("grpc.max_connection_age_grace_ms", grace_ms)],
)
r = consume(f"127.0.0.1:{port}", until_seq=1000, interval=0.25, options=OPTIONS, cap=14.0)
show(f"B. max connection age 2 s, grace {grace_ms // 1000} s", r)
server.stop(0)
A. timeout=3 on a stream that wants to run for ever: DEADLINE_EXCEEDED at 3.0 s, 5 ticks, last seq 5, details: Stream removed (Deadline Exceeded)
last tick arrived at 2.57 s; the rest of the 3 s was spent waiting for tick 6
B. max connection age 2 s, grace 1 s: UNAVAILABLE at 3.0 s, 11 ticks, last seq 11, details: max connection age
B. max connection age 2 s, grace 10 s: UNAVAILABLE at 12.1 s, 47 ticks, last seq 47, details: max connection age
In part A the call had a 3 second timeout and wanted to run for ever. It ended with DEADLINE_EXCEEDED at exactly 3.0 seconds after five ticks, and the last tick arrived at 2.57 seconds: the rest of the time was spent waiting for tick 6. A stream meant to run for hours must therefore either have no deadline (and rely on keepalive and a watchdog) or have a long one and be resumed when it ends.
In part B the server retired the connection after about 2 seconds. With a 1 second grace period the stream failed at 3.0 seconds with UNAVAILABLE and the details “max connection age”. With a 10 second grace period it ran to 12.1 seconds, about 2 seconds of age plus the full grace, and then failed the same way, with 47 ticks. In both cases the stream was cut even though it was healthy and had work left. The server asked the client to go away, gave it a grace period, and then closed the connection, and the stream died with it. The reader you build next treats this like any other failure: it resumes on a new connection.
Step 9: Put it together in a reader that resumes
You now have the pieces. A reader should (1) use keepalive options that really work, (2) watch for a quiet stream, (3) resume from the last message it saw when a stream fails, (4) retry only the failures that are worth retrying, and (5) never hand the caller a duplicate or a gap. Save this as resilient.py. The list at the top is the set of client options for long-lived streams, with production-sized numbers (a ping after 10 seconds of quiet, and a 5 second limit on the answer).
# resilient.py
"""Read a server-streaming RPC so a dead or stuck stream is noticed, resumed, and never repeats or skips a message."""
import random
import time
import grpc
import feed_pb2
from watchdog import IdleWatchdog
# Client channel options for long-lived streams (names from grpc/include/grpc/impl/channel_arg_names.h).
KEEPALIVE_OPTIONS = [
("grpc.keepalive_time_ms", 10_000), # ping after 10 s without traffic
("grpc.keepalive_timeout_ms", 5_000), # documented: close the connection if the ping is not answered in 5 s
("grpc.http2.ping_timeout_ms", 5_000), # what actually bounds the wait for a ping answer in gRPC core 1.85.1
("grpc.http2.max_pings_without_data", 0), # no cap on pings while the client sends nothing
("grpc.keepalive_permit_without_calls", 1), # keep pinging between streams too
]
class StreamGap(Exception):
"""The server skipped a message."""
class StreamFailed(Exception):
"""Out of attempts or out of time."""
def watch(
get_stub,
until_seq,
interval,
*,
idle_timeout=15.0,
budget=120.0,
max_attempts=5,
base_delay=0.2,
max_delay=5.0,
clock=time.monotonic,
sleep=time.sleep,
rng=random.random,
):
"""Yield Tick 1..until_seq once each, in order, reconnecting after a dead or stuck stream."""
deadline = clock() + budget
last = 0
attempt = 0
while last < until_seq:
remaining = deadline - clock()
if remaining <= 0:
raise StreamFailed(f"budget of {budget} s used up at seq {last}")
attempt += 1
request = feed_pb2.WatchRequest(from_seq=last, until_seq=until_seq, interval=interval)
call = get_stub().Watch(request, timeout=remaining)
dog = IdleWatchdog(call, idle_timeout)
failure = None
try:
for tick in call:
dog.poke()
if tick.seq <= last:
continue # a resend of something already handed out
if tick.seq != last + 1:
raise StreamGap(f"expected {last + 1}, got {tick.seq}")
last = tick.seq
attempt = 0 # progress resets the retry count
yield tick
except grpc.RpcError as err:
if not (dog.fired or err.code() == grpc.StatusCode.UNAVAILABLE):
raise # a real answer such as INVALID_ARGUMENT or DEADLINE_EXCEEDED: do not retry
failure = err
finally:
dog.stop()
call.cancel()
if last >= until_seq:
return
if attempt >= max_attempts:
raise StreamFailed(f"gave up after {attempt} attempts at seq {last}") from failure
delay = min(max_delay, base_delay * 2 ** max(attempt - 1, 0))
sleep(delay * (0.5 + rng() / 2)) # between 50 and 100 percent of the delay
Here is how watch() works, piece by piece. It is a generator: the caller writes for tick in watch(...) and receives ticks one at a time, while the function quietly handles failures between them.
Resuming from the last message
last is the sequence number of the last tick handed to the caller. Every new call asks the server for from_seq=last, so the server starts with the tick after it. This only works because the server can start a stream at a position the client names, which is why we put from_seq in the request in Step 1. If your real server cannot do that, give the stream a resume token, or make the messages idempotent.
Deciding what to retry
When a call fails, the reader retries in two cases: the watchdog fired (dog.fired), or the code is UNAVAILABLE, the code gRPC uses for a lost connection, a keepalive failure and a retired connection alike. Every other code is re-raised at once. INVALID_ARGUMENT or PERMISSION_DENIED would fail the same way on every attempt, and DEADLINE_EXCEEDED here means the overall budget ran out, because each call is given only the time that remains of budget as its timeout.
Counting attempts and waiting between them
A failure that happens after progress is a fresh problem, so attempt goes back to zero whenever a tick arrives, while a stream that fails repeatedly without delivering anything runs out of attempts and raises StreamFailed. Between attempts the reader sleeps for a delay that doubles each time up to max_delay, multiplied by a random factor between 0.5 and 1.0. That random factor is jitter: without it, a thousand clients that lost the same server would all come back at the same instant.
Never repeating, never skipping
A tick with seq <= last is dropped, because it is a resend of something already delivered. A tick that skips ahead raises StreamGap, because silently continuing would hide lost data. The finally block stops the watchdog and cancels the call even if the caller stops reading early, so no call is left running behind your back.
The next script puts the reader to work. It reads 40 ticks, 0.1 seconds apart, through the forwarder. One second in, it calls strand(): the connection that exists stays dead for ever, and new connections work. The settings are the lab’s fast versions of the production ones.
# step08_resilient.py
"""The finished reader: a dead path in the middle of a 40-tick stream, resumed on a new connection."""
import threading
import time
import grpc
import feed_pb2_grpc
from kit import FeedServicer, Netsplit, start_server
from resilient import KEEPALIVE_OPTIONS, watch
# Short timings so the demonstration is quick; production values are in resilient.KEEPALIVE_OPTIONS.
OPTIONS = [
("grpc.keepalive_time_ms", 500),
("grpc.keepalive_timeout_ms", 250),
("grpc.http2.ping_timeout_ms", 250),
("grpc.http2.max_pings_without_data", 0),
]
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
channel = grpc.insecure_channel(f"127.0.0.1:{link.port}", options=OPTIONS)
stub = feed_pb2_grpc.FeedStub(channel)
threading.Timer(1.0, link.strand).start() # at 1.0 s the old path dies; new connections use a working path
start = time.monotonic()
seen = []
previous = start
for tick in watch(lambda: stub, until_seq=40, interval=0.1, idle_timeout=2.0, budget=30.0, base_delay=0.1):
now = time.monotonic()
if now - previous > 0.5:
print(f" gap of {now - previous:.2f} s before tick {tick.seq}")
previous = now
seen.append(tick.seq)
print(f"received {len(seen)} ticks in {time.monotonic() - start:.1f} s; in order and without repeats: {seen == list(range(1, 41))}")
print("production options:", KEEPALIVE_OPTIONS)
channel.close()
server.stop(0)
link.close()
gap of 0.58 s before tick 9
received 40 ticks in 5.0 s; in order and without repeats: True
production options: [('grpc.keepalive_time_ms', 10000), ('grpc.keepalive_timeout_ms', 5000), ('grpc.http2.ping_timeout_ms', 5000), ('grpc.http2.max_pings_without_data', 0), ('grpc.keepalive_permit_without_calls', 1)]
The reader got all 40 ticks, in order and without a repeat. The only pause longer than half a second came before tick 9: the keepalive ping went unanswered, the connection was closed, watch() opened a new stream with from_seq=8, and the new connection used the working path. The whole run took 5.0 seconds instead of the 4.0 an untroubled run needs. Compare that with Step 2, where the same failure left a client waiting for ever.
Step 10: Test the reader, and prove the tests can fail
A reader that is only tried on a healthy network proves nothing, so the tests drive watch() with a fake stream: a small object that plays a script of ticks, pauses and errors. Save this as pytest.ini (it registers a marker that the file uses) and then as test_resilient.py:
[pytest]
markers =
live: starts a real gRPC server and a TCP link on 127.0.0.1
# test_resilient.py
import threading
import time
import grpc
import pytest
import feed_pb2
import feed_pb2_grpc
from resilient import KEEPALIVE_OPTIONS, StreamFailed, StreamGap, watch
CODES = grpc.StatusCode
class FakeRpcError(grpc.RpcError):
def __init__(self, code):
self._code = code
def code(self):
return self._code
def details(self):
return "fake"
class FakeCall:
"""A stream that follows a script: an int is a tick, a float is a pause, a StatusCode raises, 'block' waits for cancel."""
def __init__(self, script):
self.script = list(script)
self.cancelled = threading.Event()
def __iter__(self):
for item in self.script:
if self.cancelled.is_set():
raise FakeRpcError(CODES.CANCELLED)
if item == "block":
self.cancelled.wait(5) # safety cap so a broken watchdog fails the test instead of hanging it
raise FakeRpcError(CODES.CANCELLED)
if isinstance(item, CODES):
raise FakeRpcError(item)
if isinstance(item, float):
time.sleep(item)
continue
yield feed_pb2.Tick(seq=item)
def cancel(self):
self.cancelled.set()
class FakeStub:
def __init__(self, scripts):
self.scripts = list(scripts)
self.from_seqs = []
self.timeouts = []
self.calls = []
def Watch(self, request, timeout=None):
self.from_seqs.append(request.from_seq)
self.timeouts.append(timeout)
call = FakeCall(self.scripts.pop(0))
self.calls.append(call)
return call
def run(scripts, until=5, **kw):
stub = FakeStub(scripts)
sleeps = []
kw.setdefault("rng", lambda: 1.0)
seqs = [t.seq for t in watch(lambda: stub, until, 0.1, sleep=sleeps.append, **kw)]
return seqs, stub, sleeps
def test_clean_stream_needs_one_call():
seqs, stub, sleeps = run([[1, 2, 3, 4, 5]])
assert seqs == [1, 2, 3, 4, 5] and stub.from_seqs == [0] and sleeps == []
def test_resumes_after_the_last_tick_it_saw():
seqs, stub, sleeps = run([[1, 2, CODES.UNAVAILABLE], [3, 4, 5]])
assert seqs == [1, 2, 3, 4, 5] and stub.from_seqs == [0, 2] and len(sleeps) == 1
def test_stream_that_ends_early_with_ok_is_resumed():
seqs, stub, _ = run([[1, 2], [3, 4, 5]])
assert seqs == [1, 2, 3, 4, 5] and stub.from_seqs == [0, 2]
@pytest.mark.parametrize("code", [CODES.INVALID_ARGUMENT, CODES.PERMISSION_DENIED, CODES.DEADLINE_EXCEEDED])
def test_real_answers_are_not_retried(code):
with pytest.raises(FakeRpcError):
run([[1, code]])
def test_gives_up_after_max_attempts():
stub_scripts = [[CODES.UNAVAILABLE]] * 5
with pytest.raises(StreamFailed):
run(stub_scripts, max_attempts=3)
def test_attempt_count_is_exact():
scripts = [[CODES.UNAVAILABLE]] * 5
stub = FakeStub(scripts)
with pytest.raises(StreamFailed):
list(watch(lambda: stub, 5, 0.1, max_attempts=3, sleep=lambda s: None, rng=lambda: 1.0))
assert len(stub.from_seqs) == 3
def test_progress_resets_the_retry_count():
scripts = [[1, CODES.UNAVAILABLE], [2, CODES.UNAVAILABLE], [3, CODES.UNAVAILABLE], [4, 5]]
seqs, _, _ = run(scripts, max_attempts=2)
assert seqs == [1, 2, 3, 4, 5]
def test_resent_ticks_are_dropped():
seqs, _, _ = run([[1, 2, 3, CODES.UNAVAILABLE], [2, 3, 4, 5]])
assert seqs == [1, 2, 3, 4, 5]
def test_a_skipped_tick_is_an_error():
with pytest.raises(StreamGap):
run([[1, 2, 4]])
def test_idle_watchdog_cancels_a_stuck_stream_and_resumes():
seqs, stub, _ = run([[1, "block"], [2, 3, 4, 5]], idle_timeout=0.05)
assert seqs == [1, 2, 3, 4, 5] and stub.from_seqs == [0, 1]
def test_a_slow_but_moving_stream_is_not_cancelled():
seqs, stub, _ = run([[1, 0.03, 2, 0.03, 3, 0.03, 4, 0.03, 5]], idle_timeout=0.1)
assert seqs == [1, 2, 3, 4, 5] and stub.from_seqs == [0]
def test_budget_limits_the_whole_job_and_each_call():
ticks = iter([0.0, 1.0, 100.0])
scripts = [[1, CODES.UNAVAILABLE]]
stub = FakeStub(scripts)
with pytest.raises(StreamFailed, match="budget"):
list(watch(lambda: stub, 5, 0.1, budget=5.0, clock=lambda: next(ticks), sleep=lambda s: None, rng=lambda: 1.0))
assert stub.timeouts == [4.0]
def test_backoff_doubles_then_stops_growing():
scripts = [[CODES.UNAVAILABLE]] * 6
stub = FakeStub(scripts)
sleeps = []
with pytest.raises(StreamFailed):
list(watch(lambda: stub, 5, 0.1, max_attempts=6, base_delay=1.0, max_delay=5.0, sleep=sleeps.append, rng=lambda: 1.0))
assert sleeps == [1.0, 2.0, 4.0, 5.0, 5.0]
def test_backoff_has_jitter_between_half_and_full():
scripts = [[CODES.UNAVAILABLE], [1, 2, 3, 4, 5]]
_, _, low = run(scripts, base_delay=1.0, rng=lambda: 0.0)
assert low == [0.5]
def test_every_call_is_cancelled_when_the_reader_walks_away():
stub = FakeStub([[1, 2, 3, 4, 5]])
gen = watch(lambda: stub, 5, 0.1, sleep=lambda s: None)
assert next(gen).seq == 1
gen.close()
assert all(call.cancelled.is_set() for call in stub.calls)
def test_keepalive_options_set_every_knob_that_matters():
options = dict(KEEPALIVE_OPTIONS)
assert options["grpc.keepalive_time_ms"] > 0
assert 0 < options["grpc.http2.ping_timeout_ms"] <= 10_000
assert options["grpc.http2.max_pings_without_data"] == 0
assert options["grpc.keepalive_permit_without_calls"] == 1
@pytest.mark.live
def test_live_stream_survives_a_stranded_connection():
from kit import FeedServicer, Netsplit, start_server
servicer = FeedServicer()
server, port = start_server(servicer)
link = Netsplit(port)
options = [
("grpc.keepalive_time_ms", 300),
("grpc.keepalive_timeout_ms", 200),
("grpc.http2.ping_timeout_ms", 200),
("grpc.http2.max_pings_without_data", 0),
]
channel = grpc.insecure_channel(f"127.0.0.1:{link.port}", options=options)
stub = feed_pb2_grpc.FeedStub(channel)
threading.Timer(0.4, link.strand).start()
seqs = [t.seq for t in watch(lambda: stub, 20, 0.05, idle_timeout=2.0, budget=20.0, base_delay=0.05)]
channel.close()
server.stop(0)
link.close()
assert seqs == list(range(1, 21))
The fake call follows a script: an integer is a tick, a float is a pause, a status code raises that error, and "block" waits until someone cancels the call. It checks for cancellation before every item, as a real call would. The tests cover a clean stream, resuming after the last tick, a stream that ends early with OK, real answers that must not be retried, the exact attempt count, progress resetting the count, resent ticks, gaps, the watchdog, a slow stream that is still moving, the budget and per-call timeout, backoff with and without jitter, cancelling on exit, and the keepalive options. The last test is live: it starts the real server and forwarder, strands the connection 0.4 seconds in, and checks that the reader still delivers all 20 ticks. Run python -m pytest -q:
................... [100%]
19 passed in 1.89s
Plant bugs on purpose
Passing tests only mean something if they can fail. The next script copies resilient.py, plants one bug at a time (resume from zero, no duplicate filter, no gap check, retry everything, never reset the attempt count, no watchdog poke, no backoff cap, no jitter floor, one attempt too many, no per-call timeout, the old two-ping limit, no ping timeout option, no cancel on exit) and checks that the tests fail for each one.
# mutate.py
"""Plant one bug at a time in resilient.py and check that the tests notice."""
import pathlib
import shutil
import subprocess
import sys
import tempfile
HERE = pathlib.Path(__file__).parent
SRC = (HERE / "resilient.py").read_text(encoding="utf-8")
MUTANTS = [
("resume from zero instead of the last tick", "from_seq=last,", "from_seq=0,"),
("do not drop resent ticks", "if tick.seq <= last:\n continue", "if False:\n continue"),
("ignore a skipped tick", "if tick.seq != last + 1:", "if False:"),
("retry every error", "if not (dog.fired or err.code() == grpc.StatusCode.UNAVAILABLE):", "if False:"),
("never reset the retry count", "attempt = 0 # progress resets the retry count", "pass"),
("do not poke the watchdog on a tick", " dog.poke()\n", ""),
("no cap on the backoff", "min(max_delay, base_delay * 2 ** max(attempt - 1, 0))", "base_delay * 2 ** max(attempt - 1, 0)"),
("no jitter floor", "(0.5 + rng() / 2)", "rng()"),
("one attempt too many", "if attempt >= max_attempts:", "if attempt > max_attempts:"),
("forget the per-call timeout", "timeout=remaining", "timeout=None"),
("keep the two-ping limit", '("grpc.http2.max_pings_without_data", 0)', '("grpc.http2.max_pings_without_data", 2)'),
("drop the ping timeout option", ' ("grpc.http2.ping_timeout_ms", 5_000), # what actually bounds the wait for a ping answer in gRPC core 1.85.1\n', ""),
("never cancel the call on exit", " dog.stop()\n call.cancel()", " dog.stop()"),
]
caught = 0
for name, old, new in MUTANTS:
assert SRC.count(old) >= 1, f"pattern not found: {name}"
with tempfile.TemporaryDirectory() as tmp:
tmp = pathlib.Path(tmp)
for f in ("kit.py", "watchdog.py", "feed_pb2.py", "feed_pb2_grpc.py", "test_resilient.py"):
shutil.copy(HERE / f, tmp / f)
(tmp / "resilient.py").write_text(SRC.replace(old, new, 1), encoding="utf-8", newline="\n")
done = subprocess.run(
[sys.executable, "-m", "pytest", "-q", "-x", "-m", "not live", "-p", "no:cacheprovider"],
cwd=tmp, capture_output=True, text=True, timeout=120,
)
failing = [line for line in done.stdout.splitlines() if line.startswith("FAILED")]
status = "caught" if done.returncode != 0 else "SURVIVED"
caught += done.returncode != 0
print(f"{status:>8} {name} {failing[0][:70] if failing else ''}")
print(f"{caught} of {len(MUTANTS)} planted bugs caught")
caught resume from zero instead of the last tick FAILED test_resilient.py::test_resumes_after_the_last_tick_it_saw - as
caught do not drop resent ticks FAILED test_resilient.py::test_resent_ticks_are_dropped - resilient.St
caught ignore a skipped tick FAILED test_resilient.py::test_a_skipped_tick_is_an_error - IndexError
caught retry every error FAILED test_resilient.py::test_real_answers_are_not_retried[StatusCode
caught never reset the retry count FAILED test_resilient.py::test_progress_resets_the_retry_count - resil
caught do not poke the watchdog on a tick FAILED test_resilient.py::test_a_slow_but_moving_stream_is_not_cancell
caught no cap on the backoff FAILED test_resilient.py::test_backoff_doubles_then_stops_growing - as
caught no jitter floor FAILED test_resilient.py::test_backoff_has_jitter_between_half_and_ful
caught one attempt too many FAILED test_resilient.py::test_attempt_count_is_exact - assert 4 == 3
caught forget the per-call timeout FAILED test_resilient.py::test_budget_limits_the_whole_job_and_each_ca
caught keep the two-ping limit FAILED test_resilient.py::test_keepalive_options_set_every_knob_that_m
caught drop the ping timeout option FAILED test_resilient.py::test_keepalive_options_set_every_knob_that_m
caught never cancel the call on exit FAILED test_resilient.py::test_every_call_is_cancelled_when_the_reader
13 of 13 planted bugs caught
The first time I ran this, 12 of the 13 were caught. The survivor was “do not poke the watchdog on a tick”: my fake stream ignored cancellation, so the watchdog fired and nothing happened. I changed the fake to check for cancellation, and the test then failed for that bug, as it should. A survivor is always a hint about a test that is weaker than it looks.
Check the whole thing works end to end
Run the steps in order, then the tests. The pass marks are: Step 2 shows case B still open at 15 seconds; Step 3 shows experiment A ending in 60 seconds on a build that ignores the documented timeout (or about 1.5 seconds on one that honours it) and experiments B and C ending within about a second of the stall; Step 5 shows the idle stream with the default client never failing; Step 9 prints True; and pytest and the planted-bug run end with “19 passed” and “13 of 13 planted bugs caught”. This table sums up every setting used and where it goes:
| Option | Side | Lab value | Production starting point | What it does |
|---|---|---|---|---|
grpc.keepalive_time_ms |
client and server | 1,000 | 10,000 on the client | Quiet time before a PING is sent |
grpc.keepalive_timeout_ms |
client and server | 500 | 5,000 | The documented wait for the answer; ignored in grpcio 1.85.1 as measured |
grpc.http2.ping_timeout_ms |
client and server | 500 | 5,000 | What actually ended the wait in grpcio 1.85.1; not in the public option list |
grpc.http2.max_pings_without_data |
client | 0 | 0 | Removes the two-ping limit on idle streams |
grpc.keepalive_permit_without_calls |
client | not needed | 1 | Keeps pinging while no call is running |
grpc.http2.min_ping_interval_without_data_ms |
server | 500 | about half the client’s keepalive time | Shortest gap between client pings that the server accepts |
grpc.http2.max_ping_strikes |
server | 0 | leave at the default | Bad pings tolerated before GOAWAY |
grpc.max_connection_age_ms and grpc.max_connection_age_grace_ms |
server | 2,000 and 1,000 or 10,000 | your load balancer’s needs | Retires connections; streams must resume |
The production column is my suggestion for a starting point, not something the lab measured, apart from the values the steps used.
Common mistakes and how to spot them
Trusting keepalive_timeout_ms without a stall test. In grpcio 1.85.1 it did not decide anything. Run Step 3 on your build, and again after every upgrade.
Keepalive on the client only. The client recovers and the server keeps a zombie handler. Give the server keepalive options too, and make the handler wake up when the call ends.
A client that pings more often than the server allows. Look for ENHANCE_YOUR_CALM and too_many_pings in the client log. Fix the server’s minimum interval or lengthen the client’s keepalive time.
The default two-ping limit on idle streams. It looks fine until a link dies after the second ping. Set grpc.http2.max_pings_without_data to 0 where streams can be quiet.
A healthy connection mistaken for a healthy stream. Keepalive cannot see a stuck producer. Use a watchdog, and have the producer send heartbeats if it can be quiet for long.
A deadline on a stream that should live for hours. The deadline covers the whole call. Use none and rely on keepalive and a watchdog, or use a long one and resume.
A stream that cannot resume. If the server cannot start at a position the client names, a reconnect means duplicates or lost data. Design the resume token before you need it.
Retrying every error. A bad request fails the same way every time and adds load. Retry UNAVAILABLE and your own watchdog cancel, and raise the rest.
Where to go next
You now have a way to make a silent dead stream loud, measured on a real gRPC client and server, and a reader that survives it. Good next steps are our tutorials on gRPC deadlines and retries for ordinary calls, HTTP timeouts and the slow-drip response for the same silence problem over plain HTTP, graceful shutdown and rolling restarts for servers that retire connections on purpose, and resuming a stream after a dropped connection for the browser version of Step 9. If you want to understand how a system decides that a peer is dead, read failure detection with heartbeats, and for a proxy that closes idle connections out from under you, random 502 errors behind nginx.
Sources
gRPC documentation and design notes used above: the keepalive user guide for gRPC core, the gRFC A8 on client-side keepalive, the gRFC A9 on server-side connection management, the public list of channel arguments in channel_arg_names.h, and the gRPC core source for writing.cc and chttp2_transport.cc at the v1.85.1 tag. The deadlines guide and the keepalive guide on grpc.io cover the same ground at a higher level. Everything described as measured above comes from running the files in this tutorial on Windows 11 with Python 3.13.14, grpcio 1.85.1, protobuf 7.36.2 and pytest 9.2.0. I tested only the Python library, so I make no claim about gRPC in other languages.








No Comment! Be the first one.