Cloudflare’s QuePaxa Post Turns Leader Elections Into a Hedging Delay, but in a Simulation of the Paper’s Protocol a Five-Millisecond Hedge Costs Every Slot Its Fast Path, and the 2020 Outage It Cites Lasted Six Hours Because Database Replicas Had to Be Rebuilt

On November 11, 2026, Cloudflare’s research team published “Choosing QuePaxa, an unusual consensus algorithm”, the second post about Meerkat, an experimental service that keeps replicas of a control-plane log in agreement across data centers. Meerkat runs on QuePaxa, a consensus protocol from a 2023 SOSP paper by Tennage, Băsescu and five co-authors. The post argues that Raft and Multi-Paxos depend on a leader and on timeouts, that timeouts misfire on wide-area networks, and that QuePaxa replaces them with “hedging” while keeping a one-round-trip fast path.

The argument matches the paper in most places. But a post written to explain a design choice leaves out the places where the design bends, and many of its sentences can be checked. I checked ten of them against the paper, against the 2020 post-mortem the post cites, against etcd’s tuning guide, against the earlier Meerkat announcement, and against a simulator I wrote from the paper’s pseudocode. The post’s page shows a publication stamp of 14:24 UTC and a modification stamp of 17:25 UTC on November 11; I read the later version at about 19:40 UTC.

The ten claims, checked

“Post” below means the November 11 post unless I say it is the July 8 Meerkat announcement. Every number in the middle column comes from a script printed further down this page, or from the paper, post-mortem or documentation named in the cell.

What the post says	What I found	Verdict
“each round has a 50% chance of deciding, but in practice, that probability is much higher, thanks to convergence”	The paper’s bound is “at least 1/2” per round against a network scheduler that cannot see priorities. In my four random-delay models the first round decided 87% to 100% of slots.	Holds. The 50% is a worst-case bound; random delays do much better.
“it becomes exponentially easier to decide as rounds progress”	The chance of deciding in round 2 or 3, given no earlier decision, stayed between 0.90 and 0.94, close to round 1’s 0.87 to 0.99, instead of climbing. The share of slots still undecided falls geometrically.	Not supported by these runs.
“a five-millisecond hedge timeout only increases the number of active proposers”	With a 90 ms one-way delay, a five-millisecond hedge removed the fast path from every slot, and so did any equal hedge up to 60 ms: three round trips instead of one (median 523 to 530 ms against 177 ms) and 146 to 148 messages per slot instead of 12. Staggered hedges kept the fast path in 16% to 63% of slots at 30 to 50 ms. The paper’s Figure 9 shows about 2.8 steps per slot at its smallest hedge, 50 ms.	Understated. The post does add “may take more time to decide”.
In Raft, a five-millisecond timeout in a 100 ms network “would completely freeze the system”	With timeouts of 5 to 10 ms and a 180 ms round trip, none of 30 simulated clusters showed a leader at any 100 ms sample in 30 seconds. The paper says Raft and Multi-Paxos “fail to stabilize at all” below 200 ms at a 180 ms round trip.	Confirmed.
Raft “is commonly deployed with an election timeout of five to ten times the network round-trip time”	The phrase links to etcd’s v3.4 tuning page, which says election timeouts “must be at least 10 times” the round trip (the v3.6 page says the same). The paper found Raft stable from about 1.8 times. My simulator was stable from 1 round trip at low jitter.	The linked page says at least 10 times, not five.
“we can recover as fast as we want”	A slot opened with the leader down took about three round trips plus the hedge: 530 ms median at a hedge of zero and a 180 ms round trip. Raft’s crash-to-commit medians were 0.83 s, 1.5 s and 2.6 s at timeouts of 1.8 to 3.6, 5 to 10 and 10 to 20 times the round trip.	Faster than Raft by 1.2 to 4.9 times, not unbounded.
A two-second outage “can make the leader appear to be faulty and force a leader election, which can fail, and force further elections”	In the Raft simulator, after a two-second outage of the leader or of the whole network, no run went more than 2.5 s without a leader once the network healed. One cut link settled after about two leader changes.	Not reproduced in textbook Raft with randomized timeouts.
In 2020 “the partial failure of a core Cloudflare network switch caused a consensus-powered system to degrade and the dashboard to operate 80x slower than usual, for six hours”	The post-mortem makes the read-only etcd cluster the trigger. The six hours and the 80 times came from an automatic database failover, a defect that forced every replica to be rebuilt, and traffic steering.	Compressed. The cascade is what lasted.
July post: QuePaxa “maintains much higher (~10x) throughput than Raft and Multi-Paxos” under attack	The paper’s attack test gives “at least 75k cmd/sec” for QuePaxa against “2.5k cmd/sec” for Multi-Paxos and Raft, a ratio of at least 30. I found no tenfold figure in the paper.	Understated by the post.
Chart caption: the decision rate falls to “around 65%”; text: throughput drops “down to at most a third”	Pixel readout of the chart: lowest value 64.0%, never near the 33% reference line. Phase 2 decisions stay near 1% for 35 seconds after the labeled failure.	The 65% matches the chart. The third is a bound the chart does not reach.

What I built to check them

The simulator is a Python 3.13 translation of the paper’s Algorithm 3 (the recorder’s register) and Algorithm 4 (the proposer loop) with a discrete-event network around it: random one-way delays, optional slow replicas, hedged start times and crashes. Replicas are both proposers and recorders, delivery between live replicas is reliable (the paper’s assumption), and a replica talks to itself with no delay. For Raft I reused the simulator from our Raft tutorial. For the chart I read pixels. Standard library only, plus Pillow for the chart.

What I did not test

Meerkat itself: the two posts give no code, no benchmark setup and no cluster configuration, and the July post says it is “not deployed to production”. The paper’s Go prototype: I read its README and file list, not its behavior. Real networks: my delays are log-normal, not measured. Raft variants: etcd’s v3.6 flag list shows a pre-vote option set to true by default, and real Raft deployments add mitigations the post does not discuss; my Raft is the textbook algorithm, with no leader lease and no pre-vote.

QuePaxa in 138 lines

Each replica’s recorder keeps three things: the current step, the first proposal it saw in that step, and the highest proposal it saw in the step before. A proposal is a priority, a proposer number and a value. A round has four phases and the step number is four times the round plus the phase. In phase 0 a proposer adopts the best first value it hears from a majority. Phase 1 spreads it. In phase 2 it decides if its adopted proposal is still the best the recorders summarize. Phase 3 carries the best value into the next round. The leader’s shortcut: in round 1 only, its proposal carries the reserved top priority, and if every reply in its majority says that proposal was first, it decides after phase 0, one round trip.

# quepaxa.py
"""quepaxa.py: one slot of QuePaxa (SOSP 2023, Algorithms 3 and 4) on a simulated network.

Every replica is both a proposer (active) and a recorder (passive). A proposal is the tuple
(priority, proposer id, value); Python's tuple comparison is the paper's "best" (highest priority,
ties broken by proposer id). The leader's first-round proposal carries the reserved priority H.
"""
import heapq
import random

H = 1 << 62            # the leader's reserved priority; everyone else draws from 1..H-1
NIL = (0, -1, -1)      # the paper's "nil": aggregate(v, nil) = v


class Recorder:
    """Algorithm 3, the constant-space interval summary register (ISR)."""

    def __init__(self):
        self.S, self.Fc, self.Ac, self.Ap = 0, NIL, NIL, NIL

    def record(self, s, v):
        if s == self.S:
            self.Ac = max(self.Ac, v)                      # aggregate everything seen in this step
        elif s > self.S:
            self.Ap = self.Ac if s == self.S + 1 else NIL  # the step we just left (or nothing, if skipped)
            self.S, self.Fc, self.Ac = s, v, v             # first value of the new step
        return self.S, self.Fc, self.Ap                    # older steps (s < S) change nothing


class Proposer:
    def __init__(self, i):
        self.i = i
        self.s, self.p = 4, (H, i, i)     # round 1, phase 0; the value proposed is the replica's own number
        self.replies, self.done, self.started = [], False, False


class Slot:
    """n replicas deciding one slot. delay(src, dst, rng) gives a one-way message delay in ms."""

    def __init__(self, n, seed, delay, leader=0, start=None, crash=None):
        self.n, self.q = n, n // 2 + 1
        self.rng, self.delay, self.leader = random.Random(seed), delay, leader
        self.start = start or [0.0] * n            # when each proposer wakes up (its hedging delay)
        self.crash = crash or {}                   # replica -> time (ms) at which it stops for good
        self.rec = [Recorder() for _ in range(n)]
        self.prop = [Proposer(i) for i in range(n)]
        self.queue, self.seq, self.msgs = [], 0, 0
        self.decisions = []                        # (time, proposer, step, value)

    # the simulated network
    def at(self, t, fn, *args):
        self.seq += 1
        heapq.heappush(self.queue, (t, self.seq, fn, args))

    def alive(self, i, t):
        return t < self.crash.get(i, float("inf"))

    def net(self, t, src, dst, fn, *args):
        if not self.alive(src, t):
            return
        d = 0.0 if src == dst else self.delay(src, dst, self.rng)
        self.msgs += src != dst
        self.at(t + d, self.deliver, dst, fn, args)

    def deliver(self, t, dst, fn, args):
        if self.alive(dst, t):
            fn(t, *args)

    # Algorithm 4, the proposer
    def wake(self, t, i):
        pr = self.prop[i]
        if not pr.done:
            pr.started = True
            self.send_step(t, pr)

    def send_step(self, t, pr):
        pr.replies = []
        for j in range(self.n):
            p = pr.p
            if pr.s % 4 == 0 and (pr.s > 4 or self.leader != pr.i):
                p = (self.rng.randrange(1, H), p[1], p[2])      # a fresh random priority for each recorder
            self.net(t, pr.i, j, self.on_record, pr, j, pr.s, p)

    def on_record(self, t, pr, j, s, p):
        self.net(t, j, pr.i, self.on_reply, pr, s, self.rec[j].record(s, p))

    def on_reply(self, t, pr, s, reply):
        if pr.done or s != pr.s or len(pr.replies) >= self.q:
            return                                              # stale, or the quorum was already used
        pr.replies.append(reply)
        if len(pr.replies) < self.q:
            return
        R, phase = pr.replies, pr.s % 4
        if all(step == pr.s for step, _, _ in R):
            if phase == 0:
                if all(f[0] == H for _, f, _ in R):             # fast path: every recorder saw the leader first
                    return self.decide(t, pr, R[0][1])
                pr.p = max(f for _, f, _ in R)                  # adopt the best first value
            elif phase == 2:
                if pr.p == max(a for _, _, a in R):             # nothing better was spread in phase 1
                    return self.decide(t, pr, pr.p)
            elif phase == 3:
                pr.p = max(a for _, _, a in R)                  # carry the best summary into the next round
            pr.s += 1
        else:
            step, f, _ = max(R, key=lambda r: r[0])             # a recorder is ahead of us: catch up
            pr.s, pr.p = step, f
        self.send_step(t, pr)

    def decide(self, t, pr, p):
        pr.done = True
        self.decisions.append((t, pr.i, pr.s, p[2]))
        for k in range(self.n):                                 # tell everyone, so late proposers stand down
            if k != pr.i:
                self.net(t, pr.i, k, self.learn, k)

    def learn(self, t, k):
        self.prop[k].done = True

    def run(self, limit=60_000.0):
        for i in range(self.n):
            if i != self.leader or self.alive(i, 0.0):
                self.at(self.start[i], self.wake, i)
        while self.queue:
            t, _, fn, args = heapq.heappop(self.queue)
            if t > limit:
                break
            fn(t, *args)
        first = min(self.decisions) if self.decisions else None
        return {"first": first, "values": {d[3] for d in self.decisions}, "decisions": self.decisions,
                "msgs": self.msgs, "started": sum(p.started for p in self.prop)}


def wan(base=90.0, sigma=0.10, slow=(), extra=500.0):
    """One-way delay: base ms with log-normal jitter; replicas in `slow` add up to `extra` ms on every send."""
    def delay(src, dst, rng):
        d = base * rng.lognormvariate(0.0, sigma)
        return d + (rng.uniform(0.0, extra) if src in slow else 0.0)
    return delay

Two runs make sure the translation is faithful.

Check one: four single slots

The first replays four cases with no jitter, one-way delay exactly 90 ms, so the round trip is 180 ms.

# step01_one_slot.py
"""step01_one_slot.py: four single-slot runs with no jitter (every one-way delay is exactly 90 ms, so RTT = 180 ms)."""
from quepaxa import Slot, wan

CASES = [
    ("leader alone (others wait 1 s)",      dict(start=[0, 1000, 1000, 1000, 1000])),
    ("leader, hedge 5 ms",                  dict(start=[0, 5, 5, 5, 5])),
    ("leader crashed, hedge 50 ms",         dict(start=[0, 50, 50, 50, 50], crash={0: 0.0})),
    ("no leader, all five start at once",   dict(start=[0, 0, 0, 0, 0], leader=None)),
]
print(f"{'case':<36}{'first decision':>15}{'step':>6}{'phase':>6}{'value':>6}{'messages':>9}{'proposers':>10}")
for name, kw in CASES:
    r = Slot(5, seed=1, delay=wan(sigma=0.0), **kw).run()
    t, who, step, value = r["first"]
    print(f"{name:<36}{t:>12.0f} ms{step:>6}{step % 4:>6}{value:>6}{r['msgs']:>9}{r['started']:>10}")

case                                 first decision  step phase value messages proposers
leader alone (others wait 1 s)               180 ms     4     0     0       12         1
leader, hedge 5 ms                           540 ms     6     2     0      140         5
leader crashed, hedge 50 ms                  590 ms     6     2     2      100         4
no leader, all five start at once            540 ms     6     2     1      140         5

The fast path decides in one round trip (180 ms) with 12 messages: 4 requests, 4 replies and 4 decision notices. Without it, a slot decides in phase 2, three round trips (540 ms), which matches the Meerkat announcement’s “one to three round trips”.

Check two: 20,000 random slots

The second run throws 20,000 random scenarios at it: jitter, slow replicas, hedging delays from 0 to 1,000 ms, and up to two of five replicas crashing at random times.

# step02_safety.py
"""step02_safety.py: 20,000 random slots with jitter, slow replicas, hedging delays and up to two crashes."""
import random
from quepaxa import Slot, wan

N = 20_000
bad, undecided, rounds_seen, fast = 0, 0, {}, 0
for seed in range(N):
    g = random.Random(seed)
    slow = g.sample(range(5), g.choice([0, 1, 2]))
    delay = wan(base=g.choice([1.0, 90.0]), sigma=g.choice([0.05, 0.5, 1.0]), slow=slow, extra=g.choice([0.0, 500.0]))
    d = g.choice([0.0, 5.0, 50.0, 180.0, 1000.0])
    crashed = g.sample(range(5), g.choice([0, 1, 2]))
    crash = {i: g.choice([0.0, g.uniform(0, 400)]) for i in crashed}
    r = Slot(5, seed, delay, leader=g.choice([0, None]), start=[0.0] + [d] * 4, crash=crash).run()
    if not r["decisions"]:
        undecided += 1
        continue
    if len(r["values"]) != 1 or next(iter(r["values"])) not in range(5):    # agreement and validity
        bad += 1
    step = r["first"][2]
    fast += step == 4
    rounds_seen[step // 4] = rounds_seen.get(step // 4, 0) + 1
print(f"slots run                      : {N}")
print(f"slots with two different values: {bad}")
print(f"slots where nobody decided     : {undecided}")
print(f"first decision on the fast path: {fast}")
print("first decision by round        :", dict(sorted(rounds_seen.items())))

slots run                      : 20000
slots with two different values: 0
slots where nobody decided     : 0
first decision on the fast path: 5091
first decision by round        : {1: 19399, 2: 568, 3: 31, 4: 2}

No slot ever decided two values, and every slot decided. That does not prove the protocol, which the paper does, but it makes a translation slip in the code above unlikely.

The 50 percent per round, and “exponentially easier”

What the paper guarantees

The paper’s sentence is “Rounds 2 and higher in each slot are leaderless and fully asynchronous, each round guaranteed to succeed independently with probability at least 1/2.” Its liveness theorem says the abstract protocol terminates “in less than two rounds in expectation”, and the argument rests on the network scheduler being “content-oblivious”, unaware of the priorities attached to proposals. Section 4.2.5 adds that a strong network adversary can always prevent a leader-based round from succeeding, which is why QuePaxa “uses a leader only in the first round of any slot”.

So the post’s “50% chance” is the paper’s bound, correctly reported. Note what kind of statement it is. It is a floor against an adversary that picks message order without seeing the random numbers, and it holds in every round independently, which means the chance of still being undecided after r rounds is at most 2 to the power of minus r. It says nothing about rounds getting easier.

What the simulator shows

Five proposers start at the same moment with no leader, the case with the most interference. The script records the round in which the first decision lands, under four delay models, 20,000 slots each.

# step03_rounds.py
"""step03_rounds.py: five proposers start together (no leader). In which round does the first decision land?"""
import random
from quepaxa import Slot, wan

N = 20_000
MODELS = {
    "steady   (sigma 0.02)": lambda g: wan(sigma=0.02),
    "jittery  (sigma 0.30)": lambda g: wan(sigma=0.30),
    "heavy    (sigma 0.80)": lambda g: wan(sigma=0.80),
    "attacked (2 slow, +0..500 ms)": lambda g: wan(sigma=0.10, slow=g.sample(range(5), 2), extra=500.0),
}
print(f"{N} slots per row; the first decision happens in phase 2 of its round; 'cond' = P(decided in round r | undecided before it)")
print(f"{'model':<31}{'round 1':>9}{'round 2':>9}{'round 3':>9}{'round 4+':>9}{'  cond r1':>10}{'cond r2':>9}{'cond r3':>9}{'steps/slot':>12}")
for name, make in MODELS.items():
    counts = {}
    for seed in range(N):
        g = random.Random(seed)
        r = Slot(5, seed, make(g), leader=None).run()
        rnd = r["first"][2] // 4
        counts[rnd] = counts.get(rnd, 0) + 1
    total = sum(counts.values())
    by = [counts.get(k, 0) for k in (1, 2, 3)]
    rest = total - sum(by)
    left = [total, total - by[0], total - by[0] - by[1]]
    cond = [by[k] / left[k] if left[k] else float("nan") for k in range(3)]
    steps = sum((4 * k + 2 - 3) * v for k, v in counts.items()) / total        # steps counted from phase 0 of round 1
    print(f"{name:<31}{by[0]/total:>9.4f}{by[1]/total:>9.4f}{by[2]/total:>9.4f}{rest/total:>9.4f}{cond[0]:>10.4f}{cond[1]:>9.4f}{cond[2]:>9.4f}{steps:>12.3f}")

20000 slots per row; the first decision happens in phase 2 of its round; 'cond' = P(decided in round r | undecided before it)
model                            round 1  round 2  round 3 round 4+   cond r1  cond r2  cond r3  steps/slot
steady   (sigma 0.02)             1.0000   0.0000   0.0000   0.0000    1.0000      nan      nan       3.000
jittery  (sigma 0.30)             0.9889   0.0104   0.0006   0.0001    0.9889   0.9372   0.9286       3.048
heavy    (sigma 0.80)             0.8690   0.1178   0.0121   0.0011    0.8690   0.8989   0.9132       3.582
attacked (2 slow, +0..500 ms)     0.9885   0.0115   0.0000   0.0000    0.9885   1.0000      nan       3.046

The post’s “much higher” holds in all four models: the first round decided 87% of slots at worst (heavy-tailed delays) and 100% when delays barely vary. The claim that deciding gets “exponentially easier” as rounds progress does not show up. Given that a slot reached round 2, the chance it decided there was 0.94 in the jittery model and 0.90 in the heavy one, and in round 3 it was 0.93 and 0.91. In the jittery model the rate falls from 0.99 in round 1; in the heavy one it creeps up from 0.87 to 0.91. That is a roughly flat, high success rate, not a climb toward certainty. What shrinks geometrically is the share of slots left undecided, which is the paper’s claim. These are random delays, not the adversarial scheduler in the proof, so they cannot test the worst case, but within random delays like these they do not show a rising per-round chance.

The five-millisecond hedge

The post sets Raft’s fate against QuePaxa’s: “In Raft, having a five-millisecond timeout in a network with 100-millisecond latency would completely freeze the system. But in QuePaxa, a five-millisecond hedge timeout only increases the number of active proposers.” It goes on to concede that “we can react faster to leader changes by lowering the hedge timeout, but we incur extra bandwidth costs and may take more time to decide”. How much more?

What the paper tested

The paper’s Section 7.5 uses five replicas in a wide-area setting with an average 180 ms round trip and 25,000 commands per second. Its Figure 9 plots hedging delays from 50 ms upward, so the smallest hedge tested is 0.28 of the round trip, not 0.03. The abstract’s claim is that QuePaxa “maintains full performance even when its hedging delays are only 1/3 the underlying round-trip time”, and that claim is about throughput: 25,000 commands per second at every hedging delay. The paper says “the main cost of hedging in QuePaxa is increased bandwidth use”, and that “Choosing” the base delay “too small can cause too many proposers to activate” and revert to quadratic message cost. The third panel of Figure 9, average steps per slot, reads about 2.8 at 50 ms, about 1.1 at 100 ms and 1.0 from 150 ms on (my reading of the plotted points; the paper prints no table). Its text a paragraph earlier says that when the hedge is below the round trip the leader still wins “most slots and commits in one round-trip”.

What the simulator shows

The paper’s Section 5.2 says the schedule gives the second proposer a delay of δ, the third 2δ and so on, while Section 7.5 says each non-leader “waits a short time”. I ran both: “equal” (all four non-leaders wait δ) and “stagger” (δ, 2δ, 3δ, 4δ). A hedged proposer proposes unless it has learned the slot is decided, the most pessimistic reading of the paper’s rule that a proposer acts only if it “has not by then seen evidence that some other proposer” has already driven the step to completion. The leader is alive. The network mimics the paper’s wide-area setting: 90 ms one way (a 180 ms round trip), 10% jitter, n = 5.

# step04_hedge.py
"""step04_hedge.py: the leader is alive. How does the hedging delay change the fast path, latency and message cost?
RTT is 180 ms (one-way 90 ms, 10% log-normal jitter), n = 5. 'equal': every non-leader waits delta.
'stagger': the k-th non-leader waits k * delta (Section 5.2 of the paper)."""
from quepaxa import Slot, wan

N = 3000
DELTAS = [5, 18, 30, 50, 60, 75, 90, 100, 135, 180, 270, 360]
print(f"{N} slots per cell. steps/slot counts the decision step from the start of round 1 (fast path = 1, phase 2 = 3)")
print(f"{'schedule':<9}{'delta ms':>9}{'delta/RTT':>10}{'fast path':>10}{'leader wins':>12}{'steps/slot':>11}{'median ms':>10}{'p99 ms':>8}{'messages':>9}{'proposers':>10}")
for mode in ("equal", "stagger"):
    for d in DELTAS:
        starts = [0.0] + [d * (k if mode == "stagger" else 1) for k in range(1, 5)]
        lat, fast, wins, steps, msgs, started = [], 0, 0, 0, 0, 0
        for seed in range(N):
            r = Slot(5, seed, wan(), leader=0, start=starts).run()
            t, _, step, value = r["first"]
            lat.append(t)
            fast += step == 4
            wins += value == 0
            steps += step - 3
            msgs += r["msgs"]
            started += r["started"]
        lat.sort()
        print(f"{mode:<9}{d:>9}{d/180:>10.2f}{fast/N:>10.3f}{wins/N:>12.3f}{steps/N:>11.2f}{lat[N//2]:>10.0f}{lat[int(N*0.99)]:>8.0f}{msgs/N:>9.1f}{started/N:>10.2f}")
    print()

3000 slots per cell. steps/slot counts the decision step from the start of round 1 (fast path = 1, phase 2 = 3)
schedule  delta ms delta/RTT fast path leader wins steps/slot median ms  p99 ms messages proposers
equal            5      0.03     0.000       1.000       3.00       523     548    148.0      5.00
equal           18      0.10     0.000       1.000       3.00       528     553    147.9      5.00
equal           30      0.17     0.000       1.000       3.00       530     555    147.8      5.00
equal           50      0.28     0.000       1.000       3.00       530     560    147.2      5.00
equal           60      0.33     0.000       1.000       3.00       530     561    146.1      5.00
equal           75      0.42     0.004       1.000       2.99       530     560    141.4      5.00
equal           90      0.50     0.543       1.000       1.91       252     559     92.3      5.00
equal          100      0.56     0.956       1.000       1.09       177     546     53.4      5.00
equal          135      0.75     1.000       1.000       1.00       177     194     44.0      5.00
equal          180      1.00     1.000       1.000       1.00       177     194     44.0      5.00
equal          270      1.50     1.000       1.000       1.00       177     194     24.6      2.58
equal          360      2.00     1.000       1.000       1.00       177     194     12.0      1.00

stagger          5      0.03     0.000       1.000       3.00       526     551    148.0      5.00
stagger         18      0.10     0.000       1.000       3.00       530     558    146.3      5.00
stagger         30      0.17     0.157       1.000       2.69       527     558    124.2      5.00
stagger         50      0.28     0.631       1.000       1.74       268     555     85.3      5.00
stagger         60      0.33     0.606       1.000       1.79       284     556     84.7      5.00
stagger         75      0.42     0.603       1.000       1.79       189     555     75.8      4.48
stagger         90      0.50     0.857       1.000       1.29       178     553     45.6      3.62
stagger        100      0.56     0.986       1.000       1.03       177     534     30.4      3.03
stagger        135      0.75     1.000       1.000       1.00       177     194     23.2      2.40
stagger        180      1.00     1.000       1.000       1.00       177     194     20.0      2.00
stagger        270      1.50     1.000       1.000       1.00       177     194     15.1      1.39
stagger        360      2.00     1.000       1.000       1.00       177     194     12.0      1.00

Four things stand out.

The knee is at the one-way delay, not the round trip. A hedged proposer writes its own proposal to its own recorder at once. If that happens before the leader’s message arrives (90 ms), the recorder’s first value for step 4 is not the leader’s, and the leader’s fast-path test fails for every majority that includes it. With equal waits the fast path was 0% up to 60 ms, 0.4% at 75 ms, 54% at 90 ms, 96% at 100 ms and 100% from 135 ms.
Below the knee every slot costs three round trips. Median latency was 523 to 530 ms against 177 ms, with the p99 between 548 and 561 ms. The leader’s value still won every slot (the “leader wins” column), so ordering is preserved, but the one-round-trip commit is gone.
The message bill is twelvefold at n = 5. 148 messages per slot against 12. The paper’s O(n) normal case becomes its quadratic worst case, and the cost grows with the replica count.
It matches the paper’s own panel. The equal schedule gives 3.00 steps per slot at 50 ms and 1.09 at 100 ms, against about 2.8 and 1.1 read off Figure 9. That agreement is the best evidence that the model is close to what the authors measured, and it also shows that the paper’s sentence about committing “in one round-trip” holds for hedges between roughly half the round trip and the round trip, not at 50 ms.

The staggered schedule softens the cliff (63% fast path at 50 ms, 16% at 30 ms) but it still reaches 148 messages per slot at 5 ms and has a p99 near 555 ms from 30 to 90 ms. So the post’s five milliseconds in a 100 ms network is outside the range the paper tested, and in this model it costs every slot its fast path. That is a price, and it is the one the post names, but “only increases the number of active proposers” does not convey that it triples latency and multiplies traffic by twelve. One caveat: the post says Meerkat’s clients wait the hedge timeout “instead of the replicas”. That changes when each replica’s proposer starts relative to the leader’s message, so the knee could sit elsewhere; I did not model it.

“Recover as fast as we want”

The post says “we can recover as fast as we want”, because the hedge can be set small. The simulator measures a slot opened while the leader is already down. It is not the paper’s recovery metric. The paper times how long until “a new leader or alternate proposer” resumes progress under a steady load of 25,000 commands per second, and its 303 to 473 ms is below three round trips (540 ms). My reading is that slots already in flight when the leader died can finish sooner than a fresh slot, but the paper does not say.

# step05_leader_down.py
"""step05_leader_down.py: the leader is crashed when the slot opens. How long does a slot take, by hedging delay?
RTT 180 ms, n = 5 (four replicas left), 10% jitter. Same two schedules as step04."""
from quepaxa import Slot, wan

N = 3000
DELTAS = [0, 5, 18, 50, 90, 180, 360, 900]
print(f"{N} slots per cell; replica 0 (the leader) is down from t = 0")
print(f"{'schedule':<9}{'delta ms':>9}{'median ms':>10}{'p99 ms':>8}{'median / RTT':>13}{'messages':>9}{'proposers':>10}{'round 2+':>9}")
for mode in ("equal", "stagger"):
    for d in DELTAS:
        starts = [0.0] + [d * (k if mode == "stagger" else 1) for k in range(1, 5)]
        lat, msgs, started, later = [], 0, 0, 0
        for seed in range(N):
            r = Slot(5, seed, wan(), leader=0, start=starts, crash={0: 0.0}).run()
            t, _, step, _ = r["first"]
            lat.append(t)
            msgs += r["msgs"]
            started += r["started"]
            later += step // 4 > 1
        lat.sort()
        print(f"{mode:<9}{d:>9}{lat[N//2]:>10.0f}{lat[int(N*0.99)]:>8.0f}{lat[N//2]/180:>13.2f}{msgs/N:>9.1f}{started/N:>10.2f}{later/N:>9.3f}")
    print()

3000 slots per cell; replica 0 (the leader) is down from t = 0
schedule  delta ms median ms  p99 ms median / RTT messages proposers round 2+
equal            0       530     560         2.94    103.0      4.00    0.000
equal            5       535     565         2.97    103.0      4.00    0.000
equal           18       548     578         3.04    103.0      4.00    0.000
equal           50       580     610         3.22    103.0      4.00    0.000
equal           90       620     650         3.44    103.0      4.00    0.000
equal          180       710     740         3.94    103.0      4.00    0.000
equal          360       890     920         4.94    103.0      4.00    0.000
equal          900      1430    1460         7.94    103.0      4.00    0.000

stagger          0       530     560         2.94    103.0      4.00    0.000
stagger          5       541     571         3.01    103.0      4.00    0.000
stagger         18       563     611         3.13    102.8      4.00    0.001
stagger         50       597    1296         3.32     97.7      4.00    0.014
stagger         90       636    1402         3.53     95.7      4.00    0.064
stagger        180       722    1463         4.01     70.7      4.00    0.030
stagger        360       901     937         5.01     39.0      2.00    0.000
stagger        900      1442    1478         8.01     25.0      1.00    0.000

With a hedge of zero the median slot took 530 ms, 2.94 round trips, and each extra millisecond of hedge adds a millisecond. Three round trips is the floor of the leaderless path, which the July post also states: “one if the leader is proposing” and “three if a non-leader is proposing”. The staggered schedule has a tail: at hedges of 50 to 180 ms, 1.4% to 6.4% of slots needed a second round and the p99 reached 1.3 to 1.5 seconds, because late-starting proposers join a step in progress.

Raft’s side comes from the Raft tutorial’s simulator on the same network (one-way delay 85 to 95 ms, five servers). Timeouts are drawn from the range shown, the heartbeat is a fifth of the low end, and each row uses 30 seeds over 30 simulated seconds. The last column crashes a settled leader and times the first command committed by its successor.

# step06_raft_timeouts.py
"""step06_raft_timeouts.py: the Raft tutorial's simulator with the same WAN: one-way delay 85 to 95 ms (RTT about 180 ms).
Election timeouts are drawn from [low, high]; the heartbeat is low / 5. 30 simulated seconds, 30 seeds per row."""
import statistics
from cluster import Cluster

RTT = 180
ROWS = [(5, 10), (90, 180), (180, 360), (324, 648), (900, 1800), (1800, 3600)]
SEEDS, SPAN = 30, 30_000
print(f"{'timeout ms':<12}{'low/RTT':>8}{'first leader':>14}{'time with a leader':>20}{'leader changes':>16}{'crash to commit':>17}")
for low, high in ROWS:
    firsts, shares, changes, recov = [], [], [], []
    for seed in range(SEEDS):
        c = Cluster(5, seed, delay=(85, 95), election_timeout=(low, high), heartbeat=max(1, low // 5), check=False)
        have, terms, first, t = 0, set(), None, 0
        while c.now < SPAN:
            c.run_for(100)
            lead = c.leaders()
            if lead:
                have += 1
                terms.add((lead[0].id, lead[0].current_term))
                first = first if first is not None else c.now
        firsts.append(first)
        shares.append(have / (SPAN / 100))
        changes.append(max(len(terms) - 1, 0))
        # crash test on a fresh cluster: wait for a settled leader, crash it, time the first commit by the next one
        c = Cluster(5, seed, delay=(85, 95), election_timeout=(low, high), heartbeat=max(1, low // 5), check=False)
        if not c.run_until_leader(SPAN):
            continue
        c.run_for(max(4 * RTT, 2 * high))
        lead = c.leaders()
        if not lead:
            continue
        old, t0 = lead[0], c.now
        c.crash(old.id)
        if c.run_until(t0 + SPAN, stop=lambda: any(n.id != old.id for n in c.leaders())):
            new = [n for n in c.leaders() if n.id != old.id][0]
            idx = c.client(new.id, "x")
            if idx and c.run_until(t0 + SPAN, stop=lambda: new.commit_index >= idx):
                recov.append(c.now - t0)
    done = [f for f in firsts if f is not None]
    first_txt = f"{statistics.median(done)/1000:>11.1f} s" if done else f"{'none':>13}"
    rec_txt = f"{statistics.median(recov):>14.0f} ms" if recov else f"{'none':>17}"
    print(f"{f'{low}-{high}':<12}{low/RTT:>8.2f}{first_txt:>14}{statistics.mean(shares):>19.1%}{statistics.mean(changes):>16.1f}{rec_txt}")

timeout ms   low/RTT  first leader  time with a leader  leader changes  crash to commit
5-10            0.03          none               0.0%             0.0             none
90-180          0.50        11.9 s               0.5%             0.6             none
180-360         1.00         0.6 s              97.3%             0.5           895 ms
324-648         1.80         0.6 s              98.1%             0.0           834 ms
900-1800        5.00         1.3 s              95.9%             0.0          1542 ms
1800-3600      10.00         2.3 s              92.2%             0.0          2578 ms

Putting the two side by side, with QuePaxa at hedges of 0 to 180 ms (530 to 710 ms):

Raft election timeout	Multiple of the 180 ms round trip	Crash to first commit	Compared with QuePaxa’s 530 to 710 ms
324 to 648 ms	1.8 to 3.6	834 ms	1.2 to 1.6 times slower
900 to 1,800 ms	5 to 10 (the post’s “commonly deployed”)	1,542 ms	2.2 to 2.9 times slower
1,800 to 3,600 ms	10 to 20 (etcd’s “at least 10 times”)	2,578 ms	3.6 to 4.9 times slower

So the recovery advantage is real, and large against conservatively tuned Raft, but it is bounded below by three round trips plus the hedge and it is small against a Raft tuned near the paper’s stability limit. “As fast as we want” describes the hedge, not the slot.

Raft under short timeouts, outages and one bad link

The freeze is real

The first rows of the table above confirm the post’s five-millisecond claim and extend it. At timeouts of 5 to 10 ms no cluster ever showed a leader, and at 90 to 180 ms (half to one round trip) leaders existed for 0.5% of the time. From 1 round trip up, a leader existed 92% to 98% of the time, the missing share being the first election. The paper’s measurement, with its own Raft implementation on real wide-area links, put the stability line higher, at about 1.8 times the round trip. That could reflect a noisier network than my 85 to 95 ms or implementation differences (the paper’s Raft is a multi-threaded gRPC design); I modeled neither.

The storm was not reproduced

A two-second outage

The post’s deeper argument against leaders is that election failures feed themselves: “The outage will be over in two seconds, but the election can continue for much longer as candidates continue to block each other.” I tested the direct version. A settled cluster loses its leader (cut off from the other four) or loses every link for two seconds, then the network heals; the script watches 30 seconds.

# step08_raft_outage.py
"""step08_raft_outage.py: does a two-second outage start a self-sustaining election storm in textbook Raft?
Five servers, one-way delay 85 to 95 ms (RTT about 180 ms). After a leader has settled, either the leader is cut off
from the other four ('leader') or every server is cut off from every other ('all') for 2 s; then the network heals.
We then watch 30 s: seconds without a leader, and how many different (leader, term) pairs appear."""
import statistics
from cluster import Cluster

RTT, SEEDS, OUT, WATCH = 180, 30, 2000, 30_000
ROWS = [(90, 180), (180, 360), (324, 648), (900, 1800), (1800, 3600)]
print(f"{'timeout ms':<12}{'outage':<8}{'no leader (s)':>14}{'worst (s)':>10}{'leaders seen':>13}{'storms':>8}")
for low, high in ROWS:
    for kind in ("leader", "all"):
        gap, seen, storms = [], [], 0
        for seed in range(SEEDS):
            c = Cluster(5, seed, delay=(85, 95), election_timeout=(low, high), heartbeat=max(1, low // 5), check=False)
            if not c.run_until_leader(60_000):
                continue
            c.run_for(max(4 * RTT, 2 * high))
            lead = c.leaders()
            if not lead:
                continue
            old = lead[0].id
            others = set(c.nodes) - {old}
            c.partition({old}, others) if kind == "leader" else c.partition(*[{i} for i in c.nodes])
            c.run_for(OUT)
            c.heal()
            idle, pairs, t = 0, set(), 0
            while t < WATCH:
                c.run_for(100)
                t += 100
                now = c.leaders()
                idle += not now
                pairs.update((n.id, n.current_term) for n in now)
            gap.append(idle / 10)
            seen.append(len(pairs))
            storms += idle / 10 > 10                      # more than 10 s without a leader after the network healed
        if gap:
            print(f"{f'{low}-{high}':<12}{kind:<8}{statistics.median(gap):>14.1f}{max(gap):>10.1f}{statistics.mean(seen):>13.1f}{storms:>8}")
        else:
            print(f"{f'{low}-{high}':<12}{kind:<8}{'no settled leader':>45}")

timeout ms  outage   no leader (s) worst (s) leaders seen  storms
90-180      leader                              no settled leader
90-180      all                                 no settled leader
180-360     leader             0.0       0.9          2.0       0
180-360     all                0.2       2.2          2.2       0
324-648     leader             0.0       0.0          1.7       0
324-648     all                0.2       0.4          1.9       0
900-1800    leader             0.0       0.1          2.0       0
900-1800    all                0.2       1.6          2.0       0
1800-3600   leader             0.0       0.1          1.9       0
1800-3600   all                0.1       2.5          1.9       0

Across 30 runs per row, no run went more than 2.5 seconds without a leader after the network healed, and on average about two leaders appeared (the old one and its successor). At 90 to 180 ms no cluster settled before the outage, so those rows are empty. Randomized timeouts do their job.

One cut link

The 2020 failure had a different shape: one follower lost its link to the leader but kept its link to the third server. I reproduced that shape with three servers and a 60-second cut.

# step09_raft_one_link.py
"""step09_raft_one_link.py: the 2020 shape of failure. Three servers; one follower (F) loses its link to the leader (L)
but still reaches the third server (M). The cut stays in place for 60 s. Textbook Raft, no leader lease, no pre-vote."""
import statistics
from cluster import Cluster

RTT, SEEDS, WATCH = 180, 30, 60_000
print(f"{'timeout ms':<12}{'time with a leader':>20}{'leader changes':>16}{'longest gap (s)':>17}{'last change (s)':>17}{'M leads at end':>16}")
for low, high in [(180, 360), (324, 648), (900, 1800), (1800, 3600)]:
    shares, changes, gaps, last, m_leads = [], [], [], [], 0
    for seed in range(SEEDS):
        c = Cluster(3, seed, delay=(85, 95), election_timeout=(low, high), heartbeat=max(1, low // 5), check=False)
        if not c.run_until_leader(60_000):
            continue
        c.run_for(max(4 * RTT, 2 * high))
        lead = c.leaders()
        if not lead:
            continue
        L = lead[0].id
        F, M = [i for i in c.nodes if i != L]
        c.partition({L, M}, {M, F})                       # L-M and M-F work; L-F is cut
        have, pairs, gap, worst, t, stamp = 0, set(), 0, 0, 0, 0
        while t < WATCH:
            c.run_for(100)
            t += 100
            now = c.leaders()
            have += bool(now)
            gap = 0 if now else gap + 100
            worst = max(worst, gap)
            new = {(n.id, n.current_term) for n in now} - pairs
            if new:
                stamp = t
            pairs |= new
        shares.append(have / (WATCH / 100))
        changes.append(len(pairs) - 1)
        gaps.append(worst / 1000)
        last.append(stamp / 1000)
        m_leads += any(n.id == M for n in c.leaders())
    print(f"{f'{low}-{high}':<12}{statistics.mean(shares):>19.1%}{statistics.mean(changes):>16.1f}{max(gaps):>17.1f}"
          f"{statistics.median(last):>17.1f}{m_leads:>13} of {len(shares)}")

timeout ms    time with a leader  leader changes  longest gap (s)  last change (s)  M leads at end
180-360                   98.9%             2.4              1.3              1.6           28 of 28
324-648                   98.9%             2.2              1.5              1.9           30 of 30
900-1800                  97.3%             2.3              2.6              4.7           30 of 30
1800-3600                 94.9%             2.4              3.3              8.6           30 of 30

Textbook Raft changed leaders about twice in the minute, with a leader present 95% to 99% of the time. In every run that had settled before the cut (28 or 30 per row) the server that can talk to both ended as leader, and the last leader change came a median of 1.6, 1.9, 4.7 and 8.6 seconds after the cut for the four timeout ranges. The 2020 post-mortem describes a cluster that stayed read-only until the switch recovered, so whatever made that incident worse was in the details of that deployment (version, configuration, load), which the post-mortem does not give. This does not show that leaders are free. It shows that the specific storm in the post depends on more than “Raft elections interfere”, and it is why the Raft side of the argument should be read as “badly tuned or partially broken Raft”, not “Raft”.

The 2020 outage, as its own post-mortem tells it

The post supports its case with one sentence: “in 2020, the partial failure of a core Cloudflare network switch caused a consensus-powered system to degrade and the dashboard to operate 80x slower than usual, for six hours. Critically, the system lost no data. But it became unavailable for writes until the issue was resolved, which is unacceptable.” The post-mortem, “A Byzantine failure in the real world” (Tom Lianza and Chris Snook, November 27, 2020), tells a longer story.

Time (UTC, 2020-11-02)	What the post-mortem says
14:43	A network switch partly fails; some control protocols keep working while the data plane drops packets.
14:44	One etcd node (node 1) cannot reach the leader (node 3) but can reach node 2. It “repeatedly initiated leader elections” while the third “repeatedly voted for” the leader, which “made the cluster read-only until the faulty switch recovered”.
14:45	With etcd read-only, two database clusters cannot report a healthy primary, so a replica is promoted automatically, “without error or data loss”. A defect in the cluster manager “requires a rebuild of all database replicas” after a promotion.
After 14:45	For the cluster behind API and dashboard authentication, a replica was needed to carry the read load. With none online the primary was overloaded: “This is when the main impact started.” Rebuilding “would take several hours”.
During the incident	API reads were steered to the secondary data center. Dashboard sessions could not follow: “we were unfortunately making the user experience on the dashboard worse.”
21:20	“The instant the first database replica rebuilt, it put itself back into service, and performance resumed to normal levels.”

The incident lasted six hours and 33 minutes; the API success rate dipped to 75% and the dashboard was “as much as 80 times slower than normal”. So the post’s facts are right and its causation is compressed. Consensus unavailability was the trigger, in the first minute. What turned it into a six-hour incident was an automatic failover, a replica-rebuild defect, and a dashboard that could not be moved across regions. The post-mortem’s “without error or data loss” is about the database promotion. The post-mortem does not say when etcd accepted writes again, so “unavailable for writes until the issue was resolved” cannot be checked against it. The post-mortem’s own postscript adds that the community considered the failure “better characterized as an omission fault rather than a Byzantine fault”, a more specific class that, the post-mortem says, “can be tolerated without BFT protocols”. None of this says QuePaxa would not have helped with the trigger. It says the six hours were mostly not consensus.

The Meerkat post’s 10x

The July 8 announcement says QuePaxa suits networks “in which an imaginary adversary can launch targeted attacks on replica connections” and that the authors “found that it maintains much higher (~10x) throughput than Raft and Multi-Paxos during such conditions”. The paper’s attack experiment (Section 7.4) uses five replicas on a wide-area network and an attacker who “increases the egress packet latency of a minority of replicas up to 500ms dynamically in 5s time epochs”. Its text reads: “QuePaxa sustains a throughput of at least 75k cmd/sec under 380ms median latency. In contrast, the throughput of Multi-Paxos and Raft saturate at 2.5k cmd/sec.” That is at least 30 times, and the “at least” matters because the throughput axis of the paper’s Figure 8 stops at 80,000. I searched the paper’s text for a tenfold figure and found none. The Meerkat post may be summarizing another figure, or rounding conservatively, but the number it attributes to “the authors” is not the authors’ headline number.

Reading Cloudflare’s decision-rate chart

The post’s only measurement is a two-panel chart: relative decision rate (top) and the share of decisions by phase (bottom), with markers for “leader failure” at 15:00:32 and “leader replaced” at 15:01:25. The caption says “The decision rate decreases to around 65% of its original rate, and returns to 100% when the faulty leader is replaced, a minute after failing”, and the text says throughput drops “down to at most a third of the usual rate” when a decision needs three phases. I calibrated the axes from the printed tick labels and read the purple line and the blue and orange areas by pixel color.

Download the image from the post and run the script beside it.

curl -sL -o quepaxa_chart.png "https://blog.cloudflare.com/_emdash/api/media/file/01M73MN2G9VJ180TS373QNGVJJ.png"
python step07_chart_readout.py quepaxa_chart.png

# step07_chart_readout.py
"""step07_chart_readout.py: read the numbers off Cloudflare's decision-rate chart (the PNG in the QuePaxa post).
Axes were calibrated from the printed tick labels: x = 160.2 px per minute with 14:59 at x = 267;
top panel: 100% at y = 173 and 25% at y = 447; bottom panel: 100% at y = 722 and 0% at y = 991."""
import sys
from PIL import Image

PNG = sys.argv[1] if len(sys.argv) > 1 else "quepaxa_chart.png"
im = Image.open(PNG).convert("RGB")
PURPLE, BLUE, ORANGE = (109, 63, 160), (70, 127, 208), (246, 162, 91)


def near(c, ref, tol):
    return abs(c[0] - ref[0]) + abs(c[1] - ref[1]) + abs(c[2] - ref[2]) < tol


def clock(x):                                   # seconds after 14:59:00
    return (x - 267) / 160.2 * 60


def hms(sec):
    sec = int(round(sec)) + 14 * 3600 + 59 * 60
    return f"{sec // 3600}:{sec // 60 % 60:02d}:{sec % 60:02d}"


rate, share2 = {}, {}
for x in range(180, 1895):
    ys = [y for y in range(140, 541) if near(im.getpixel((x, y)), PURPLE, 45)]
    if ys:
        rate[x] = 100 - (sum(ys) / len(ys) - 173) / ((447 - 173) / 75)
    px = [im.getpixel((x, y)) for y in range(716, 992)]
    blue, orange = sum(near(c, BLUE, 40) for c in px), sum(near(c, ORANGE, 40) for c in px)
    share2[x] = 100 * blue / (blue + orange)      # dashed marker lines hide a few pixels, so use the ratio

fail, repl = 513, 655                           # the two dashed vertical lines
base = [v for x, v in rate.items() if x < fail]
print(f"leader failure at {hms(clock(fail))}, leader replaced at {hms(clock(repl))}: {clock(repl) - clock(fail):.0f} s apart")
print(f"rate before the failure: mean {sum(base) / len(base):.1f}% (range {min(base):.1f} to {max(base):.1f})")
xm = min((x for x in rate if fail <= x <= 700), key=lambda x: rate[x])
print(f"lowest rate: {rate[xm]:.1f}% at {hms(clock(xm))}")
print(f"rate at the 'leader replaced' line: {rate[repl]:.1f}%;  at 15:09: {rate[1890]:.1f}%")


def first(series, test, lo, hi):
    return next((x for x in range(lo, hi) if x in series and test(series[x])), None)


print("\nseconds after the failure line until the rate first drops below:")
for lim in (95, 90, 80, 70, 66):
    x = first(rate, lambda v: v < lim, fail, 700)
    print(f"  {lim}%: {'never' if x is None else f'{clock(x) - clock(fail):.0f} s'}")
x = first(rate, lambda v: v > 95, repl, 1000)
print(f"after the replacement the rate is back above 95% after {clock(x) - clock(repl):.0f} s")

print("\nshare of decisions made in phase 2 (3 round trips); the rest are phase 0 (1 round trip)")
print(f"{'time':>9}{'rate %':>9}{'phase 2 %':>11}")
for label, sec in [("-30 s", -30), ("failure", 0), ("+10 s", 10), ("+20 s", 20), ("+25 s", 25), ("+30 s", 30), ("+35 s", 35), ("+40 s", 40), ("+45 s", 45), ("replaced", 53), ("+60 s", 60), ("+70 s", 70), ("+100 s", 100), ("+150 s", 150)]:
    x = int(round(fail + sec * 160.2 / 60))
    print(f"{label:>9}{rate.get(x, float('nan')):>9.1f}{share2[x]:>11.1f}")
x50 = first(share2, lambda v: v > 50, fail, 700)
x90 = first(share2, lambda v: v > 90, fail, 700)
print(f"\nphase 2 share first above 50% after {clock(x50) - clock(fail):.0f} s and above 90% after {clock(x90) - clock(fail):.0f} s")

leader failure at 15:00:32, leader replaced at 15:01:25: 53 s apart
rate before the failure: mean 98.8% (range 98.1 to 99.7)
lowest rate: 64.0% at 15:01:15
rate at the 'leader replaced' line: 66.9%;  at 15:09: 108.5%

seconds after the failure line until the rate first drops below:
  95%: 1 s
  90%: 10 s
  80%: 30 s
  70%: 34 s
  66%: 41 s
after the replacement the rate is back above 95% after 34 s

share of decisions made in phase 2 (3 round trips); the rest are phase 0 (1 round trip)
     time   rate %  phase 2 %
    -30 s     98.1        0.0
  failure     97.8        0.0
    +10 s     89.6        0.0
    +20 s     89.6        1.1
    +25 s     89.6        1.1
    +30 s     78.5        1.5
    +35 s     69.1        1.5
    +40 s     67.8       22.0
    +45 s     64.0       79.3
 replaced     66.9      100.0
    +60 s     67.7       85.4
    +70 s     72.1        7.2
   +100 s     96.2        1.1
   +150 s    102.5        0.0

phase 2 share first above 50% after 43 s and above 90% after 46 s

What the readings say, to within a point or so:

The rate before the marker averaged 98.8%, fell to 64.0% at 15:01:15 and was 66.9% at the replacement marker. “Around 65%” is right, and the marker gap is 53 seconds, so “a minute” is right.
The fall was gradual, not a step: below 90% ten seconds after the failure marker, below 80% after 30 seconds. Phase 2 decisions stayed at about 1% until 35 seconds after the marker, rose to 22% at 40 seconds and 79% at 45 seconds, and reached 100% at the replacement marker. For the first half minute the chart does not look like a leader that stopped; the post does not say what fault was injected.
The rate was back above 95% 34 seconds after the replacement, and reads 108.5% at 15:09, so “returns to 100%” hides a half-minute climb and an overshoot.
When 100% of decisions were phase 2 (three round trips) the rate was about 65% to 67%, not 33%. A one-slot-at-a-time system would show a third. 65% means more than one slot in flight, or batching, or a load that was not saturating the leader. The post gives none of these. The “at most a third” is therefore a bound for sequential slots that the chart itself does not approach, and the dotted 33% line in the figure is never close.

What the post leaves out

Setup behind the chart. Replica count, regions, round trip, load, hedge delay and the kind of failure are all missing. Without them the 95% fast-path figure (“upwards of 95%”) cannot be compared with anything, and my simulator says the fast-path share depends on the hedge relative to the one-way delay, from 0% to 100%.
A deviation from the paper. Meerkat has “clients wait the hedge timeout instead of the replicas”, “to reduce the amount of messages”. The paper’s Figure 9 numbers, and the simulator above, are for replica-side waiting.
The assumptions behind the 50%. The paper calls its protocol “asynchronous crash-stop”, and its liveness bound assumes a network that cannot see proposal priorities; a footnote warns that side channels “could compromise” that assumption. The July post says Meerkat does not handle Byzantine faults; neither post mentions the scheduler assumption.
Scale. The paper’s hedging and attack experiments use five replicas. The July post reports proofs of concept with “up to 50 replicas”. With every proposer active, the paper’s cost is quadratic, and the simulator’s 12 to 148 messages at n = 5 is the same effect at small scale.
The fast path exists only in round 1. The paper says QuePaxa “uses a leader only in the first round of any slot”. The post’s sentence “will always have the highest possible priority” is true of round 1, and the post itself later says there is “no more leader fast path” after it.
Raft’s mitigations. Real Raft implementations add protections such as pre-vote, which etcd’s v3.6 flag list shows enabled by default. The post’s Raft is the textbook algorithm.

What to take from this

These are my conclusions from the checks above, not Cloudflare’s.

The hedge delay replaces the election timeout as the knob, and the failure it guards against moves from an outage to a bill. The paper’s liveness holds at any hedge, and in my model the price of a small one is latency (three round trips instead of one) and messages (twelvefold at five replicas). A team adopting this should measure the fast-path share and messages per decision, not only throughput.
Set the hedge from the one-way delay, not the round trip. In the model the fast path survives only when the leader’s message reaches the recorders before they start their own proposals.
Recovery gains are real and bounded. 1.2 to 4.9 times faster than Raft in my runs, depending almost entirely on how conservatively the Raft side was tuned.
Read incident write-ups from the primary document. The 2020 post-mortem is clearer about what lasted than the one-line summary of it.
Ask for the experiment. The chart is the only evidence in the post that Meerkat behaves as claimed, and it leaves out the details needed to judge it. The July post says Cloudflare is preparing a manuscript for peer review; the setup belongs there.

Run it yourself

You need Python 3.13 (I used 3.13.14) and Pillow for the chart script. Put the files from this page in one folder. For the Raft scripts, copy raft.py and cluster.py from the code blocks of the Raft tutorial into the same folder. Every script is seeded, so the tables repeat exactly; the whole set takes about a minute.

folder:
  quepaxa.py
  step01_one_slot.py  step02_safety.py  step03_rounds.py  step04_hedge.py  step05_leader_down.py
  step06_raft_timeouts.py  step07_chart_readout.py  step08_raft_outage.py  step09_raft_one_link.py
  raft.py  cluster.py        (copied from the Raft tutorial)
  quepaxa_chart.png          (downloaded from the Cloudflare post, command above)

python -m pip install pillow
python step01_one_slot.py
python step02_safety.py
python step03_rounds.py
python step04_hedge.py
python step05_leader_down.py
python step06_raft_timeouts.py
python step07_chart_readout.py quepaxa_chart.png
python step08_raft_outage.py
python step09_raft_one_link.py

The failure-detector tutorial, “How to Detect Failed Nodes in Python”, measures the same trade-off from the timeout side: what a fixed threshold costs when delays are heavy-tailed.

Sources

Cloudflare, “Choosing QuePaxa, an unusual consensus algorithm” (November 11, 2026).
Cloudflare, “Introducing Meerkat: an experiment in global consensus” (July 8, 2026).
Tennage, Băsescu et al., “QuePaxa: Escaping the Tyranny of Timeouts in Consensus”, SOSP 2023, doi 10.1145/3600006.3613150.
Lianza and Snook, “A Byzantine failure in the real world”, Cloudflare, November 27, 2020.
etcd, Tuning (v3.6) and Configuration flags (v3.6).
dedis/quepaxa, the paper’s Go prototype (README and file list read).
Our Raft tutorial for the simulator used on the Raft side.

Tags:

Search the Site

Recent Posts

Categories

Articles 296 Posts

News 301 Posts

Learning Hub 268 Posts

Cloudflare’s QuePaxa Post Turns Leader Elections Into a Hedging Delay, but in a Simulation of the Paper’s Protocol a Five-Millisecond Hedge Costs Every Slot Its Fast Path, and the 2020 Outage It Cites Lasted Six Hours Because Database Replicas Had to Be Rebuilt

Table Of Content

The ten claims, checked

What I built to check them

What I did not test

QuePaxa in 138 lines

Check one: four single slots

Check two: 20,000 random slots

The 50 percent per round, and “exponentially easier”

What the paper guarantees

What the simulator shows

The five-millisecond hedge

What the paper tested

What the simulator shows

“Recover as fast as we want”

Raft under short timeouts, outages and one bad link

The freeze is real

The storm was not reproduced

A two-second outage

One cut link

The 2020 outage, as its own post-mortem tells it

The Meerkat post’s 10x

Reading Cloudflare’s decision-rate chart

What the post leaves out

What to take from this

Run it yourself

Sources

Tags:

Share

Microsoft’s November Patch Tuesday Fixes 1,094 CVEs by Its Own Count, Not a “Record” 1,235, and the Only Exploited Bug Is an Outlook Flaw That Can Leak an NTLMv2 Hash Without the Usual Warning

How to Recover Lost Commits and Files in Git With reflog and fsck, and Snapshot Uncommitted Work Before a Script or AI Agent Can Wipe It

No Comment! Be the first one.

Leave a Reply Cancel reply

Latest

Trending

Related Posts

Categories