How to Build Leader Election in Python to Stop Split-Brain Writes

If you run more than one copy of a service (which you should, for reliability), you eventually hit a problem: some jobs must only be done by exactly one instance at a time. A cron-style scheduler that fires a nightly billing run. A background worker that polls a queue and must not process the same message twice. A cache warmer that would waste money if five replicas all ran it simultaneously. You don’t want five copies of that job racing each other, and you don’t want zero copies running because you accidentally built something that lets every replica back off forever.

The standard fix is called leader election: a group of interchangeable processes (the “candidates”) agree, without a human in the loop, that exactly one of them is “the leader” and is allowed to do the exclusive work. If the leader crashes, another candidate takes over automatically. This is the same mechanism Kubernetes itself uses internally so that only one copy of kube-scheduler or kube-controller-manager is active in a highly available control plane, even though several replicas are running.

In this tutorial you’ll build leader election completely from scratch in Python, using nothing but the standard library’s sqlite3 module as a stand-in for a real coordination service like etcd or a Kubernetes Lease object. By the end you will have personally reproduced, and fixed, the exact bug that makes naive distributed locking dangerous: a paused leader waking up and corrupting data after another instance has already taken over. That bug, and its fix, is the actual point of this tutorial. The lock itself is the easy part.

What you’ll build and the terms you need first

Before writing any code, it’s worth being precise about three terms, because sloppy definitions here are exactly how the bug creeps in.

• Lease: a time-limited claim on leadership. A candidate acquires a lease for a fixed duration (its “TTL”, time to live) and must renew it before that duration elapses, or it expires and someone else can take over. A lease is different from a plain lock because it self-expires: if the leader crashes and never releases anything, the system still recovers on its own once the TTL runs out.
• Fencing token: a number that increases every single time the lease changes hands. Whoever currently holds the lease also holds the current highest token. As you’ll see, the lease alone is not enough to guarantee correctness. The fencing token is what actually protects your data.
• Split brain: the failure mode where two processes each believe, at the same time, that they are the one true leader, and both act on that belief. This is what fencing tokens exist to prevent.

You’ll build this in five layers: a shared lease store that any number of processes can safely race against, a candidate loop that tries to become leader and renews on a schedule, a protected resource that only the leader should be allowed to write to, a reproduction of the split-brain bug with that resource unprotected, and finally the fencing-token fix. Then you’ll connect what you built to how Kubernetes’s own Lease API and client-go’s leaderelection package do the same thing in production.

Prerequisites

• Python 3.10 or newer. This tutorial was built and tested on Python 3.13. Everything here uses only the standard library (sqlite3, time, threading, subprocess) plus pytest for the verification suite at the end.
• Basic comfort reading SQL (you’ll see SELECT, INSERT, and UPDATE statements, nothing exotic).
• A terminal where you can run Python scripts. Everything runs on a single machine; nothing here needs Docker, a database server, or a cloud account.
• Some familiarity with what a race condition is helps, but isn’t required. You’ll see one happen in real time.

Create a working folder and install pytest, which is the only third-party dependency you’ll need:

mkdir leader-election-tutorial
cd leader-election-tutorial
python -m venv venv
venv\Scripts\activate    # on Windows
# source venv/bin/activate   # on macOS/Linux
pip install pytest

Step 1: Build a shared lease store

In a real multi-machine deployment, the lease store would live on a separate coordination service (etcd, ZooKeeper, or Kubernetes’s own API server backed by etcd) that every candidate talks to over the network. To keep this tutorial runnable on a single laptop with zero infrastructure, you’ll use one shared SQLite file instead. SQLite is a good stand-in here for a specific reason: it enforces that only one write transaction can be active at a time, even across separate operating system processes, which is exactly the primitive leader election needs.

Create lease_store.py:

import sqlite3
import time


class LeaseStore:
    def __init__(self, db_path):
        self.db_path = db_path
        with self._connect() as conn:
            conn.execute(
                """
                CREATE TABLE IF NOT EXISTS leases (
                    name TEXT PRIMARY KEY,
                    holder_identity TEXT NOT NULL,
                    fencing_token INTEGER NOT NULL,
                    renew_time REAL NOT NULL,
                    lease_duration_seconds REAL NOT NULL
                )
                """
            )

    def _connect(self):
        # isolation_level=None puts the connection in autocommit mode so we
        # control exactly when a write transaction (and its lock) starts,
        # using an explicit BEGIN IMMEDIATE below.
        return sqlite3.connect(self.db_path, timeout=10.0, isolation_level=None)

    def try_acquire_or_renew(self, name, holder_identity, lease_duration_seconds):
        """Attempt to become, or remain, the leader for `name`.

        Returns (acquired, fencing_token, reason). `acquired` is True only
        if `holder_identity` is the leader after this call returns.
        """
        conn = self._connect()
        try:
            conn.execute("BEGIN IMMEDIATE")
            row = conn.execute(
                "SELECT holder_identity, fencing_token, renew_time, "
                "lease_duration_seconds FROM leases WHERE name = ?",
                (name,),
            ).fetchone()
            now = time.time()

            if row is None:
                token = 1
                conn.execute(
                    "INSERT INTO leases (name, holder_identity, fencing_token, "
                    "renew_time, lease_duration_seconds) VALUES (?, ?, ?, ?, ?)",
                    (name, holder_identity, token, now, lease_duration_seconds),
                )
                conn.commit()
                return True, token, "acquired (lease was empty)"

            current_holder, current_token, renew_time, _duration = row
            expired = (now - renew_time) > lease_duration_seconds

            if current_holder == holder_identity:
                conn.execute(
                    "UPDATE leases SET renew_time = ?, lease_duration_seconds = ? "
                    "WHERE name = ?",
                    (now, lease_duration_seconds, name),
                )
                conn.commit()
                return True, current_token, "renewed"

            if expired:
                new_token = current_token + 1
                conn.execute(
                    "UPDATE leases SET holder_identity = ?, fencing_token = ?, "
                    "renew_time = ?, lease_duration_seconds = ? WHERE name = ?",
                    (holder_identity, new_token, now, lease_duration_seconds, name),
                )
                conn.commit()
                return (
                    True,
                    new_token,
                    f"acquired (took over from {current_holder!r}, its lease had expired)",
                )

            conn.rollback()
            return False, None, f"held by {current_holder!r}, not yet expired"
        except Exception:
            conn.rollback()
            raise
        finally:
            conn.close()

    def read(self, name):
        conn = self._connect()
        try:
            return conn.execute(
                "SELECT holder_identity, fencing_token, renew_time, "
                "lease_duration_seconds FROM leases WHERE name = ?",
                (name,),
            ).fetchone()
        finally:
            conn.close()

The detail that makes this whole tutorial work is the single line conn.execute("BEGIN IMMEDIATE"). SQLite’s own transaction documentation is explicit about what this does: “IMMEDIATE causes the database connection to start a new write immediately, without waiting for a write statement. The BEGIN IMMEDIATE might fail with SQLITE_BUSY if another write transaction is already active on another database connection.” That single sentence is the whole locking primitive this tutorial is built on: SQLite itself guarantees that “SQLite supports multiple simultaneous read transactions coming from separate database connections, possibly in separate threads or processes, but only one simultaneous write transaction.” Two processes can never both be inside a BEGIN IMMEDIATE block on the same database file at the same instant. One of them will always either wait or fail first. That’s a real, OS-enforced mutual exclusion primitive, not something we’re simulating with in-process logic.

Everything else in try_acquire_or_renew is ordinary business logic that runs safely because it’s wrapped inside that lock: check who currently holds the lease and whether their lease has expired, then either let the current holder renew, let a new candidate take over an expired lease, or reject a candidate that’s too late.

Step 2: Race two candidates for the same lease

Create demo1_race_and_renew.py:

import os
from lease_store import LeaseStore

DB = "demo1.db"
if os.path.exists(DB):
    os.remove(DB)

store = LeaseStore(DB)

print("=== Two candidates race for the same lease ===")
a1 = store.try_acquire_or_renew("scheduler-job", "candidate-a", lease_duration_seconds=5)
b1 = store.try_acquire_or_renew("scheduler-job", "candidate-b", lease_duration_seconds=5)
print("A:", a1)
print("B:", b1)

print("\n=== A renews before its lease expires ===")
a2 = store.try_acquire_or_renew("scheduler-job", "candidate-a", lease_duration_seconds=5)
print("A:", a2)

print("\n=== B tries again while A still holds a fresh lease ===")
b2 = store.try_acquire_or_renew("scheduler-job", "candidate-b", lease_duration_seconds=5)
print("B:", b2)

Run it:

python demo1_race_and_renew.py

Real output from this exact script:

=== Two candidates race for the same lease ===
A: (True, 1, 'acquired (lease was empty)')
B: (False, None, "held by 'candidate-a', not yet expired")

=== A renews before its lease expires ===
A: (True, 1, 'renewed')

=== B tries again while A still holds a fresh lease ===
B: (False, None, "held by 'candidate-a', not yet expired")

Candidate A got there first and claimed fencing token 1. Candidate B’s attempt failed cleanly: it didn’t crash, didn’t corrupt anything, it just learned it lost and should back off and try again later. That’s the whole contract a candidate needs to respect: check the result, and if acquired is false, don’t do the exclusive work.

Step 3: Real failover when a leader disappears

A lease that never expires isn’t useful: if the leader crashes, nobody would ever be allowed to take over. Create demo2_failover.py to see a real, wall-clock-timed expiry and takeover:

import os
import time
from lease_store import LeaseStore

DB = "demo2.db"
if os.path.exists(DB):
    os.remove(DB)

store = LeaseStore(DB)

print("=== A acquires a short-lived lease (1 second) ===")
a1 = store.try_acquire_or_renew("scheduler-job", "candidate-a", lease_duration_seconds=1)
print("A:", a1)

print("\n=== A crashes: it never calls try_acquire_or_renew again ===")
print("(sleeping 1.3s, real wall-clock time, so the lease actually expires)")
time.sleep(1.3)

print("\n=== B tries to acquire the now-expired lease ===")
b1 = store.try_acquire_or_renew("scheduler-job", "candidate-b", lease_duration_seconds=1)
print("B:", b1)

Real captured output:

=== A acquires a short-lived lease (1 second) ===
A: (True, 1, 'acquired (lease was empty)')

=== A crashes: it never calls try_acquire_or_renew again ===
(sleeping 1.3s, real wall-clock time, so the lease actually expires)

=== B tries to acquire the now-expired lease ===
B: (True, 2, "acquired (took over from 'candidate-a', its lease had expired)")

Notice the fencing token jumped from 1 to 2. Hold onto that fact; it’s the entire point of Step 6.

Choosing a lease duration

There’s a direct trade-off here: a short lease duration means fast failover when a leader genuinely dies, but it also means a leader that’s briefly slow (a garbage-collection pause, a slow disk write, a momentary network hiccup) can lose leadership even though it’s actually still healthy. A long lease duration is more forgiving of brief hiccups but leaves your system without a working leader for longer after a real crash. A common approach is to renew well before the lease is due to expire, so a single missed or slow tick doesn’t cost you leadership. The Candidate class in Step 7 renews at one-third of the lease duration, giving a healthy leader two extra chances to check back in before it would actually lose the lease.

Step 4: Prove this works across real, separate OS processes

Everything so far ran inside one Python process, which doesn’t prove much: SQLite’s locking claim is specifically about separate connections, possibly from separate processes. Let’s actually test that. Create race_worker.py, a script meant to be launched as its own process:

"""Run as a standalone process: `python race_worker.py <db> <candidate-name>`.
Prints one JSON line with the result of a single acquire attempt.

A small random jitter is added before the attempt. Without it, on this
machine the process launched first reliably wins every time, because
Windows process-creation overhead dwarfs the tiny amount of work each
worker does, launch order ends up mattering more than the lock
contention we're actually trying to demonstrate. The jitter makes the
outcome depend on real scheduling instead of launch order.
"""
import json
import random
import sys
import time

from lease_store import LeaseStore

if __name__ == "__main__":
    db_path, candidate_name = sys.argv[1], sys.argv[2]
    time.sleep(random.uniform(0, 0.05))
    store = LeaseStore(db_path)
    acquired, token, reason = store.try_acquire_or_renew(
        "scheduler-job", candidate_name, lease_duration_seconds=10
    )
    print(json.dumps({"candidate": candidate_name, "acquired": acquired, "token": token, "reason": reason}))

A quick, honest aside: the jitter comment above isn’t hypothetical. The first version of this script had no time.sleep(random.uniform(0, 0.05)) line, and worker-0 (the first process launched) won every single one of five consecutive test runs. Launching eight processes with subprocess.Popen doesn’t mean they all start executing Python at the same instant; on this machine, process creation overhead was large enough, relative to the tiny amount of work each worker does, that launch order alone decided the winner. Adding a few milliseconds of random jitter before each worker’s attempt fixed that and produced a genuine, varying race.

Now the driver script, demo3_multiprocess_race.py:

import json
import os
import subprocess
import sys

DB = "demo3.db"
if os.path.exists(DB):
    os.remove(DB)

NUM_CANDIDATES = 8
python_exe = sys.executable

print(f"=== Launching {NUM_CANDIDATES} real, separate OS processes, all racing for the same lease ===")
procs = [
    subprocess.Popen([python_exe, "race_worker.py", DB, f"worker-{i}"], stdout=subprocess.PIPE, text=True)
    for i in range(NUM_CANDIDATES)
]
results = [json.loads(p.communicate()[0].strip()) for p in procs]

winners = [r for r in results if r["acquired"]]
for r in results:
    print(r)

print(f"\n{len(winners)} of {NUM_CANDIDATES} processes acquired the lease.")
assert len(winners) == 1, "expected exactly one winner"
print("PASS: exactly one real OS process became leader.")

Run it a few times in a row. Real captured output from five consecutive runs on this machine:

=== run 1 ===
{'candidate': 'worker-1', 'acquired': True, 'token': 1, ...}
1 of 8 processes acquired the lease.
PASS: exactly one real OS process became leader.
=== run 2 ===
{'candidate': 'worker-2', 'acquired': True, 'token': 1, ...}
1 of 8 processes acquired the lease.
PASS: exactly one real OS process became leader.
=== run 3 ===
{'candidate': 'worker-3', 'acquired': True, 'token': 1, ...}
1 of 8 processes acquired the lease.
PASS: exactly one real OS process became leader.
=== run 4 ===
{'candidate': 'worker-4', 'acquired': True, 'token': 1, ...}
1 of 8 processes acquired the lease.
PASS: exactly one real OS process became leader.
=== run 5 ===
{'candidate': 'worker-2', 'acquired': True, 'token': 1, ...}
1 of 8 processes acquired the lease.
PASS: exactly one real OS process became leader.

Different workers win across different runs, which is what a genuine race looks like, and every single run still produced exactly one winner out of eight real, independent operating-system processes. That’s the mutual-exclusion guarantee working exactly as SQLite’s documentation promises.

Step 5: The danger, reproduced honestly, without fencing tokens

This is the step that actually matters. So far, the lease has behaved perfectly. Now let’s build the specific scenario where “the lease looks correct” and “the system is actually correct” come apart, because this is the mistake that makes hand-rolled distributed locking dangerous in production.

First, a protected resource that only the leader is supposed to write to. Create protected_resource.py:

import sqlite3


class ProtectedResource:
    def __init__(self, db_path):
        self.db_path = db_path
        conn = sqlite3.connect(db_path)
        conn.execute(
            "CREATE TABLE IF NOT EXISTS resource ("
            "id INTEGER PRIMARY KEY CHECK (id = 1), "
            "value TEXT, written_by TEXT, fencing_token INTEGER NOT NULL DEFAULT 0)"
        )
        conn.execute(
            "INSERT OR IGNORE INTO resource (id, value, written_by, fencing_token) "
            "VALUES (1, NULL, NULL, 0)"
        )
        conn.commit()
        conn.close()

    def write_unsafe(self, value, written_by):
        """Writes with NO fencing check. This is the bug we'll reproduce."""
        conn = sqlite3.connect(self.db_path)
        conn.execute(
            "UPDATE resource SET value = ?, written_by = ? WHERE id = 1",
            (value, written_by),
        )
        conn.commit()
        conn.close()

    def read(self):
        conn = sqlite3.connect(self.db_path)
        row = conn.execute(
            "SELECT value, written_by, fencing_token FROM resource WHERE id = 1"
        ).fetchone()
        conn.close()
        return row

Now, demo4_split_brain_unsafe.py. Read the comments carefully; this is a realistic sequence of events, not a contrived one:

import os
import time

from lease_store import LeaseStore
from protected_resource import ProtectedResource

DB_LEASES = "demo4_leases.db"
DB_RESOURCE = "demo4_resource.db"
for f in (DB_LEASES, DB_RESOURCE):
    if os.path.exists(f):
        os.remove(f)

store = LeaseStore(DB_LEASES)
resource = ProtectedResource(DB_RESOURCE)

print("=== Candidate A becomes leader ===")
acquired, token_a, reason = store.try_acquire_or_renew(
    "resource-writer", "candidate-a", lease_duration_seconds=1
)
print(f"A: acquired={acquired} token={token_a} reason={reason}")

print(
    "\n=== A is about to write, but pauses right here: a long GC pause, a "
    "stalled disk write, a slow network call, or just a descheduled thread. "
    "It has NOT written yet, and it has NOT renewed its lease. ==="
)
time.sleep(1.3)  # A is still 'paused'; real wall-clock time passes

print("\n=== Meanwhile, A's lease has expired. B takes over. ===")
acquired_b, token_b, reason_b = store.try_acquire_or_renew(
    "resource-writer", "candidate-b", lease_duration_seconds=1
)
print(f"B: acquired={acquired_b} token={token_b} reason={reason_b}")

print("\n=== B does its job and writes the resource ===")
resource.write_unsafe("B's fresh value", "candidate-b")
print("Resource is now:", resource.read())

print(
    "\n=== A finally wakes up. It never checked back in with the lease "
    "store, so as far as A's own state is concerned, it is STILL the "
    "leader. It finishes the write it started before it paused. ==="
)
resource.write_unsafe("A's STALE value", "candidate-a")
print("Resource is now:", resource.read())

Real output, unedited:

=== Candidate A becomes leader ===
A: acquired=True token=1 reason=acquired (lease was empty)

=== A is about to write, but pauses right here: a long GC pause, a stalled disk write, a slow network call, or just a descheduled thread. It has NOT written yet, and it has NOT renewed its lease. ===

=== Meanwhile, A's lease has expired. B takes over. ===
B: acquired=True token=2 reason=acquired (took over from 'candidate-a', its lease had expired)

=== B does its job and writes the resource ===
Resource is now: ("B's fresh value", 'candidate-b', 0)

=== A finally wakes up. It never checked back in with the lease store, so as far as A's own state is concerned, it is STILL the leader. It finishes the write it started before it paused. ===
Resource is now: ("A's STALE value", 'candidate-a', 0)

A’s stale write silently clobbered B’s correct write. Nothing in this program raised an exception, logged a warning, or noticed anything was wrong. This is split brain, and it’s exactly the failure mode Martin Kleppmann described in his well-known critique of naive distributed locking with Redis’s Redlock algorithm: a lock service can guarantee that only one client believes it holds the lock at a time, but it cannot control what a client does after it was paused for longer than the lease’s TTL. As Kleppmann puts it, once a client is delayed past the lease’s expiry, “you simply cannot make any assumptions about timing, which is why the code above is fundamentally unsafe, no matter what lock service you use.”

It’s worth being precise about when this matters. Kleppmann draws a useful distinction: a lock can be used for efficiency (“taking a lock saves you from unnecessarily doing the same work twice… If the lock fails and two nodes end up doing the same piece of work, the result is a minor increase in cost”) or for correctness (“taking a lock prevents concurrent processes from stepping on each others’ toes and messing up the state of your system… the result is a corrupted file, data loss, permanent inconsistency”). If your leader election is only an efficiency optimization, like avoiding running the same expensive computation twice, the bug you just reproduced is a minor annoyance. If it’s protecting correctness, like this tutorial’s resource write, that same bug is silent data corruption. Fencing tokens are how you close that gap.

Step 6: Fix it by fencing the resource itself

The fix does not live in the lease store, and it does not live in the candidate. It lives in the resource being protected, because that’s the only place that can actually stop a stale write from landing. Add a fenced write method to protected_resource.py:

    def write_fenced(self, value, written_by, fencing_token):
        """Writes only if fencing_token is >= the highest token ever seen."""
        conn = sqlite3.connect(self.db_path, isolation_level=None)
        try:
            conn.execute("BEGIN IMMEDIATE")
            highest_seen = conn.execute(
                "SELECT fencing_token FROM resource WHERE id = 1"
            ).fetchone()[0]
            if fencing_token < highest_seen:
                conn.rollback()
                return (
                    False,
                    f"rejected: fencing token {fencing_token} is behind the "
                    f"highest token this resource has already seen ({highest_seen})",
                )
            conn.execute(
                "UPDATE resource SET value = ?, written_by = ?, fencing_token = ? "
                "WHERE id = 1",
                (value, written_by, fencing_token),
            )
            conn.commit()
            return True, f"accepted with fencing token {fencing_token}"
        except Exception:
            conn.rollback()
            raise
        finally:
            conn.close()

This mirrors exactly what Kleppmann's article recommends: "you need to include a fencing token with every write request to the storage service... a fencing token is simply a number that increases (e.g. incremented by the lock service) every time a client acquires the lock." The resource remembers the highest token it has ever accepted and rejects anything lower, which is the same principle described in the article's own example: "the storage server remembers that it has already processed a write with a higher token number... and so it rejects the request with token 33."

Now re-run the exact same scenario as Step 5, but write through write_fenced instead of write_unsafe. Here is the relevant section of demo5_split_brain_fixed.py:

print("\n=== B writes, presenting its fencing token to the resource ===")
ok, msg = resource.write_fenced("B's fresh value", "candidate-b", fencing_token=token_b)
print(f"B's write: ok={ok}, {msg}")

print(
    "\n=== A wakes up and finishes its write, presenting its OLD fencing "
    "token. The resource itself checks the token, not just the lease "
    "store, so it doesn't matter that A never checked back in. ==="
)
ok, msg = resource.write_fenced("A's STALE value", "candidate-a", fencing_token=token_a)
print(f"A's write: ok={ok}, {msg}")
print("Resource is now:", resource.read())

Real output:

=== B writes, presenting its fencing token to the resource ===
B's write: ok=True, accepted with fencing token 2
Resource is now: ("B's fresh value", 'candidate-b', 2)

=== A wakes up and finishes its write, presenting its OLD fencing token. The resource itself checks the token, not just the lease store, so it doesn't matter that A never checked back in. ===
A's write: ok=False, rejected: fencing token 1 is behind the highest token this resource has already seen (2)
Resource is now: ("B's fresh value", 'candidate-b', 2)

Same exact sequence of events as Step 5. Same paused, confused candidate A trying to write stale data. The only thing that changed is that the resource now checks a number before accepting a write, and that single check is what turns silent corruption into a clean, loud rejection. B's write survives untouched.

Step 7: Put it together in a full candidate loop

The demos above called try_acquire_or_renew by hand, one call at a time, so each step would be easy to follow. A real service instead runs a loop: try to become leader, and if you are the leader, do your work and renew on a schedule; if you aren't, wait and check again. Create candidate.py:

import time

from lease_store import LeaseStore


class Candidate:
    def __init__(self, db_path, lease_name, identity, lease_duration_seconds=6):
        self.store = LeaseStore(db_path)
        self.lease_name = lease_name
        self.identity = identity
        self.lease_duration_seconds = lease_duration_seconds
        # Renew at 1/3 of the lease duration: a candidate gets at least two
        # extra chances to renew before the lease actually expires.
        self.renew_interval = lease_duration_seconds / 3
        self.is_leader = False
        self.fencing_token = None

    def tick(self):
        acquired, token, reason = self.store.try_acquire_or_renew(
            self.lease_name, self.identity, self.lease_duration_seconds
        )
        was_leader = self.is_leader
        self.is_leader = acquired
        self.fencing_token = token
        return {
            "identity": self.identity,
            "is_leader": self.is_leader,
            "fencing_token": self.fencing_token,
            "reason": reason,
            "became_leader": acquired and not was_leader,
            "lost_leadership": was_leader and not acquired,
        }

To see it run end to end, spin up two candidates as real background threads, let one become leader and renew a few times, then kill it and watch the other take over. Here's a real, complete, unedited timeline captured from running exactly that (with a 3-second lease duration so the whole thing finishes in under 10 seconds):

t= 0.01s  starting candidate-a and candidate-b
t= 0.01s  candidate-a: BECAME LEADER (fencing token 1)
t= 0.03s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 1.04s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 1.04s  candidate-a: still leader, renewed (token 1)
t= 2.06s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 2.06s  candidate-a: still leader, renewed (token 1)
t= 3.08s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 3.09s  candidate-a: still leader, renewed (token 1)
t= 4.01s  candidate-a crashes now (its thread is stopped without releasing the lease)
t= 4.09s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 5.10s  candidate-b: standby (held by 'candidate-a', not yet expired)
t= 6.13s  candidate-b: BECAME LEADER (fencing token 2)
t= 7.16s  candidate-b: still leader, renewed (token 2)
t= 8.19s  candidate-b: still leader, renewed (token 2)

Look closely at the timing: candidate-a's last renewal lands at t=3.09s with a 3-second lease duration, so its lease was due to expire at roughly t=6.09s. Candidate-b actually notices and takes over at t=6.13s, four hundredths of a second later. That's not a coincidence; it's the TTL doing exactly what it's supposed to do, measured on the wall clock, not asserted in a comment.

How production systems do this

Everything you built maps directly onto how Kubernetes itself does leader election, which is worth knowing both because you'll likely deploy services onto Kubernetes and because it validates that the pattern you just built by hand is the real pattern, not a toy simplification of it.

Kubernetes has a first-class API object for exactly this called a Lease, in the coordination.k8s.io/v1 API group. Its official field reference describes fields that map almost one-to-one onto the SQLite columns you just built:

• holderIdentity: "Contains the identity of the holder of a current lease."
• leaseDurationSeconds: "A duration that candidates for a lease need to wait to force acquire it. This is measured against the time of last observed renewTime." This is your TTL.
• acquireTime and renewTime: when the lease was first acquired, and when the current holder last checked in. This is your renew_time column.
• leaseTransitions: "The number of transitions of a lease between holders." This is conceptually the same counter as your fencing token, tracking how many times leadership has changed hands.

Kubernetes documents the underlying acquisition mechanism the same way you just implemented it: "When the Lease does not exist or has expired (current time > renewTime + leaseDurationSeconds), candidate instances attempt to update the Lease with their identity. Kubernetes relies on optimistic concurrency control via the object's resourceVersion: only one update succeeds due to version mismatch on concurrent attempts. The instance whose update is accepted becomes the leader." That "version mismatch on concurrent attempts" is etcd's equivalent of your BEGIN IMMEDIATE transaction: an atomic compare-and-swap that only one concurrent writer can win, enforced by the API server rather than by SQLite's file locking.

kube-scheduler and kube-controller-manager both run this way in a highly available control plane: several replicas run at once, but only the one holding the current Lease is actually active, exactly like your candidate-a and candidate-b threads. In application code, this same pattern is implemented by client-go's leaderelection package, which any Go service can import directly rather than reimplementing this logic. Kubernetes 1.37 also adds a beta refinement called Coordinated Leader Election, which layers a separate LeaseCandidate object and a selection strategy (currently OldestEmulationVersion, preferring the candidate with the lowest emulation version) on top of this same Lease mechanism, specifically to make leader selection deterministic during version-skewed cluster upgrades; the base acquire-and-renew mechanic underneath is unchanged. Kubernetes has also been improving the failover side of this: a newer, still-alpha feature called ControllerManagerReleaseLeaderElectionLockOnExit lets kube-controller-manager release its lease immediately on a graceful shutdown instead of making the next candidate wait out the full TTL, which is precisely the failover-latency trade-off you saw in Step 3.

The fencing-token half of this story predates Kubernetes and comes from a wider body of distributed-systems work. If you use ZooKeeper as a lock service instead, Kleppmann notes you can reuse its own zxid or znode version number directly as a ready-made fencing token, for exactly the same reason: it's a value that's guaranteed to only ever increase.

Common mistakes and gotchas

Using the wrong clock for a value that crosses process boundaries

It's tempting to reach for time.monotonic() instead of time.time() for the renew_time column, since monotonic clocks can't jump backward the way wall clocks can (say, from an NTP correction). Don't. Python's own documentation for time.monotonic() is explicit: "The reference point of the returned value is undefined, so that only the difference between the results of two calls is valid." On this specific Windows machine, running two separate Python processes and printing time.monotonic() from each at nearly the same moment actually produced two nearly identical values (both around 15435.6 seconds), because Windows happens to implement it as a system-wide counter. But that behavior isn't a documented guarantee, and relying on it would mean your lease implementation silently breaks the moment it runs somewhere that doesn't share that same implementation detail. renew_time gets written by one process and read by a different one, so it needs a clock whose values are actually meaningful across that boundary. That's time.time(), not time.monotonic().

Who owns the clock in a real multi-machine deployment

This tutorial's demos all run on one machine, so every candidate shares the exact same system clock, and comparing wall-clock timestamps across processes is safe. That stops being true the moment your leaders are running on different physical machines with clocks that can drift apart. This is a large part of why Kleppmann is skeptical of algorithms like Redlock that ask several independent client machines to each independently judge whether a lease has expired using their own local clock. The safer pattern, and the one both etcd and the Kubernetes API server actually use, is to make the lock service itself the single authority on time: candidates ask the coordination service whether a lease is still valid, and the service answers using its own one clock, rather than every candidate trusting its own local idea of "now."

Tuning the renewal interval too tight

If a leader only renews right before its lease expires, one slow tick (a garbage-collection pause, a momentary disk stall) can cost it leadership even though it's perfectly healthy, which needlessly triggers a failover and a burst of split-brain risk while the handoff settles. Renewing at a fraction of the TTL, this tutorial used one-third, gives a healthy leader multiple chances to check back in before it actually loses anything.

Forgetting that the fencing check belongs on the resource, not the lock

It's a natural instinct to think the fix for Step 5's bug belongs in the lease store or the candidate loop. It doesn't. Nothing about candidate A's own code is wrong; A genuinely did hold the lease when it started its write, and there is no way for A to know, from inside its own paused process, that it has since lost leadership. The only reliable place to stop the stale write is at the resource actually being protected, which is exactly why write_fenced lives on ProtectedResource and not on Candidate.

Verify it all works end-to-end

Pull everything into a pytest suite so you (or a CI pipeline) can re-verify all of this automatically instead of re-running scripts by hand. Save this as test_lease_store.py:

import os
import time

import pytest

from lease_store import LeaseStore
from protected_resource import ProtectedResource


@pytest.fixture
def db_path(tmp_path):
    return str(tmp_path / "leases.db")


@pytest.fixture
def resource_path(tmp_path):
    return str(tmp_path / "resource.db")


def test_first_candidate_wins_empty_lease(db_path):
    store = LeaseStore(db_path)
    acquired, token, reason = store.try_acquire_or_renew("job", "a", 5)
    assert acquired is True
    assert token == 1


def test_second_candidate_loses_active_lease(db_path):
    store = LeaseStore(db_path)
    store.try_acquire_or_renew("job", "a", 5)
    acquired, token, reason = store.try_acquire_or_renew("job", "b", 5)
    assert acquired is False


def test_holder_can_renew(db_path):
    store = LeaseStore(db_path)
    store.try_acquire_or_renew("job", "a", 5)
    acquired, token, reason = store.try_acquire_or_renew("job", "a", 5)
    assert acquired is True
    assert reason == "renewed"


def test_takeover_after_expiry_bumps_fencing_token(db_path):
    store = LeaseStore(db_path)
    store.try_acquire_or_renew("job", "a", lease_duration_seconds=0.2)
    time.sleep(0.3)
    acquired, token, reason = store.try_acquire_or_renew("job", "b", lease_duration_seconds=0.2)
    assert acquired is True
    assert token == 2


def test_fencing_tokens_are_strictly_increasing_across_many_handoffs(db_path):
    store = LeaseStore(db_path)
    tokens = []
    for candidate in ["a", "b", "c", "d"]:
        acquired, token, _ = store.try_acquire_or_renew("job", candidate, lease_duration_seconds=0.15)
        assert acquired is True
        tokens.append(token)
        time.sleep(0.2)
    assert tokens == list(range(1, len(tokens) + 1))


def test_protected_resource_rejects_stale_fencing_token(resource_path):
    resource = ProtectedResource(resource_path)
    resource.write_fenced("first", "a", fencing_token=5)
    ok, msg = resource.write_fenced("stale", "a", fencing_token=3)
    assert ok is False
    value, written_by, token = resource.read()
    assert value == "first"


def test_protected_resource_accepts_equal_or_higher_token(resource_path):
    resource = ProtectedResource(resource_path)
    resource.write_fenced("v1", "a", fencing_token=5)
    ok, _ = resource.write_fenced("v2", "b", fencing_token=6)
    assert ok is True
    value, written_by, token = resource.read()
    assert (value, written_by, token) == ("v2", "b", 6)


def test_eight_real_processes_only_one_wins(tmp_path):
    import json
    import subprocess
    import sys

    db = str(tmp_path / "race.db")
    here = os.path.dirname(os.path.abspath(__file__))
    procs = [
        subprocess.Popen(
            [sys.executable, os.path.join(here, "race_worker.py"), db, f"w{i}"],
            stdout=subprocess.PIPE, text=True,
        )
        for i in range(8)
    ]
    results = [json.loads(p.communicate()[0].strip()) for p in procs]
    winners = [r for r in results if r["acquired"]]
    assert len(winners) == 1

Run the full suite:

python -m pytest test_lease_store.py -v

Real output:

collected 8 items

test_lease_store.py::test_first_candidate_wins_empty_lease PASSED        [ 12%]
test_lease_store.py::test_second_candidate_loses_active_lease PASSED     [ 25%]
test_lease_store.py::test_holder_can_renew PASSED                        [ 37%]
test_lease_store.py::test_takeover_after_expiry_bumps_fencing_token PASSED [ 50%]
test_lease_store.py::test_fencing_tokens_are_strictly_increasing_across_many_handoffs PASSED [ 62%]
test_lease_store.py::test_protected_resource_rejects_stale_fencing_token PASSED [ 75%]
test_lease_store.py::test_protected_resource_accepts_equal_or_higher_token PASSED [ 87%]
test_lease_store.py::test_eight_real_processes_only_one_wins PASSED      [100%]

============================== 8 passed in 1.51s ==============================

If you've followed along, your own run should show all tests passing too, including the eight-real-process race, which is the strongest evidence available that the mutual exclusion here isn't an illusion created by single-process testing.

Next steps

You now have a from-scratch, honestly tested implementation of leader election with fencing tokens, and a clear map from every piece of it to the real APIs (Kubernetes Lease objects, client-go's leaderelection package, etcd) that production systems use for the same job. A few natural directions to take this further:

• If you want to see the durability half of this problem, where a process needs to survive a crash without losing data rather than coordinate with other processes, the site's write-ahead log tutorial builds exactly that from scratch.
• If your leader's own "do the work" step calls a downstream dependency that might be slow or down, pair leader election with the circuit breaker and rate limiter patterns covered elsewhere on this site, so your one elected leader doesn't take the whole system down by hammering something that's already struggling.
• If you're deploying this onto an actual Kubernetes cluster instead of a single machine, look at the Lease API directly rather than reimplementing this in SQLite: Python's official Kubernetes client exposes it, and it gets you etcd's durability and multi-machine consistency for free instead of a single shared file.