How to Build a Changed Block Tracking System in Python to Speed Up Incremental Backups

Imagine you are running a database on a 500 gigabyte Kubernetes persistent volume, and a backup job needs to protect it every few hours. Copying all 500 gigabytes every single run wastes network bandwidth, ties up the storage backend, and can blow straight through a maintenance window, even though maybe 50 megabytes of that volume actually changed since the last backup. Changed block tracking is the technique that fixes this: instead of copying an entire volume, you work out exactly which fixed-size chunks of data (blocks) changed since the last backup, and copy only those.

This is not a hypothetical problem, and it is not old news either. Kubernetes’ own native Changed Block Tracking (CBT) API moved to Beta in March 2026, and a September 14, 2026 blog post from the project walks through exactly what changed in that release. The API lets backup vendors ask a storage driver “which blocks changed between these two volume snapshots” over a standard gRPC interface instead of reverse-engineering every storage vendor’s proprietary format. By the end of this tutorial you will understand exactly what that API does and why, because you will have built a working, simplified version of the same idea yourself, entirely in Python, with no cluster, no CSI driver, and no cloud account required.

What You Will Build

You will build a small changed-block-tracking library from scratch that:

• Divides a simulated storage volume into fixed-size blocks and fingerprints each one with a checksum
• Compares two fingerprint snapshots to find exactly which blocks changed
• Merges adjacent changed blocks into larger contiguous ranges, the same optimization the real API uses
• Writes an incremental backup containing only the changed bytes, and measures the real savings
• Chains two incremental backups together, and personally reproduces a dangerous, silent-corruption bug that happens when you apply them out of order
• Measures, with real numbers, why block size is a genuine tradeoff and not a free parameter

Along the way you will see exactly how this maps onto Kubernetes’ real Changed Block Tracking API: the actual gRPC service definition, the actual message field names, and the actual components that make it work in production.

Prerequisites

• Python 3.10 or newer. This tutorial was written and personally tested on Python 3.13.14, but nothing here depends on a feature newer than 3.10.
• Comfort with basic file I/O, dictionaries, and reading a small class. No prior storage, backup, or Kubernetes experience is assumed; every term is defined the first time it is used.
• Everything here uses only the Python standard library: hashlib, dataclasses, json, random, shutil. pytest is used only for the optional automated test suite in the last step (pip install pytest).
• No live Kubernetes cluster, CSI storage driver, or cloud account of any kind. This tutorial builds the underlying concept from scratch; a later section shows exactly how it maps onto the real Kubernetes API so you know what to look for once you do have a cluster.

Step 1: Build the Shared Checksumming and Diffing Library

Everything in this tutorial is built on four small, testable functions. Put them in a file named cbt_lib.py, since every later step imports from it.

"""Shared building blocks for the changed block tracking demo."""
import hashlib
import json
from dataclasses import dataclass, asdict


@dataclass(frozen=True)
class BlockMetadata:
    """Mirrors the real CSI/Kubernetes SnapshotMetadata BlockMetadata message:
    a zero-based byte offset plus a size, describing one contiguous data range.
    """
    byte_offset: int
    size_bytes: int


def checksum_blocks(path, block_size):
    """Read a file block by block and return {block_index: sha256_hex_digest}.

    This dict is our stand-in for a storage snapshot: a compact fingerprint
    of every block's contents at one point in time, without keeping the
    block contents themselves.
    """
    checksums = {}
    with open(path, "rb") as f:
        index = 0
        while True:
            block = f.read(block_size)
            if not block:
                break
            checksums[index] = hashlib.sha256(block).hexdigest()
            index += 1
    return checksums


def changed_block_indices(base_checksums, target_checksums):
    """Compare two per-block checksum snapshots and return the sorted list
    of block indices whose checksum differs (added, removed, or modified).
    """
    changed = []
    for index in sorted(set(base_checksums) | set(target_checksums)):
        if base_checksums.get(index) != target_checksums.get(index):
            changed.append(index)
    return changed


def merge_into_ranges(block_indices, block_size):
    """Merge a sorted list of block indices into contiguous BlockMetadata
    ranges: adjacent changed blocks get reported as one bigger range instead
    of many small ones.
    """
    ranges = []
    for index in block_indices:
        offset = index * block_size
        if ranges and ranges[-1].byte_offset + ranges[-1].size_bytes == offset:
            prev = ranges[-1]
            ranges[-1] = BlockMetadata(prev.byte_offset, prev.size_bytes + block_size)
        else:
            ranges.append(BlockMetadata(offset, block_size))
    return ranges

Two design choices are worth calling out before moving on. First, checksum_blocks returns a plain dictionary keyed by block index, not a list; a real volume snapshot at time A and time B might not even have the same number of blocks if the volume grew, and a dict makes that comparison trivial with set(base) | set(target). Second, the BlockMetadata dataclass’s two field names, byte_offset and size_bytes, are not arbitrary. They are the exact field names used by the real Kubernetes and CSI SnapshotMetadata API, which the last major section of this tutorial shows you verbatim.

Add two more functions to the same file for computing a whole-file checksum (used only to verify a restore later) and for actually copying changed ranges into an incremental backup file plus a small JSON manifest describing where each range came from:

def sha256_file(path):
    """Whole-file SHA-256, used only to verify a restore is byte-for-byte correct."""
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while True:
            chunk = f.read(1024 * 1024)
            if not chunk:
                break
            h.update(chunk)
    return h.hexdigest()


def write_incremental_backup(volume_path, ranges, delta_path, manifest_path):
    """Copy only the given byte ranges out of volume_path into delta_path,
    back to back, and write a JSON manifest recording which range landed at
    which offset, so a restore can put each range back where it belongs.
    """
    manifest = []
    total_bytes = 0
    with open(volume_path, "rb") as src, open(delta_path, "wb") as dst:
        for r in ranges:
            src.seek(r.byte_offset)
            data = src.read(r.size_bytes)
            dst.write(data)
            manifest.append({"byte_offset": r.byte_offset, "size_bytes": r.size_bytes})
            total_bytes += len(data)
    with open(manifest_path, "w", encoding="utf-8") as f:
        json.dump(manifest, f)
    return total_bytes


def apply_incremental(restore_path, delta_path, manifest_path):
    """Overlay one incremental backup's changed ranges onto restore_path,
    which must already exist (either the base backup or the result of
    applying earlier incrementals in the chain).
    """
    with open(manifest_path, "r", encoding="utf-8") as f:
        manifest = json.load(f)
    with open(delta_path, "rb") as delta, open(restore_path, "r+b") as dst:
        for entry in manifest:
            data = delta.read(entry["size_bytes"])
            dst.seek(entry["byte_offset"])
            dst.write(data)

Step 2: Create a Simulated Volume and Take a Baseline Snapshot

Real storage volumes (a cloud disk, a SAN LUN, a Kubernetes PersistentVolume backed by a CSI driver) are enormous arrays of bytes divided into fixed-size blocks, typically 512 bytes to a few kilobytes each. Simulate one with a plain file on disk, using a 4096-byte block size, which matches a common filesystem block size:

"""Step 1: create a simulated block volume and take our first ('base') snapshot."""
import random

from cbt_lib import checksum_blocks

VOLUME_PATH = "volume.bin"
BLOCK_SIZE = 4096
VOLUME_SIZE = 10 * 1024 * 1024  # 10 MiB


def create_volume(path, size, seed):
    rng = random.Random(seed)
    with open(path, "wb") as f:
        remaining = size
        while remaining > 0:
            chunk = min(1024 * 1024, remaining)
            f.write(rng.randbytes(chunk))
            remaining -= chunk


if __name__ == "__main__":
    create_volume(VOLUME_PATH, VOLUME_SIZE, seed=42)
    base_snapshot = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)

    print(f"Volume size:      {VOLUME_SIZE:,} bytes")
    print(f"Block size:       {BLOCK_SIZE:,} bytes")
    print(f"Number of blocks: {len(base_snapshot):,}")
    print("First 3 block checksums (first 12 hex chars):")
    for i in range(3):
        print(f"  block {i}: {base_snapshot[i][:12]}")

Save this as step1_setup.py and run it. It uses a seeded random generator, so anyone running it gets identical output:

Volume size:      10,485,760 bytes
Block size:       4,096 bytes
Number of blocks: 2,560
First 3 block checksums (first 12 hex chars):
  block 0: 27ea8434ddf3
  block 1: 7596317f7671
  block 2: d48c92a570a8

base_snapshot is your first changed-block-tracking snapshot. It is not a copy of the volume, it is a dictionary of 2,560 short hex strings, one per block. That is the whole trick behind cheap change detection: keep a small fingerprint of what a block looked like, not the block itself, and compare fingerprints later instead of comparing full contents.

Step 3: The Naive Approach, a Full Backup Every Time

Before building anything clever, establish the baseline every backup system starts from: copy the entire volume.

"""Step 2: the naive approach every backup starts with: copy the entire volume."""
import os
import shutil

from step1_setup import VOLUME_PATH

FULL_BACKUP_PATH = "backup_full_0.bin"

if __name__ == "__main__":
    shutil.copyfile(VOLUME_PATH, FULL_BACKUP_PATH)
    size = os.path.getsize(FULL_BACKUP_PATH)
    print(f"Full backup 0 written: {size:,} bytes copied out of the volume")

Save as step2_full_backup.py and run it:

Full backup 0 written: 10,485,760 bytes copied out of the volume

On a local 10 MiB test file, copying takes a fraction of a second no matter which approach you use, so timing this locally would not teach you much. What changed block tracking actually saves in production is network egress and remote storage cost for a volume that might be hundreds of gigabytes and gets backed up every few hours. The metric that matters is bytes moved, not wall-clock time on a laptop, so from here on this tutorial measures bytes.

Step 4: Simulate Writes and Detect Which Blocks Changed

Now simulate an application actually using the volume: write to a scattered handful of blocks, the way a database or filesystem naturally would, then take a second snapshot and diff it against the first.

"""Step 3: simulate a batch of writes to the volume, take a second snapshot,
and compute which blocks actually changed.
"""
import random

from cbt_lib import checksum_blocks, changed_block_indices, merge_into_ranges
from step1_setup import VOLUME_PATH, BLOCK_SIZE


def pick_change_pattern(rng, total_blocks, n_clusters, cluster_sizes):
    """Pick a realistic mix of isolated single-block writes and small
    multi-block runs (a program rewriting one field touches one block;
    rewriting a whole record touches several adjacent ones).
    """
    indices = set()
    for _ in range(n_clusters):
        start = rng.randrange(0, total_blocks - 10)
        size = rng.choice(cluster_sizes)
        for i in range(size):
            if start + i < total_blocks:
                indices.add(start + i)
    return indices


def apply_writes(path, block_indices, block_size, seed):
    rng = random.Random(seed)
    with open(path, "r+b") as f:
        for index in sorted(block_indices):
            f.seek(index * block_size)
            f.write(rng.randbytes(block_size))


if __name__ == "__main__":
    base_snapshot = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)
    total_blocks = len(base_snapshot)

    pattern_rng = random.Random(1001)
    round1_indices = pick_change_pattern(
        pattern_rng, total_blocks, n_clusters=15, cluster_sizes=(1, 1, 1, 2, 3, 5)
    )
    apply_writes(VOLUME_PATH, round1_indices, BLOCK_SIZE, seed=2001)

    snapshot1 = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)
    changed = changed_block_indices(base_snapshot, snapshot1)
    ranges = merge_into_ranges(changed, BLOCK_SIZE)

    print(f"Blocks actually written in round 1: {len(round1_indices)}")
    print(f"Blocks flagged as changed by the diff: {len(changed)}")
    print(f"Merged into {len(ranges)} contiguous ranges:")
    for r in ranges:
        print(f"  offset={r.byte_offset:>9,}  size={r.size_bytes:>6,}")

    with open("round1_indices.txt", "w") as f:
        f.write(",".join(str(i) for i in sorted(round1_indices)))

Save as step3_first_change_round.py and run it:

Blocks actually written in round 1: 42
Blocks flagged as changed by the diff: 42
Merged into 14 contiguous ranges:
  offset=  315,392  size=12,288
  offset=  847,872  size= 4,096
  offset=  983,040  size= 4,096
  offset=1,527,808  size=12,288
  offset=1,691,648  size=20,480
  offset=3,145,728  size=12,288
  offset=3,338,240  size=12,288
  offset=6,279,168  size=12,288
  offset=6,328,320  size= 4,096
  offset=6,725,632  size=12,288
  offset=6,819,840  size=20,480
  offset=8,269,824  size=20,480
  offset=10,256,384  size= 4,096
  offset=10,280,960  size=20,480

Two things to notice. First, the diff caught exactly 42 of 42 changed blocks, with zero false positives and zero misses, because SHA-256 checksums make an undetected collision between two different 4,096-byte blocks astronomically unlikely. Second, 42 individually changed blocks collapsed into only 14 ranges once adjacent blocks were merged, because some of the writes landed next to each other (a 5-block cluster becomes one 20,480-byte range instead of five separate 4,096-byte entries). That merge step matters a lot once you get to Step 5’s numbers: fewer, larger ranges mean less bookkeeping overhead for whatever consumes this metadata next.

Step 5: Turn the Diff Into an Incremental Backup and Measure the Savings

Now use those ranges for what they exist for: copy only the changed bytes into a small incremental backup, instead of copying the whole volume again.

"""Step 4: write an incremental backup containing only the changed ranges,
and measure how many bytes it actually saves compared to a second full backup.
"""
import os

from cbt_lib import checksum_blocks, changed_block_indices, merge_into_ranges, write_incremental_backup
from step1_setup import VOLUME_PATH, BLOCK_SIZE, VOLUME_SIZE
from step2_full_backup import FULL_BACKUP_PATH

if __name__ == "__main__":
    base_snapshot = checksum_blocks(FULL_BACKUP_PATH, BLOCK_SIZE)
    current_snapshot = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)

    changed = changed_block_indices(base_snapshot, current_snapshot)
    ranges = merge_into_ranges(changed, BLOCK_SIZE)

    bytes_written = write_incremental_backup(
        VOLUME_PATH, ranges, "delta_1.bin", "delta_1_manifest.json"
    )

    full_backup_bytes = VOLUME_SIZE
    savings_pct = (1 - bytes_written / full_backup_bytes) * 100

    print(f"A second full backup would copy: {full_backup_bytes:,} bytes")
    print(f"Incremental backup 1 copies:     {bytes_written:,} bytes")
    print(f"Bytes saved:                     {savings_pct:.1f}%")
    print(f"delta_1.bin on disk:              {os.path.getsize('delta_1.bin'):,} bytes")

Save as step4_incremental_backup.py and run it:

A second full backup would copy: 10,485,760 bytes
Incremental backup 1 copies:     172,032 bytes
Bytes saved:                     98.4%
delta_1.bin on disk:              172,032 bytes

That is a 98.4 percent reduction in bytes moved and stored for this particular change pattern, purely because the diff let the backup skip the 2,518 blocks (2,560 total minus the 42 that changed) that never changed. In a database that only touches a small fraction of its pages between backups, and most real databases do, this kind of ratio is exactly why backup vendors care so much about getting this right.

Step 6: Chain Two Incrementals, and a Gotcha That Corrupts Restores Silently

One incremental backup is not the whole story. In practice you take a base backup, then an incremental, then another incremental, and so on, and eventually you have to restore from a chain of them. This step simulates a second round of writes, takes a second incremental backup relative to the first, and then does two restores: one correct, and one that skips a step to show exactly what breaks.

"""Step 5: a second round of writes, a second incremental backup, and the
backup-chain gotcha: incrementals must be replayed in order.
"""
import random
import shutil

from cbt_lib import (
    checksum_blocks,
    changed_block_indices,
    merge_into_ranges,
    write_incremental_backup,
    apply_incremental,
    sha256_file,
)
from step1_setup import VOLUME_PATH, BLOCK_SIZE
from step2_full_backup import FULL_BACKUP_PATH
from step3_first_change_round import pick_change_pattern, apply_writes

if __name__ == "__main__":
    with open("round1_indices.txt") as f:
        round1_indices = {int(x) for x in f.read().split(",")}

    # This is the state of the volume right after round 1: our "snapshot 1".
    snapshot1 = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)
    total_blocks = len(snapshot1)

    pattern_rng = random.Random(1002)
    round2_pattern = pick_change_pattern(
        pattern_rng, total_blocks, n_clusters=10, cluster_sizes=(1, 1, 2, 3)
    )
    # Force a few of round 1's blocks to be touched again, so the restore
    # chain has both overlapping and non-overlapping changes to deal with.
    forced_overlap = set(sorted(round1_indices)[:3])
    round2_indices = round2_pattern | forced_overlap
    apply_writes(VOLUME_PATH, round2_indices, BLOCK_SIZE, seed=3001)

    snapshot2 = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)
    changed2 = changed_block_indices(snapshot1, snapshot2)
    ranges2 = merge_into_ranges(changed2, BLOCK_SIZE)
    bytes2 = write_incremental_backup(VOLUME_PATH, ranges2, "delta_2.bin", "delta_2_manifest.json")

    round1_only = round1_indices - round2_indices
    print(f"Round 2 wrote {len(round2_indices)} blocks ({len(forced_overlap)} re-touch round 1's blocks)")
    print(f"Incremental backup 2 copies {bytes2:,} bytes across {len(ranges2)} ranges")
    print(f"Blocks round 1 changed that round 2 never touched again: {len(round1_only)}")
    print()

    # --- Correct restore: replay the chain in order: base, then incr 1, then incr 2 ---
    shutil.copyfile(FULL_BACKUP_PATH, "restore_correct.bin")
    apply_incremental("restore_correct.bin", "delta_1.bin", "delta_1_manifest.json")
    apply_incremental("restore_correct.bin", "delta_2.bin", "delta_2_manifest.json")
    correct_matches = sha256_file("restore_correct.bin") == sha256_file(VOLUME_PATH)
    print(f"Correct restore (base -> incr1 -> incr2) matches the live volume: {correct_matches}")

    # --- Wrong restore: skip incremental 1 and go straight from base to incr 2 ---
    shutil.copyfile(FULL_BACKUP_PATH, "restore_wrong.bin")
    apply_incremental("restore_wrong.bin", "delta_2.bin", "delta_2_manifest.json")
    wrong_matches = sha256_file("restore_wrong.bin") == sha256_file(VOLUME_PATH)
    wrong_checksums = checksum_blocks("restore_wrong.bin", BLOCK_SIZE)
    live_checksums = checksum_blocks(VOLUME_PATH, BLOCK_SIZE)
    corrupted_blocks = changed_block_indices(wrong_checksums, live_checksums)
    print(f"Wrong restore  (base -> incr2, incr1 skipped) matches the live volume: {wrong_matches}")
    print(f"Blocks silently wrong in that restore: {len(corrupted_blocks)}")
    print(f"  (expected to equal round-1-only block count: {len(round1_only)})")

Save as step5_second_round_and_chain.py and run it:

Round 2 wrote 18 blocks (3 re-touch round 1's blocks)
Incremental backup 2 copies 73,728 bytes across 10 ranges
Blocks round 1 changed that round 2 never touched again: 39

Correct restore (base -> incr1 -> incr2) matches the live volume: True
Wrong restore  (base -> incr2, incr1 skipped) matches the live volume: False
Blocks silently wrong in that restore: 39
  (expected to equal round-1-only block count: 39)

This is the single most important gotcha in this whole tutorial, and it deserves attention: applying delta_2 straight onto the base backup does not raise an exception, does not print a warning, and produces a file of the exact right size. It just silently reconstructs the wrong data in 39 blocks, specifically the blocks that round 1 changed and round 2 never touched again. The manifest for delta_2 only ever recorded the ranges that changed between snapshot 1 and snapshot 2; it has no way to know, and no reason to know, about changes that happened before snapshot 1 existed. An incremental backup is only ever correct relative to the specific snapshot it was taken against, which means a restore chain has to be replayed in the exact order the backups were taken, with nothing skipped. Any backup tool that lets you cherry-pick which incrementals to apply is a tool that will eventually let someone build the wrong restore chain by accident.

Step 7: Why Block Size Is a Tradeoff, Not a Free Parameter

Every example so far used a fixed 4,096-byte block size without asking whether that number was a good choice. It is worth measuring, with real numbers, why it matters. Apply the exact same tiny, scattered edits to a copy of the volume, then re-run the diff at three different block sizes.

"""Step 6: block size is a tuning knob, not a free parameter. Apply the exact
same sparse, byte-level writes and re-run the diff at three different block
sizes to see the tradeoff directly.
"""
import random
import shutil

from cbt_lib import checksum_blocks, changed_block_indices, merge_into_ranges
from step1_setup import VOLUME_PATH, VOLUME_SIZE

BASE_PATH = "tradeoff_base.bin"
EDITED_PATH = "tradeoff_edited.bin"
N_EDITS = 20
EDIT_SIZE = 4  # bytes touched per edit -- smaller than any block size we test


def apply_sparse_edits(path, n_edits, edit_size, volume_size, seed):
    rng = random.Random(seed)
    with open(path, "r+b") as f:
        for _ in range(n_edits):
            pos = rng.randrange(0, volume_size - edit_size)
            f.seek(pos)
            f.write(rng.randbytes(edit_size))


if __name__ == "__main__":
    shutil.copyfile(VOLUME_PATH, BASE_PATH)
    shutil.copyfile(VOLUME_PATH, EDITED_PATH)
    apply_sparse_edits(EDITED_PATH, N_EDITS, EDIT_SIZE, VOLUME_SIZE, seed=77)

    print(f"Applied {N_EDITS} scattered edits of {EDIT_SIZE} bytes each "
          f"({N_EDITS * EDIT_SIZE} real bytes changed) to a copy of the volume.")
    print()
    header = f"{'block size':>12} {'changed blocks':>15} {'ranges':>8} {'bytes to copy':>15} {'overhead vs real bytes':>24}"
    print(header)

    for block_size in (512, 4096, 65536):
        base_sums = checksum_blocks(BASE_PATH, block_size)
        edited_sums = checksum_blocks(EDITED_PATH, block_size)
        changed = changed_block_indices(base_sums, edited_sums)
        ranges = merge_into_ranges(changed, block_size)
        total_bytes = sum(r.size_bytes for r in ranges)
        overhead = total_bytes / (N_EDITS * EDIT_SIZE)
        print(f"{block_size:>12,} {len(changed):>15,} {len(ranges):>8,} {total_bytes:>15,} {overhead:>23.1f}x")

Save as step6_blocksize_tradeoff.py and run it:

Applied 20 scattered edits of 4 bytes each (80 real bytes changed) to a copy of the volume.

  block size  changed blocks   ranges   bytes to copy   overhead vs real bytes
         512              20       20          10,240                   128.0x
       4,096              20       20          81,920                  1024.0x
      65,536              20       14       1,310,720                 16384.0x

Only 80 real bytes changed across the whole 10 MiB volume, but the amount of data that has to move in the incremental backup depends entirely on the block size: 10,240 bytes at 512-byte blocks, 81,920 bytes at 4,096-byte blocks, and a striking 1,310,720 bytes, more than 16,000 times the real change, at 65,536-byte blocks. A single 4-byte write anywhere inside a block forces the entire block to be treated as changed, since that is the smallest unit the system can report on. The number of ranges tells the other half of the story: at 65,536 bytes, some of the 20 scattered edits happened to land inside the same 64 KB block, so 20 changed blocks collapsed into only 14 ranges, meaning less metadata to track but far more wasted bytes per range. Smaller blocks minimize wasted bytes in the incremental backup at the cost of more metadata entries to track; larger blocks minimize metadata at the cost of copying far more unchanged data alongside every real change. There is no universally correct block size, only a tradeoff tuned to how sparse or clustered your real write pattern is.

How This Maps to Kubernetes’ Real Changed Block Tracking API

Everything above is a from-scratch, checksum-based approximation built so it can run with nothing but the Python standard library. The real Kubernetes feature solves the exact same problem with a formally specified gRPC service. Understanding the mapping is what lets you read the real API’s documentation and immediately know what each piece is for.

Why Kubernetes Needed This At All

Kubernetes’ own announcement of the alpha version of this feature lays out the motivation plainly. Traditional full-volume backup approaches face three problems: “long backup windows” where “full volume backups can take hours for large datasets, making it difficult to complete within maintenance windows”; “high resource utilization” since “backup operations consume substantial network bandwidth and I/O resources”; and “increased storage costs” because “repetitive full backups store redundant data, causing storage requirements to grow linearly even when only a small percentage of data actually changes between backups.” Those are precisely the three costs Step 5’s 98.4 percent savings number was demonstrating on a toy scale.

The Three Real Components

That same announcement describes the implementation as three primary components. The CSI SnapshotMetadata Service API is “an API, offered by gRPC, that provides volume snapshot and changed block data,” and is implemented by the storage vendor’s own CSI driver. The SnapshotMetadataService API is “a Kubernetes CustomResourceDefinition (CRD) that advertises CSI driver metadata service availability and connection details to cluster clients,” so a backup application can discover which drivers on the cluster support this at all. The External Snapshot Metadata Sidecar is “an intermediary component that connects CSI drivers to backup applications via a standardized gRPC interface,” the piece that actually sits between a backup tool and the driver. The Kubernetes CSI developer documentation describes what that sidecar actually does day to day: it authenticates and authorizes each backup application’s request, then “acts as a proxy as it fetches the desired metadata from the CSI driver and streams it directly to the requesting application,” which keeps that traffic off the Kubernetes API server entirely.

The Real gRPC Service Definition

The real protobuf schema, published in the kubernetes-csi/external-snapshot-metadata repository, defines a service with exactly two RPCs:

service SnapshotMetadata {
  rpc GetMetadataAllocated(GetMetadataAllocatedRequest)
      returns (stream GetMetadataAllocatedResponse) {}
  rpc GetMetadataDelta(GetMetadataDeltaRequest)
      returns (stream GetMetadataDeltaResponse) {}
}

message BlockMetadata {
  int64 byte_offset = 1;
  int64 size_bytes = 2;
}

GetMetadataAllocated answers “which parts of this single snapshot actually have data in them,” useful for a first, base backup of a sparse volume. GetMetadataDelta answers “which parts changed between this base snapshot and this later target snapshot,” exactly the question changed_block_indices() answers in this tutorial, and it takes a base_snapshot_id plus a target_snapshot_name as arguments for that reason. Both RPCs return a list of BlockMetadata messages, and as you can see, the real schema uses the same byte_offset and size_bytes field names this tutorial’s BlockMetadata dataclass borrowed on purpose.

Notice both RPCs return a stream of responses, not one single response. A real volume can be measured in terabytes, so its full changed-range list might not fit comfortably in one gRPC message or in memory at once. That is also why both request messages include a starting_offset field: per the spec’s own comments, a client “should specify this value to be the offset of the byte position immediately after the last byte of the last data range received, if continuing an interrupted operation,” which is the production-grade version of resuming a paused download. This tutorial’s 10 MiB test volume never needed that, since its entire changed-range list comfortably fits in a Python list in memory, but a real backup vendor’s client code has to handle exactly this.

What Changed in the September 2026 Beta

This feature shipped as Alpha in September 2025, and moved to Beta with the external-snapshot-metadata project’s v1.0.0 release in March 2026. A September 14, 2026 Kubernetes blog post by Prasad Ghangal of Veeam Kasten (a Kubernetes-native backup vendor, and exactly the kind of company that consumes this API) walks through what changed in that Beta release: the SnapshotMetadataService CRD was promoted from v1alpha1 to v1beta1, now served at cbt.storage.k8s.io/v1beta1, with the schema itself left unchanged. The feature still applies only to block volumes; file-volume and network file-share changed-list tracking are explicitly out of scope. The announcement’s own “Trying it out” section puts the real-driver checklist plainly: make sure your CSI driver supports volume snapshots and ships the external-snapshot-metadata sidecar, install the v1beta1 CRD, create a SnapshotMetadataService resource for that driver, and then use a client, either the project’s own snapshot-metadata-lister reference tool or your own implementation, to call GetMetadataAllocated and GetMetadataDelta. The project’s own hostpath CSI driver example is the suggested starting point for seeing the full flow end to end on a real cluster.

What This Tutorial Deliberately Simplifies

The CSI spec does not mandate how a driver computes GetMetadataDelta internally, only the wire contract for reporting the result. This tutorial computed changes by checksumming every block after the fact and comparing, which is simple to build with nothing but the standard library, but it is not necessarily how a production storage system does it. Many real storage backends track a write-time dirty bitmap that gets updated the instant a block is written, so answering “what changed” never requires re-reading and re-hashing the whole volume; others rely on copy-on-write snapshot metadata they already maintain for other reasons. The checksum-and-compare approach used here is a teaching approximation chosen specifically because it needs no block-device access and no storage driver at all, not a claim about how any particular CSI driver is implemented.

Common Mistakes and Gotchas

• Reaching for file size or modification time instead of content. A raw block device does not have a meaningful “last modified” timestamp the way a regular file does, and its size never changes on a write. This absence is part of why block-level checksumming or dirty-bitmap tracking exists in the first place: there is no cheap metadata shortcut available below the filesystem layer.
• Picking a block size without measuring your real write pattern. Step 7 showed the same 80 real changed bytes turning into anywhere from 10,240 to 1,310,720 bytes of incremental payload purely based on block size. A block size tuned for a database’s small, scattered page writes will be badly wrong for a workload that rewrites data in large contiguous chunks, and vice versa.
• Applying incrementals out of order, or skipping one in a chain. Step 6 showed this produces no error, no warning, and a file of the exact correct size, while still being silently wrong in every block that an earlier, skipped incremental was responsible for. Always verify a restore chain’s ordering and completeness before trusting an incremental restore, ideally with a whole-volume checksum comparison like this tutorial’s sha256_file() check, not just “did the restore command exit with status 0.”
• Assuming a diff with zero false positives means a diff with zero false negatives. This tutorial’s SHA-256-based diff never produced a false positive across any of its runs, which is expected given how collision-resistant SHA-256 is, but that says nothing about a different implementation. A dirty-bitmap-based tracker that fails to record a write during a crash, for instance, could report a block as unchanged when it was not, which is a false negative and far more dangerous than a false positive, since it silently drops real data from the backup with no visible symptom at all.

Step 8: Verify Everything With an Automated Test Suite

Manually running scripts and eyeballing output is how you build understanding, but a test suite is how you keep the guarantees in place as code changes. Save this as test_cbt.py in the same directory:

"""Automated tests for the changed-block-tracking building blocks."""
import random

from cbt_lib import (
    checksum_blocks,
    changed_block_indices,
    merge_into_ranges,
    write_incremental_backup,
    apply_incremental,
    sha256_file,
    BlockMetadata,
)


def make_volume(path, size, seed):
    rng = random.Random(seed)
    with open(path, "wb") as f:
        f.write(rng.randbytes(size))


def test_checksums_are_deterministic(tmp_path):
    path = tmp_path / "vol.bin"
    make_volume(path, 4096 * 5, seed=1)
    first = checksum_blocks(path, 4096)
    second = checksum_blocks(path, 4096)
    assert first == second


def test_no_changes_detected_for_identical_snapshots(tmp_path):
    path = tmp_path / "vol.bin"
    make_volume(path, 4096 * 5, seed=1)
    snap_a = checksum_blocks(path, 4096)
    snap_b = checksum_blocks(path, 4096)
    assert changed_block_indices(snap_a, snap_b) == []


def test_a_single_modified_byte_flags_exactly_one_block(tmp_path):
    path = tmp_path / "vol.bin"
    make_volume(path, 4096 * 5, seed=1)
    base = checksum_blocks(path, 4096)
    with open(path, "r+b") as f:
        f.seek(4096 * 2)  # inside block index 2, not on a boundary
        f.write(b"\x00")
    target = checksum_blocks(path, 4096)
    assert changed_block_indices(base, target) == [2]


def test_growing_the_volume_flags_the_new_blocks(tmp_path):
    path = tmp_path / "vol.bin"
    make_volume(path, 4096 * 3, seed=1)
    base = checksum_blocks(path, 4096)
    with open(path, "ab") as f:
        f.write(random.Random(2).randbytes(4096 * 2))
    target = checksum_blocks(path, 4096)
    assert changed_block_indices(base, target) == [3, 4]


def test_merge_into_ranges_merges_only_adjacent_blocks():
    ranges = merge_into_ranges([0, 1, 2, 5, 6, 10], block_size=100)
    assert ranges == [
        BlockMetadata(byte_offset=0, size_bytes=300),
        BlockMetadata(byte_offset=500, size_bytes=200),
        BlockMetadata(byte_offset=1000, size_bytes=100),
    ]


def test_merge_into_ranges_handles_empty_input():
    assert merge_into_ranges([], block_size=100) == []


def test_incremental_backup_roundtrip_reconstructs_exact_bytes(tmp_path):
    volume = tmp_path / "vol.bin"
    base_backup = tmp_path / "base.bin"
    make_volume(volume, 4096 * 8, seed=5)

    import shutil

    shutil.copyfile(volume, base_backup)
    base_sums = checksum_blocks(base_backup, 4096)

    # Modify two non-adjacent blocks.
    with open(volume, "r+b") as f:
        f.seek(4096 * 1)
        f.write(random.Random(9).randbytes(4096))
        f.seek(4096 * 6)
        f.write(random.Random(10).randbytes(4096))

    target_sums = checksum_blocks(volume, 4096)
    changed = changed_block_indices(base_sums, target_sums)
    ranges = merge_into_ranges(changed, 4096)
    assert len(ranges) == 2  # the two writes were not adjacent

    delta_path = tmp_path / "delta.bin"
    manifest_path = tmp_path / "delta_manifest.json"
    write_incremental_backup(volume, ranges, delta_path, manifest_path)

    restored = tmp_path / "restored.bin"
    shutil.copyfile(base_backup, restored)
    apply_incremental(restored, delta_path, manifest_path)

    assert sha256_file(restored) == sha256_file(volume)


def test_incremental_of_incremental_only_works_when_applied_in_order(tmp_path):
    import shutil

    volume = tmp_path / "vol.bin"
    base_backup = tmp_path / "base.bin"
    make_volume(volume, 4096 * 8, seed=20)
    shutil.copyfile(volume, base_backup)
    snap0 = checksum_blocks(base_backup, 4096)

    with open(volume, "r+b") as f:
        f.seek(4096 * 1)
        f.write(random.Random(21).randbytes(4096))
    snap1 = checksum_blocks(volume, 4096)
    ranges1 = merge_into_ranges(changed_block_indices(snap0, snap1), 4096)
    write_incremental_backup(volume, ranges1, tmp_path / "d1.bin", tmp_path / "m1.json")

    with open(volume, "r+b") as f:
        f.seek(4096 * 4)
        f.write(random.Random(22).randbytes(4096))
    snap2 = checksum_blocks(volume, 4096)
    ranges2 = merge_into_ranges(changed_block_indices(snap1, snap2), 4096)
    write_incremental_backup(volume, ranges2, tmp_path / "d2.bin", tmp_path / "m2.json")

    correct = tmp_path / "correct.bin"
    shutil.copyfile(base_backup, correct)
    apply_incremental(correct, tmp_path / "d1.bin", tmp_path / "m1.json")
    apply_incremental(correct, tmp_path / "d2.bin", tmp_path / "m2.json")
    assert sha256_file(correct) == sha256_file(volume)

    skipped = tmp_path / "skipped.bin"
    shutil.copyfile(base_backup, skipped)
    apply_incremental(skipped, tmp_path / "d2.bin", tmp_path / "m2.json")
    assert sha256_file(skipped) != sha256_file(volume)

Run it with pytest test_cbt.py -v:

============================= test session starts =============================
platform win32 -- Python 3.13.14, pytest-9.1.1, pluggy-1.6.0
collected 8 items

test_cbt.py::test_checksums_are_deterministic PASSED                     [ 12%]
test_cbt.py::test_no_changes_detected_for_identical_snapshots PASSED     [ 25%]
test_cbt.py::test_a_single_modified_byte_flags_exactly_one_block PASSED  [ 37%]
test_cbt.py::test_growing_the_volume_flags_the_new_blocks PASSED         [ 50%]
test_cbt.py::test_merge_into_ranges_merges_only_adjacent_blocks PASSED   [ 62%]
test_cbt.py::test_merge_into_ranges_handles_empty_input PASSED           [ 75%]
test_cbt.py::test_incremental_backup_roundtrip_reconstructs_exact_bytes PASSED [ 87%]
test_cbt.py::test_incremental_of_incremental_only_works_when_applied_in_order PASSED [100%]

============================== 8 passed in 0.14s ==============================

Notice these tests use pytest’s tmp_path fixture for every file, which gives each test its own throwaway directory rather than reusing the volume.bin from the earlier steps. That matters because the two chain-ordering tests deliberately construct a fresh, small, three-block scenario where the “corrupted” result is checked with a direct assertion rather than a printed number, which is what makes this a regression test and not just a demo: if a future change to apply_incremental or merge_into_ranges ever broke the ordering guarantee, test_incremental_of_incremental_only_works_when_applied_in_order would fail immediately instead of silently producing wrong backups in production.

Verify the Whole Thing End to End

To confirm everything in this tutorial reproduces on your own machine, create a fresh directory, save all seven files (cbt_lib.py, step1_setup.py through step6_blocksize_tradeoff.py, and test_cbt.py), and run them in order:

python step1_setup.py
python step2_full_backup.py
python step3_first_change_round.py
python step4_incremental_backup.py
python step5_second_round_and_chain.py
python step6_blocksize_tradeoff.py
pytest test_cbt.py -v

Because every random number generator in this tutorial is explicitly seeded, every number in this post, the 42 changed blocks, the 14 merged ranges, the 98.4 percent savings, the 39 silently corrupted blocks, the block-size overhead table, should reproduce exactly on any machine running a compatible Python version. If any of those numbers come out different, that is a signal something in the code was transcribed incorrectly, not that the concept is unreliable.

Next Steps

From here, a natural next step is trying this against a real cluster once you have Docker or another container runtime available: spin up a local kind cluster, install the CSI hostpath driver from the kubernetes-csi/csi-driver-host-path project along with its external-snapshot-metadata sidecar, and call the real GetMetadataDelta RPC with a gRPC client instead of this tutorial’s in-process Python functions. The official CSI developer documentation and the Kubernetes enhancement proposal (KEP-3314) are the best starting points for that.

If you found the checksum-and-compare technique in this tutorial interesting, this site’s Merkle tree tutorial builds a related but different structure for detecting tampering across a whole dataset using far fewer comparisons than checksumming every block individually. For the durability half of the backup story (making sure a write is never lost even if the process crashes mid-operation), see the write-ahead log tutorial. And for the equally important, opposite concern to this post (verifying that a backup you already took can actually be restored, rather than efficiently capturing it in the first place), see the CNCF disaster recovery guide on why backups are an unreliable signal until you test restoring them.