CNCF’s Disaster Recovery Guide Turns Kubernetes Backups Into an Unreliable Signal

A backup job that reports Completed tells you that a process ran without throwing an error. It does not tell you that the data actually landed somewhere else, that it landed in a state your application can use, or that restoring it will put your system back the way it was. A guide published on the CNCF blog this week walks through three reproducible Kubernetes failure scenarios that expose exactly that gap, and argues that most teams are testing the wrong thing when they think they are testing disaster recovery.

The piece, Kubernetes disaster recovery: Guidance from three reproducible failure scenarios, was written by CNCF Ambassadors Saiyam Pathak and Saloni Narang, who both help lead the Kubesimplify cloud native education community. Pathak also chairs CNCF’s TAG Operational Resilience, the group that hosts the broader recovery-guidance effort this piece feeds into, which gives it more than the usual blog-post authority: it reads like a practitioner’s field notes rather than a vendor pitch, and it deliberately scopes itself away from compliance checklists to focus on one narrow question: when you actually try to recover a stateful application in Kubernetes, where does the process break?

What “Backup Complete” Actually Promises

The authors frame disaster recovery around two standard reliability metrics: Recovery Point Objective (RPO), how much data you can afford to lose, and Recovery Time Objective (RTO), how long you can afford to be down. Both numbers are meaningless unless recovery actually works, and the guide breaks a full recovery into four layers that all have to come back together: cluster infrastructure, workload definitions, persistent data, and traffic routing.

Its central observation is that teams routinely verify each layer in isolation and declare victory. A cluster rebuilds cleanly. A GitOps pipeline reconciles every manifest. A volume snapshot finishes without error. Each of those checks can pass on its own while the seams between them silently fail, and the three scenarios in the guide are built to reproduce exactly those seam failures rather than any single component breaking outright.

To make the scenarios reproducible rather than theoretical, the authors built a lab with a production Kubernetes cluster, a separate recovery cluster, an S3-compatible backup store, and a Git service, running a PostgreSQL workload seeded with data whose contents can be checked after a restore, not just whether the restore command exited successfully. The full lab is published on GitHub as kubecon-japan-dr-demo, so any of the three failures can be reproduced rather than taken on faith.

Scenario One: The Backup That Says Done but Isn’t

The first scenario targets the most common blind spot: trusting a backup tool’s own success status. Using Velero, the CNCF Sandbox project for backing up and migrating Kubernetes applications and their persistent volumes, the guide shows that a Completed status on a backup job confirms only that Velero’s process finished, not that the underlying volume bytes were fully transferred to external storage or that the application’s data was in a consistent state when the snapshot was taken.

Getting real consistency requires pre- and post-backup hooks that pause an application’s writes before the snapshot and release them afterward. Velero’s own documentation describes this with a filesystem-freeze example: the hook exists specifically so that “all pending disk I/O operations have completed prior to taking a snapshot.” Left unconfigured, Velero will still report success, because from its point of view nothing went wrong; it simply never asked the application to pause. The scenario also flags a second, easy-to-miss failure mode: restoring into different infrastructure than the backup was taken from often requires an explicit storage class mapping, and skipping that step can leave a restore silently pointed at the wrong storage backend.

Scenario Two: A Green GitOps Dashboard, an Empty Database

The second scenario is the one most likely to catch experienced teams off guard, because it produces a dashboard that looks entirely healthy. GitOps controllers reconcile a cluster’s actual state against manifests stored in Git: they recreate deployments, services, and persistent volume claims exactly as declared. What they do not do is restore the data that used to live inside those volumes, because that data was never described in Git to begin with.

In the lab, a full GitOps sync completed cleanly, every resource showed green, and the database underneath was empty. The manifests were correct; the intent was correct; the data simply was not there, because reconciling declared state and restoring stored state are two unrelated operations that happen to look similar on a status page. The fix the guide walks through is not a single tool but a sequence: remove the empty, falsely-healthy application first, then restore it from an actual data backup rather than a fresh GitOps sync. Git recovers what the system is supposed to look like; a backup tool recovers what was actually in it. Treating either one as sufficient on its own is what produces the empty-database-with-a-green-dashboard failure.

Scenario Three: Two Snapshots, One Broken Invariant

The third scenario covers applications that span more than one persistent volume, a common shape for anything with separate storage for, say, orders and payments. Snapshotting each volume independently, even moments apart, can produce a restore where each individual volume looks fine but the combined dataset violates the application’s own rules. The guide’s concrete example: a restore that left 25 payment records with no matching orders, because the payments volume was snapshotted at a slightly different instant than the orders volume.

The fix is a feature that only recently became fully supported: VolumeGroupSnapshot, which reached General Availability in Kubernetes v1.36 under the groupsnapshot.storage.k8s.io/v1 API group after an alpha in v1.27 and two beta cycles. Kubernetes’ own release announcement describes it as an API that lets a CSI driver “take crash-consistent snapshots for a set of volumes” at the same point in time, so a restore rehydrates every related volume together instead of stitching together snapshots taken at different moments. Mechanically, Kubernetes selects which PersistentVolumeClaims belong together using label selectors, the same pattern shown in the CSI project’s own documentation, where PVCs are tagged with a shared label like app.kubernetes.io/name=postgresql before a group snapshot is requested. The feature is only supported for CSI volume drivers, so it is worth checking which driver a given storage backend uses before assuming group snapshots are available.

What Actually Counts as a Recovery Test

The guide draws a sharp line between two things that get called “testing disaster recovery” but are not the same exercise. As the authors put it plainly: “A recovery test is not deleting a pod and watching it return; that tests workload reconciliation.” Kubernetes is very good at reconciliation, restarting a killed pod is close to instant, and that speed can quietly convince teams their recovery story is solid when all they have actually exercised is the scheduler.

A real recovery test, by the guide’s definition, restores a complete application into a genuinely clean target, checks that the restored data matches expected contents rather than just confirming resources exist, and measures how long the whole process took against the RTO the business actually needs. None of the three failure scenarios above would be caught by a reconciliation test, since in every case the cluster’s resources looked fine; only checking the data itself, after a real restore, surfaced the problem.

Where the Ecosystem Still Has Gaps

The guide is candid about what Kubernetes and its surrounding tooling still do not solve. It names three open gaps: there is no standard contract for failing over an application between clusters, core Kubernetes has no built-in concept of an application-level “recovery unit” that bundles a workload with its data and its dependencies for backup purposes, and the industry default is still to treat a completed backup job as proof that recovery works, with no forced rehearsal step in between.

That last gap is the throughline connecting all three scenarios: a green backup job, a green GitOps sync, and a green set of individual volume snapshots were each, on their own, real and accurate signals. None of them was a recovery guarantee, and nothing in a typical pipeline forces a team to find that out before an actual incident does.

An Open Invitation, Not a Finished Standard

The authors are explicit that this is a snapshot of current practice rather than a finished framework. The write-up sits under a broader effort called the Cloud Native Business Continuity initiative, which operates as an open proposal under CNCF’s TAG Operational Resilience and is actively looking for contributors to help build “a landscape gap analysis, updated backup and DR guidance, and reference architectures” for the wider ecosystem. For teams running stateful workloads on Kubernetes, the more immediately useful takeaway is smaller and more concrete: pull the lab repository, reproduce at least one of the three scenarios against your own stack, and check whether your own “backup complete” notification would have caught it.

For related coverage of how the CNCF community is turning individual practitioner experience into shared operational guidance, see sxz.io’s earlier pieces on Kyverno’s shift from a security gate to a platform primitive and on using Kubernetes’ ExternalName service to de-risk a namespace migration, both similarly grounded in a practitioner’s own hands-on experience rather than a vendor announcement.