CNCF’s Linkerd Playbook Turns Kubernetes Failover From a Runbook Into a Label

Every team running Kubernetes across more than one region eventually hits the same failure mode: a cluster goes down, and the identical copy of the service running in another region might as well not exist, because nothing is wired to treat the two as one thing. A July 27 CNCF Blog post by Dominik Táskai, a CNCF Ambassador and Linkerd contributor, walks through how Linkerd’s multicluster extension closes that gap, and in doing so shows just how much operational nuance sits underneath the phrase “multi-region Kubernetes.”

Failover Has Been a Runbook, Not a System

Táskai’s framing is blunt: without a mesh layer, failover today usually means restoring service, repointing DNS, and waiting out an outage that the extra region was supposed to prevent in the first place. Linkerd’s multicluster extension is built to remove that DNS-repoint step entirely. It lets several clusters present a single service as one load-balanced endpoint, so a client calling that service does not need to know or care which cluster actually answers.

The post’s real contribution is pointing out that Linkerd’s official task docs demonstrate each multicluster mode in isolation, and that is not how production platforms actually use them. “A real platform almost never picks one multicluster mode,” Táskai writes. Some services want automatic failover with zero application changes, others need to reach one specific remote service by name for data locality reasons, and many platforms need both patterns active on the same set of linked clusters at the same time.

Three Ways to Link Clusters, and Why Real Platforms Need More Than One

Linkerd’s own documentation confirms the shape of the problem: the project supports three basic forms of multi-cluster communication, hierarchical (labeled “gateway” in practice), flat, and federated. Each one is selected per service through a Kubernetes label rather than a cluster-wide setting, which is what makes mixing all three on the same set of clusters possible.

Gateway Mode: Works on Almost Any Network

In gateway mode, Linkerd deploys a gateway component on the target cluster and routes cross-cluster requests through it. Per Linkerd’s documentation, this is the least demanding option on the network: it works on almost any topology because it only requires that the gateway IP of the destination cluster be reachable by pods on the source cluster. The label for this mode is mirror.linkerd.io/exported=true, and it mirrors the remote service locally as <svc>-<cluster>.

Flat Mode: Direct Pod-to-Pod Routing

Flat mode skips the gateway entirely. Traffic goes straight to remote pods, which requires pod IP addresses to be routable across clusters, in other words a flat network. Linkerd’s documentation dates this capability to Linkerd 2.14. Its label is mirror.linkerd.io/exported=remote-discovery, and per the CNCF post it is the mode to reach for when “data locality matters” and the application needs to address a specific cluster’s copy of a service by name.

Federated Mode: One Name, Automatic Failover

Federated mode is built for the failover story specifically. Every same-named service across the linked clusters is unioned into a single virtual service, written as <svc>-federated, and load-balanced automatically across the clusters. Its label is mirror.linkerd.io/federated=member. This is what lets a client keep calling one name while Linkerd routes around a cluster that has gone down, with nothing in the application itself needing to change.

Picking the Right Mode Per Service

The CNCF post reduces the choice to a short set of questions. Does a service need automatic failover with no application changes? Use federated. Does it need traffic splitting with explicit weights? Use a flat mirror combined with a TrafficSplit. Is the service a singleton that only exists in one cluster? Use a plain mirror, gateway or flat depending on what the network allows. Because the mode lives on a per-service label instead of a cluster-wide flag, a platform team can run all three at once, which is exactly what the post’s own demo does.

Wiring Three GKE Clusters Into One Mesh

To make the tradeoffs concrete, the post builds a working reference architecture: three Google Kubernetes Engine clusters, named west, east, and north, connected through full-mesh VPC peering (west to east, east to north, and north to west) with the --export-custom-routes and --import-custom-routes flags enabled on every peering connection, so pod CIDRs actually get advertised between the networks. Each cluster gets non-overlapping pod and service CIDRs, which the post calls a hard requirement for the peering to function at all.

A Shared Trust Anchor for Cross-Cluster mTLS

Before any traffic crosses a cluster boundary, the demo installs Linkerd on all three clusters using a shared trust anchor: one root certificate authority issuing separate per-cluster issuer certificates for west, east, and north. That structure is what lets each cluster verify the identity of workloads running in the other two. It lines up with Linkerd’s own stated design goal of a unified trust domain, in which source and destination identity are validated at every hop, whether the call stays inside one cluster or crosses the public internet.

With the mesh installed, the demo deploys three services that each exercise a different mode at once: a frontend service running federated, an api service running flat-mirrored, and an analytics service running gateway-mirrored, all inside the same three-cluster mesh. Then comes the test that matters: the post kills an entire cluster and confirms traffic actually redistributes, instead of just trusting that it would.

The Gotchas the Docs Don’t Warn You About

Three lessons from the build stand out because they would be genuinely painful to debug blind. First, creating VPC peering is not enough by itself. Both sides need the route export and route import flags, or pod CIDRs never get advertised, and the resulting symptom is brutal: DNS resolves fine, then connections simply hang, with nothing obvious pointing at routing as the cause. Second, a regional GKE cluster quietly multiplies node count. Asking for one node with --num-nodes 1 actually produces three nodes, one per zone in the region, so a demo that wants exactly one node per cluster has to pin --node-locations to a single zone. Third, gateway mode and flat mode cannot share a single Link custom resource; a Link is one or the other, so getting both behaviors onto the same cluster means creating two separate Links, one for flat and one for gateway, each with a matching controller on the consuming side.

None of this requires exotic infrastructure. The post notes the entire three-cluster demo runs on three Standard GKE clusters with a single e2-medium node each, at roughly $10 to $15 a day including the management fee and a small gateway load balancer, with a teardown script that removes everything afterward.

What Production Actually Requires

The post closes its technical walkthrough with a checklist that reads like it was written by someone who has been paged for a multicluster failure before. Every cluster should link to every other cluster bidirectionally, so the federated service is complete everywhere. Certificates should come from cert-manager with a shared certificate authority rather than being hand-rolled manually, and every cluster still needs its own separate issuer certificate, a step the post explicitly warns not to skip. NetworkPolicies should restrict cross-cluster traffic to only the services that actually need it, and Linkerd’s own authorization policies should provide finer-grained access control on top of that. Linkerd-Viz metrics belong in an existing Prometheus and Grafana stack, with an alert specifically on a federated service’s endpoint count dropping. Link custom resources and multicluster configuration belong in version control, the same as everything else in a GitOps workflow. And failover itself should be tested on a schedule, not assumed: scale a cluster to zero in staging and confirm that traffic actually redistributes.

The Checklist Item That Doubles as a Chaos Engineering Practice

That last item, testing failover regularly by scaling a cluster to zero in staging, is the same exercise the post’s own demo runs as its chaos test. Turning that one-time demo step into a recurring, scheduled staging exercise is what keeps the other items on the checklist honest instead of theoretical, since certificates rotate, links get added, and configuration drifts long after the initial rollout is done.

Why This Matters Beyond One Service Mesh

Linkerd has been a CNCF project since January 2017, and it moved to Graduated, the foundation’s highest project maturity tier, in July 2021, so this is not an experimental corner of the ecosystem. What the post really documents is a maturation point for multi-region Kubernetes generally: failover is moving from a manual, DNS-based runbook toward a declarative, per-service policy enforced by the mesh layer itself. Kubernetes has no native answer for treating clusters spread across regions as a single logical system, which is exactly the gap projects like Linkerd’s multicluster extension are built to fill.

The honest caveat sits in the post’s own closing line: building this was, in the author’s words, “mostly wiring,” not deep complexity. But the gotchas above show that the wiring is precisely where production rollouts lose real time if nobody on the team has hit them before. For any organization running Kubernetes in more than one region today, the practical takeaway is not to adopt federation everywhere by default. It is that the mode a given service needs, federated, flat, or gateway, is a decision that belongs on a Kubernetes label attached to that service, not buried in a runbook that only gets read once an outage is already underway.

Sources

• CNCF Blog: Federating clusters for zero-downtime Kubernetes
• Linkerd documentation: Multi-cluster communication
• CNCF Projects: Linkerd

Image credit: Data center network aisle showing rack-mounted patch panels and switching equipment. Photo by Robert.Harker, licensed CC BY-SA 3.0 via Wikimedia Commons; cropped, resized, and converted to WebP for sxz.io.