Google Cloud Suffers a Second Fiber-Maintenance Outage in Two Weeks Despite Promised Fixes

Google Cloud has published a preliminary root-cause report for a partial outage that cut network connectivity to part of its us-central1-b region on September 1: an engineer performing routine hardware maintenance accidentally disconnected every fiber path serving the affected zone. It is the second outage in less than two weeks that Google has traced back to fiber-related maintenance work. The first came on August 20 in the us-west1 region, and it happened despite reforms Google said it would make after that earlier incident.

A 13-Minute Mistake in Iowa

According to Google’s service health report, the September 1 incident began at 07:41 Pacific time and ran until 11:52, just over four hours. During the peak of the outage, “traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss,” Google wrote. Virtual machines inside the affected zone could still reach each other, but lost all contact with anything outside it.

Google designs its cloud data centers with what it calls redundancy across multiple routing devices: fiber paths and the hardware that terminates them are physically separated within each site and run on diverse power sources, specifically so that losing a single device or a single fiber path, or even two or three at once in most cases, does not interrupt customer traffic. That design assumes failures happen one or two at a time. The September 1 incident defeated it by taking out every path at once.

Google’s report names “the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure” as the immediate trigger. A procedural error meant the maintenance action “sequentially unplugged 100% of fiber paths across all devices within 13 minutes,” and the speed and nature of the mistake meant no warning reached the engineer before every path was already disconnected.

The outage reached well beyond compute and container services. Google’s own list of affected products includes AlloyDB for PostgreSQL, Apigee, Cloud Filestore, Cloud Run, Cloud Spanner, App Engine, BigQuery, Cloud Bigtable, Cloud Dataflow, Cloud SQL, Compute Engine, Kubernetes Engine, Hybrid Connectivity, Looker, and Virtual Private Cloud, effectively every layer of the stack that depends on the region’s network fabric.

Recovery took both automation and a physical fix. Google’s engineering teams moved traffic away from the disconnected routers, automatically shifting compatible workloads to healthy capacity elsewhere in the region. But the zone itself did not fully recover until technicians located the disconnected optical links and reseated the fibers by hand. Traffic flow only normalized once that physical work was done, and Google redirected capacity back to the repaired zone afterward.

A Second Fiber Incident in Two Weeks

This was not Google’s first fiber-related outage this quarter. On August 20, a separate incident disrupted services in the us-west1 region, centered on Google’s Oregon data centers, for two hours and 22 minutes. Google’s report on that incident says the trouble “originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers” in the region. Unlike the September incident, the automated systems meant to reroute traffic around the problem failed to work as intended. Google says network congestion set in “as volumes exceeded available bandwidth” once rerouting failed, and that cascaded into “severe packet loss, request throttling, and increased latency” across 33 affected products, including BigQuery, Cloud Storage, Compute Engine, Kubernetes Engine, and IAM.

After the August 20 incident, Google committed to specific fixes: reforming scheduled maintenance protocols, adding stricter safety checks and circuit-redundancy requirements before routine maintenance, and refining automated regional traffic failover mechanisms. Less than two weeks later, a second maintenance procedure on Google’s fiber network caused a second regional outage, this time by defeating the network’s physical redundancy directly rather than exposing a gap in automated failover.

Two Different Failures, One Common Thread

The two incidents are not the same failure. The August 20 outage in us-west1 was primarily a capacity and failover problem: fiber maintenance reduced available bandwidth, and the systems meant to route around that reduction did not do so cleanly. The September 1 outage in us-central1-b was a process-execution problem: a maintenance procedure let one person’s mistake cascade through every redundant fiber path in a single site within 13 minutes, faster than any warning system caught it.

What connects them is where the failure lives. Google’s public case for regional resilience rests on network-layer and software-layer redundancy: diverse fiber paths, diverse power, automated traffic engineering. Stricter safety checks and circuit-redundancy requirements before routine maintenance were explicitly among the fixes Google promised after August 20, and a maintenance procedure that let a technician sequentially disconnect every redundant fiber path in one site without tripping a warning is close to the exact failure those requirements would be expected to catch. Both outages point to the same underlying gap: a person working at the physical layer, during a maintenance window, can still undo the guarantees that redundancy is supposed to provide. The Register, which first reported the September incident, described it as the kind of basic procedural lapse that is easy to overlook at a company that otherwise engineers some of the most redundant infrastructure on the internet.

Google has not said publicly whether the September incident will prompt a further round of maintenance-procedure changes beyond what it promised after August 20. For customers running workloads in a single Google Cloud region or zone, the two incidents are a reminder that a cloud provider’s advertised physical redundancy assumes correlated failures will not happen, an assumption that can still be broken by one person with the wrong cable in hand.