Summary

• An electrical control misconfiguration prevented backup generators from starting automatically during the 20 October outage in Google's London region.
• Google's 4 November report identifies additional recovery problems, including incomplete Cloud SQL failovers, Kubernetes disruption, and an Apigee configuration that placed two database replicas in one zone.
• The principal regional impact lasted six hours and eight minutes, but individual services experienced longer disruption and some remediation work remains outstanding.

Google Cloud has identified additional weaknesses in its London infrastructure following October’s major outage, revealing how a failed generator control system was compounded by database recovery problems, insufficient spare computing capacity, and software components that were not adequately separated between availability zones.

The company’s incident report, updated on 4 November, provides a more detailed account of the disruption on 20 October, when an external electrical fault took equipment in the europe-west2-b zone offline. The main impact lasted six hours and eight minutes, but some managed services experienced longer interruptions while their underlying systems recovered.

Among the new findings, Google says only 48.5% of regional high-availability Cloud SQL instances completed their automated failover successfully. Google Kubernetes Engine experienced disruption lasting seven hours and 34 minutes, while an Apigee configuration error left two of three database replicas in the same zone that lost power.

The additional findings extend the investigation beyond the electrical equipment responsible for the initial failure. They show how supposedly resilient software arrangements can become dependent on shared infrastructure or configuration choices that are exposed only during an unusual failure sequence.

Backup generation failed during maintenance

The affected facility normally operates with two independent utility power paths, supported by emergency generators and uninterruptible power supply batteries. That arrangement is intended to allow critical computing equipment to remain operational when an external electricity source becomes unavailable.

However, the facility was temporarily operating with one primary feed while corrective work was carried out following a generator load test on 14 October. The second feed had been disconnected and its circuit breaker locked open, leaving the remaining feed responsible for supplying the critical load.

At 06:53 BST on 20 October, an underground cable fault interrupted that remaining supply. The emergency generator controls should have detected the loss of electricity and started the generator plant automatically.

Instead, programmable logic controllers detected voltage on the disconnected feed because their monitoring points were located upstream of its open circuit breaker. The software did not adequately consider the breaker position, so both redundant controllers incorrectly determined that a usable power source remained available.

The generators consequently failed to start. Battery systems maintained the load for approximately five minutes, but the site lost power at 07:00 BST when their available reserves were exhausted.

This was a common configuration failure affecting duplicated controllers rather than the failure of a single controller that its redundant counterpart could overcome. The equipment had redundancy, but both control paths made the same incorrect assessment.

Site engineers restored power through a standby utility arrangement, with electricity returning to the data halls by 08:20 BST. Restarting the computing systems required additional work because storage devices and other equipment had shut down unexpectedly.

Recovery problems extended across the region

The physical outage affected europe-west2-b, one of the three zones making up Google’s London region. Yet customers using resources elsewhere in the region also experienced errors and delays as applications redirected work away from the unavailable facility.

Some surviving infrastructure became congested when redirected database operations and computing workloads consumed available capacity. Other services continued trying to access metadata or management components associated with the failed zone, causing timeouts even though their own equipment remained powered.

The updated report establishes that these dependencies affected individual products in different ways. The regional incident duration provides a measure of the main disruption, but it does not describe the full recovery period for every service.

Cloud SQL initiated automatic failover for regional high-availability instances, although only 48.5% completed that process successfully at the outset. Recovery of the remaining instances was delayed principally by problems involving Regional Persistent Disk, including storage device stalls.

By 13:15 BST, Google says 99.95% of Cloud SQL zonal instances had recovered automatically. The distinction between zonal instances and regional high-availability configurations is important because their architectures and expected recovery behaviour differ.

Google Kubernetes Engine experienced a longer period of disruption as the failed zone lost its worker nodes and surviving zones attempted to accommodate displaced workloads. Autoscaling was constrained by spare capacity and delays in the underlying Compute Engine interfaces.

The Kubernetes incident lasted seven hours and 34 minutes, with final resolution at 14:37 BST. The service’s recovery illustrates how additional computing capacity cannot necessarily be created quickly when a regional failure has already increased demand across the remaining infrastructure.

Replica placement weakened Apigee resilience

The investigation also identified a separate configuration problem in Apigee X, Google’s application programming interface management service.

Its underlying datastore was intended to retain three replicas distributed across the region’s availability zones. Instead, a scheduling misconfiguration had placed two replicas in europe-west2-b, leaving the service without the required quorum when that zone lost power.

As a result, a small number of affected Apigee clusters experienced elevated server errors for seven hours and 19 minutes. Google says it has rebalanced unevenly distributed replicas and introduced monitoring intended to detect similar placement problems.

The company also plans to enforce placement rules ensuring that each datastore replica occupies a different zone. The change addresses a failure that was not caused directly by the generators, but which made the consequences of their failure more severe.

AlloyDB encountered another form of recovery difficulty. Regional primary instances experienced delays of up to 95 minutes in receiving automated failover notifications, while approximately 4% experienced stalled failovers requiring manual intervention because of Persistent Disk problems.

More than 99% of affected AlloyDB instances had recovered by 14:16 BST, with the final zonal primary instances returning by 16:28 BST. Other products, including BigQuery, Dataflow, and Looker, experienced different combinations of queuing, connectivity, and recovery delays.

Remediation extends beyond electrical controls

Google reconfigured the generator controllers after the incident and completed further validation on 26 October. Its published actions also include revised emergency procedures and additional testing of the electrical switching arrangements.

The company intends to check similar controller configurations at other facilities, with 15 November identified as the target for its wider validation exercise. Several other remedial measures have expected completion dates, rather than published confirmation that every activity is finished.

At the cloud service layer, engineering teams are reviewing full-cluster power loss testing, failure detection, storage recovery, and the way regional services behave when an availability zone disappears unexpectedly.

Those changes address two different requirements: preventing a similar physical failure and ensuring that a future zone failure does not produce unnecessary disruption in services designed to remain available elsewhere in the region.

The updated findings also qualify how resilience should be assessed by customers. A workload distributed across zones may still encounter problems if its database, management components, or recovery arrangements depend on resources unavailable during a failure.

Some individual Compute Engine failovers were unsuccessful because of limited capacity or difficulties replicating their storage. Google’s report also explains that engineers delayed parts of the recovery to avoid risking data loss after the unexpected shutdown of storage systems.

Such precautions can lengthen restoration times, but they involve a different operational consideration from simply restarting a failed server. Recovering data-dependent applications requires confidence that storage and associated metadata remain consistent.

Google’s latest findings provide a fuller account of the London disruption than its initial status updates. An electrical fault exposed a controller configuration error, while the subsequent recovery revealed weaknesses in capacity planning, service dependencies, and database replica placement. The remaining remediation work will determine how those lessons are incorporated into the wider platform.