A major cloud platform disruption in region eu01 occurred between 23:45 UTC on July 08, 2026 and 17:17 UTC on July 09, 2026. The impact of the incident evolved over time and was driven by slow response times of the object storage service.
The Incident was triggered by the monthly Kubernetes patch day. During scheduled patch days, automated updates of Kubernetes worker nodes are applied to keep the operating system secure and up to date. This update process takes place during a maintenance time window specified by the customer. Operating system images and disk volumes are distributed across shards across the platform. In the days preceding this event, new shards had been added to the infrastructure. These shards were designed to populate their caches by downloading operating system images only upon first use. This pre-warming of the caches did not work as expected leading to a state where common images had not been pre-cached during normal operations. This caused high response times in the object storage service, as well as delays in volume provisioning for block storage.
When the automated updates of Kubernetes worker nodes started, despite being spread over time, it caused the redistribution of many volumes to these new, un-cached shards. This influx of activity resulted in widespread cache misses, which subsequently triggered a large volume of simultaneous image pulls rather than waiting for the completion of the first download. This surge overwhelmed the object storage service, causing significant performance degradation. While the specific reasons for the failure of existing protective mechanisms are still under investigation, the degradation of the object storage service had cascading effects on dependent services throughout the platform. A small fraction of database instances experienced “file-system-full” conditions as log backups failed to complete in a timely manner. Observability services were unable to query long-term metrics, and configuration changes for Cloud Foundry could not be applied. Additionally, Kubernetes lost the ability to schedule new workers, which caused the patching process to run beyond its designated window and prevented auto-scale functionality from operating as expected.
Implemented backend caching to successfully improve overall caching efficiency and performance across the platform.
Added additional network paths and devices to reduce traffic load and resolve the overload situation on the affected network infrastructure.
Extended the scheduled timeframe for SKE updates and patches to prevent highly concentrated system loads within a short period.
Deployed additional load balancers for the Object Storage service to better distribute and manage periods of increased load.