cloud.google.com · Saved

Details of Google Cloud GCVE incident | Google Cloud Blog

cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident

What Jamie wrote

I shared this event in Weekly Thing 289 where Google Cloud literally deleted the account of a customer erasing all information with no backups. Google released this note about what happened.

During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period.

This reminds me of an AWS S3 outage a few years ago also caused by a maintenance activity and an engineer putting the wrong value in a field. These maintenance programs are places where corners often get cut. The team will think "Hey, we have professionals using this. It is okay. if we don't catch every error scenario." But even professionals make mistakes. There is no truly "safe" user.