Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations.
Stop and delete operations between Wednesday, 2020-01-29 04:00 US/Pacific and 09:00 US/Pacific may have failed or appear stuck. Stop and delete operations created after 09:00 US/Pacific should complete as expected, even if they had previously failed or are stuck.
Mitigation is underway and we are currently working through the backlog of operations.
Currently running instances are not impacted. Other zones in us-central1 are not impacted.
Products which depend on GCE may also be unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore
We will provide an update by Wednesday, 2020-01-29 11:00 US/Pacific with current details.
Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”.
Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation.
Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.