Instance operations stuck in pending

[Geoffrey Arnold]

Hello,

Around 6:37PM CT today our primary (us-central-b) and and failover (us-central-a) databases are stuck with pending operations. The primary is lists the following operations:

Delete backup Operation is pending

Backup Operation is pending

Jun 28, 2019, 6:29:37 PM Failover Failover operation in progress

The failover lists the following operations:

Recreate replica Operation is pending

I am unable to connect either the primary or failover databases.

Any help would be greatly appreciated,

Geoff. 

 

[Geoffrey Arnold]

Looks like some trouble in us-central1-b:

Anyone else experiencing issues?

[Ken Winke]

Yep, I'm in the same boat with a primary in central-b and failover in central-a. Operations show processes running the failover a few hours ago but it's completely hung, and I can't stop/restart/fail/clone/restore or do anything to get a service running again.

[Austin Quinn]

I too am having this issue. Best part is I cannot restore a backup to a new instance. Great design!

master instance is in us-central1-b and has the below:

Failover has this:

[Ken Winke]

I noticed my instance and failover came back as running, but they aren't operational. Any command fails due to ongoing operations still even though it's showing memory and storage usage again. The primary is unable to show users/databases from the gcp portal. Restarts/failovers don't work and I can't change locations on it, delete the replica, or even connect to mysql via the google shell due to the ongoing operations error still.

I see how to list operations, but there's no way to cancel or kill them.

On Friday, June 28, 2019 at 7:02:26 PM UTC-5, Geoffrey Arnold wrote:

[Elliott (Google Cloud Platform Support)]

Hello Ken,

Please note that Google Groups are reserved for general Google Cloud Platform-end product discussions and not for reporting issues, which is why I suggest moving the troubleshooting to Issue Tracker[1], where issues can be turned private in case we need to gather any project specific details.

[1] https://issuetracker.google.com/issues/new?component=187134&template=0