Cannot restore database backups with replicas. GCP outage caused failover operation to get stuck, cannot delete or promote replica.

[Austin Quinn]

GCP had an outage. Failover did not work, now both the master and the failover replica are stuck in a maintenance state. You cannot restore when you have replicas. I cannot delete the replica because it is stuck in maintenance mode. Restoring backups should not depend on any of the old nodes. That defeats the purpose of having backups a bit.

What can we do here? a backup is only as good as the ability to restore.

[Ken Winke]

I was able to get the automated backup restored to a brand new instance I created after this occurred. But I lost the current day of data doing so.

However, my new problem is I'm unable to delete/remove these failed instances from my project, so I'm still being charged for them.

 

[Harmit Rishi (Cloud Platform Support)]

Hello,

Thank you for using Google Groups!

In regards to both of your issues, I believe it would be worth while for support to take a closer look at the Cloud SQL instances involved. Due to this, would you be able to create a private issue with us following the instructions provided here? Making the issue private will ensure that your instance details will not be publicly visible. 

I look forward to assisting you there.