URGENT - Google Cloud SQL Incident #17015 - not even Google's on-demand backup running for all Cloud SQL gen 1 instances

[Rob]

We have about a dozen Google Cloud SQL gen 1 instances in production.

Per Google Cloud SQL Incident #17015 backups were failing on all of them for about a week before Google noticed. Ugh.

Per "Daily backups are being taken for all Cloud SQL First Generation instances" note that as of this post we STILL we have two instances that are NOT being backed up by google - the "on demand" backs are not being run for some reason on two of our instances.

(happy to PM which two instance names are having issues)

[Marilu (Cloud Platform Support)]

Hello Rob,

I'll recommend monitoring the Google Cloud Status Dashboard to get updates on this matter, as our engineers are still working on a fix for this issue. We apologize for any inconvenience you may be experiencing.

If the issue is marked as fixed on the Status Dashboard and you're still experiencing issues the two instances that you mentioned, I'll suggest opening a public Issue tracker and we'll be happy to further investigate. 

As a workaround, I'll suggest doing a manual backup using any of the Database Backup Methods for MySQL, since for first generation instances you can't create an on-demand backup.

I hope this helps,

Marilu

[Marilu (Cloud Platform Support)]

The Dashboard reports now that the issue was fixed yesterday, if you're still experiencing some issues with automatic backups for the two instances that you mentioned, let us know and we'll be happy to further investigate.