Cloud SQL instance stuck in "another operation was already in progress"

[Freark van der Bos]

Hello out there, 

Interestingly I've got a cloud sql instance (2nd gen mysql 5.7) which is stuck in "another operation was already in progress". I am seeing the following things:

- before my first restart of the instance there was a throbber being animated

- trying to connect to the db using 'gcloud beta sql connect cloudsql-1 --user=root' in a cloudshell gives me 'ERROR: (gcloud.beta.sql.connect) HTTPError 403: Operation failed because another operation was already in progress.'

- trying to do the same from my local terminal results in the same error message.

- my first restart was not possible through the console due to those options being grayed out

- using gcloud from my laptop to restart the database instance was successful, but did not resolve the issue

- the webconsole did not show an animating throbber anymore, visually the issue looked resolved

- for instance, adding an IP to the authorized networks under 'access control' in the console gave me a 'Operation failed because another operation was already in progress.' popup.

- in the stackdriver logging there are no other event logged for this database instance between finishing the restart and attempting to add an IP to the whitelist

- under the operations tab in the web console there are not operations mentioned after the restart (which has the status 'done')

- I disabled 2 applications we're running which can trigger backups and restores of individual database through a service account, in case they were causing trouble

- I restarted the database instance again

- My various operations through the console and gcloud run into the same issues.

My question is, short of creating a new database cluster and using that one instead, what would be my options to get the database instance out of this state?

Regards,

Freark

[Kenworth (Google Cloud Platform)]

Can you send privately your project-id and name of your SQL instance?

[Freark van der Bos]

Seems the database instance is working again this Monday morning. Looking at the operations log it seems that somewhere before Aug 12, 2017, 9:50:23 AM CEST it started working again.

I had also made a bug in you issue tracker (no 64599275) which contains this information privately.

Let me also ask the question I asked there here publicly so the answer might be useful to others: Do you have any knowledge about thing we should or should avoid doing to prevent this from happening in the future?

Thanks for taking the time to look into it!

[Kenworth (Google Cloud Platform)]

Thank you for filing a bug on Public Issue Tracker. Your issue is still currently investigated so I do not have any input yet on how it will be prevented. And since your case is already handled by my colleague, I would recommend redirecting all future  discussions to that issue tracker. Please feel free to contribute on this thread once the investigation is finished. 

[Derek Foster]

Hello, I have exactly the same issue with MySQL 5.7 instance with failover replica on three different projects when trying to delete them - I need these instances deleted but can't do it because of the error.

Is there a known solution yet?

Derek

[Muhannad Virtusa]

Me too. Having the same problem.

[Harmit Rishi (Cloud Platform Support)]

Hi, 

It seems this thread is almost two years old. As the support agent previously mentioned on this thread and the public issue tracker here, we will have to look into the SQL instance for further clarification on your specific issue that you are experiencing. With that being said, you may feel free to open a private issue on our Issue Tracker here. 

Note: Creating a private issue will ensure that the public will not be able to see any confidential information regarding your Google Cloud Platform project or Cloud SQL instance.