Instance is in infinite restarting state
65 views
Skip to first unread message
Armen Haroyan
Feb 10, 2017, 11:59:13 AM2/10/17
to Google Cloud SQL discuss
Hello
Out instance is in infinite restarting state, and we can't do anything with it, any help would be appreciated.
Out instance is in infinite restarting state, and we can't do anything with it, any help would be appreciated.
Kamran (Google Cloud Support)
Feb 10, 2017, 7:31:21 PM2/10/17
to Google Cloud SQL discuss
Could you please send your project ID and instance name to me through private message. I'll be glad to investigate this issue as soon as I receive the information.
Sincerely,
Armen Haroyan
Feb 13, 2017, 3:23:10 AM2/13/17
to Google Cloud SQL discuss
Hello Kamran ! thanks for the response !
We finally were able to contact google and they said that the sql connections must run maximum of 600 seconds, after that it is deadlocked, and they need to manually open it. And now we have same situation again, we can't start it and need to wait for google to respond ....
What can be done ?
We finally were able to contact google and they said that the sql connections must run maximum of 600 seconds, after that it is deadlocked, and they need to manually open it. And now we have same situation again, we can't start it and need to wait for google to respond ....
What can be done ?
Armen Haroyan
Feb 13, 2017, 7:04:56 AM2/13/17
to Google Cloud SQL discuss
2017-02-10T16:02:22.555549Z 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
It appears that the instance had a lock that was held for more than 5 minutes and innodb crashed the server because of that. It's possible there was a deadlock happening.
In order to prevent this in the future we recommend looking over your current queries to see if any could cause a deadlock.
This is the answer from them, so at the time this happened we were running a migration, which was making a connection to sql, accessing 100 databases, altering few tables, closing connection, connecting again, accessing the next 100 databases, doing the changes etc etc...
We have around 13 000 databases. So this occurred when the script was working on 6600th databases.
I was disconnecting and connecting every 100 databases, so the timeout wont bother me again, but again we were deadlocked.
And the greatest issue is that google cloud shows the status of the cloud instance green, as if everything is just good. But you can't access to the instance, and you can't restart, reset, start/stop it because you always see An unknowsn error ocurred in Operations tab.
We still waiting for them to respond, 3 hours passed, no response from them, and we losing tones of money because of this.
We even bought the gold support package, so we get answers faster ... still nothing. Too much disappointed in services, really
It appears that the instance had a lock that was held for more than 5 minutes and innodb crashed the server because of that. It's possible there was a deadlock happening.
In order to prevent this in the future we recommend looking over your current queries to see if any could cause a deadlock.
This is the answer from them, so at the time this happened we were running a migration, which was making a connection to sql, accessing 100 databases, altering few tables, closing connection, connecting again, accessing the next 100 databases, doing the changes etc etc...
We have around 13 000 databases. So this occurred when the script was working on 6600th databases.
I was disconnecting and connecting every 100 databases, so the timeout wont bother me again, but again we were deadlocked.
And the greatest issue is that google cloud shows the status of the cloud instance green, as if everything is just good. But you can't access to the instance, and you can't restart, reset, start/stop it because you always see An unknowsn error ocurred in Operations tab.
We still waiting for them to respond, 3 hours passed, no response from them, and we losing tones of money because of this.
We even bought the gold support package, so we get answers faster ... still nothing. Too much disappointed in services, really
Carlos (Cloud Platform Support)
Feb 14, 2017, 9:54:15 AM2/14/17
to Google Cloud SQL discuss
Hi Armen,
I am sure the Tech Support team will be able to help you with this issue. There is an SLA for each packages, you can always call them to get updates.
Reply all
Reply to author
Forward
0 new messages