InnoDB: Assertion failure, Cannot allocate 18446744073709546192 bytes of memory

[Jacob Hagstedt P Suorra]

Hi!

On Monday (170807) I noticed that the connection from our services to our Cloud SQL instance (2nd generation) started to get shakey. Today (Tuesday 170808) it totally broke down on us.

If I check under the "Operations" tab, I can see that an backup failed with "An unknown error occurred":

and now the entire instance is locked (pretty much everything greyed out) and it says it is "Under maintenance":

Checking the error logs for the instance, I found this:

ERROR] InnoDB: Cannot allocate 18446744073709546192 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.

(For the complete log, please see the attached file mysql_err_log_1.txt)

This made me find this link: https://bugs.mysql.com/bug.php?id=70185 where the user seems to have nearly exactly the same problem as we're experiencing. He also suggests two solutions:

1) add 'innodb_force_recovery=4' to [mysqld] in my.cnf

or....

2) set 'innodb_change_buffering=none' in my.cnf 

But I am however not able to test them since gcloud won't allow me to neither access the cnf file, nor set the flags using gcloud:

gcloud sql instances patch [INSTANCE-NAME] --database-flags innodb_force_recovery=4

    The following message will be used for the patch API method.

    {"project": [PROJECT-NAME], "name": [INSTANCE-NAME], "settings": {"databaseFlags": [{"name": "innodb_force_recovery", "value": "4"}]}}

    WARNING: This patch modifies a value that requires your instance to be

    restarted. Submitting this patch will immediately restart your

    instance if it's running.

    Do you want to continue (Y/n)?  Y

ERROR: (gcloud.sql.instances.patch) HTTPError 404: Flag requested cannot be set.

I've also tried to create two new instances: One exactly the same as the original one, and one with double the resources (vCPU, RAM and SSD) just in case we'd hit the resource ceiling, and then restore these new instances to a previous version of the now failed instance using the backups. I've tried two different backup dates: The first one one day before this problem was noticed (170806) and one a couple of days before (170804), but both of the new instances start immediately to experience the same issues (Please see the logs from one of the new instances in attached file mysql_err_log_2.txt)

I've also tried to do an export using one of the new instances, via the Cloud Web UI, but that results in this error: mysqldump: Couldn't execute 'show table status like 'email\_meta'': Lost connection to MySQL server during query (2013).

All our services that tires to access any of the new instances get one of the following errors:

(2013, 'Lost connection to MySQL server during query')

(2006, 'MySQL server has gone away')

I'm also not able to do an export using a mysql client application. I've tried with Sequel Pro (for mac) without luck, I just get an error saying the connection failed.

What does this log mean? What can I do to check my tables in case any of them are corrupted etc? 

Any kind of help and suggestion is highly appreciated! 

[Kenworth (Google Cloud Platform)]

This is a duplicate of this thread.

[Jacob Hagstedt P Suorra]

Yes, I'm sorry, I didn't realise the posts wasn't visible directly. Should I remove this post in some way?