Tablespace free space info is corrupted

[Lucas Taniguti]

Hello,

After a mistake (too much requests) our Cloud SQL got corrupted and keeps restarting.

After some research, this could solve the problem:

[mysqld]

innodb_force_recovery = 1

But we do not have access to system configuration.

So... We accepted that we have lost 1 day and start the recovering work.

Obs: We tried to clone the instance, but the clone never started.

Two questions:

- Is it possible to Google Support recover the database?

- We can't delete the clone instance because of its status (PENDING_CREATE). How to proceed?

Part of the log message:

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

12:03:13 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Thank you,

[Karthick (Cloud Platform Support)]

Hello Lucas, 

Enabling the backup provides the best way to restore SQL instance and recover the lost data.  However, to delete your unsuccessful creation of SQL clones, I would recommend to raise a concern in google public issue tracker platform by marking the issue private. Also please provide the following information along with it, so the engineering team can assist: 

1. Project Number.

2. Name of the affected instances. 

3. Output of "gcloud sql instances delete INSTANCEID --verbosity-debug"

4. gcloud sql instances describe INSTANCEID

[Vitor De Mario]

Hi. I'm on Lucas' team.

Our backups have always been enabled, every 24h. However, when the problem started happening the backup failed as well.

This is why Lucas said "We accepted that we have lost 1 day and start the recovering work". We started working from the backup of a previous day and did all we could to recover data through other means.

We are now running on a new instance and we have left the problematic one behind. We would still like to access the old one, there might be data on it that we haven't been able to recover, it would be useful for us. The logs are still showing lots of errors constantly and we can't fix those errors directly as Lucas has explained.

Regarding the clone, we have opened a case and it has been answered. The clone is no longer an issue, thanks.

Regards,

Vitor De Mario

[Karthick (Cloud Platform Support)]

Hello, 

As this requires manual intervention, I will get the Cloud SQL team involved and take a deeper look. Could you please confirm your Project Number and SQL instance ID along with the snippets of error log.  However, as setting flag innodb_force_recovery can be very unsafe, please understand that I cannot guarantee that they'll do it. 

[Vitor De Mario]

Hi,

We are open to other ways of recovering the database, not necessarily innodb_force_recovery. Any help is appreciated.

Project number: 77************
SQL instance ID: w***********

I'm sending part of the error log attached.

Regards,
Vitor

On Tue, Jan 9, 2018 at 5:17 PM 'Karthick (Cloud Platform Support)' via Google Cloud SQL discuss <google-cloud-sql-discuss@googlegroups.com> wrote:

Hello, 

As this requires manual intervention, I will get the Cloud SQL team involved and take a deeper look. Could you please confirm your Project Number and SQL instance ID along with the snippets of error log.  However, as setting flag innodb_force_recovery can be very unsafe, please understand that I cannot guarantee that they'll do it. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-sql-discuss/ta-9vEpWTeU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-sql-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/0c1cde80-d3b4-4ec0-9199-bfc1a7629d24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Karthick (Cloud Platform Support)]

Thank you for the information. I will get back with you once I have any update.  

[Karthick (Cloud Platform Support)]

Hi Vitor, 

Thank you for your patience. The issue is now fixed. Please confirm on the stability of the instance and let me know. 

[Vitor De Mario]

Hi Karthick,

I had some problems connecting to the instance after the issue got fixed but I succeeded in taking full backups out of it, which was impossible before.

That's enough for us, thank you very much.

Regards,

Vitor De Mario

On Fri, Jan 19, 2018 at 8:05 PM 'Karthick (Cloud Platform Support)' via Google Cloud SQL discuss <google-cloud...@googlegroups.com> wrote:

Hi Vitor, 

Thank you for your patience. The issue is now fixed. Please confirm on the stability of the instance and let me know. 

--

You received this message because you are subscribed to a topic in the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-sql-discuss/ta-9vEpWTeU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-cloud-sql-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/e233037b-e8f1-4225-aa32-d77198f1e646%40googlegroups.com.