Cloud SQL Failover Replica Unavailable

[Arcadie Condrat]

Hello,

We have a master + failover replica cluster which worked fine until the failover suddenly rendered unavailable. There is the following trace in the logs:

2017-12-27T15:40:12.603534Z 2 [ERROR] Error reading packet from server for channel '': Lost connection to MySQL server during query (server_errno=2013)

2017-12-27T15:40:12.603580Z 2 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006549' at position 29800383 for channel ''

2017-12-27T15:40:14.003316Z 1 [ERROR] Slave SQL for channel '': Could not execute Update_rows event on table carts; Can't find record in 'carts', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.006505, end_log_pos 22807607, Error_code: 1032

2017-12-27T15:40:14.003350Z 1 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.006505' position 22806296

We can't stop/start slave as this requires super privilege. Did anyone see something like this before? What are the options for trying to restore a failed replication in Cloud SQL?

Is it safe to delete the existing failover instance and create a new one?

Your help is appreciated!

Thanks,

Arcadie Condrat

[Yannick (Cloud Platform Support)]

Hello Arcadie, if you are able to delete the failed replica and create a new one, that would be a good way to resolve your immediate issue. If you are unable to do so or are still experiencing issues with your Cloud SQL instances that require access you do not have, you should create a private issue on our Issue Tracker and provide the relevant project number and instance names along with the description of your issue.