Got an error reading communication packets and Net packets out of order error

[Stenn Kool]

I'm seeing the "Got an error reading communication packets" error in the error log and the "Net packets out of order" message in my applications, those applications are written in different languages and running on different connections.

Some of them are running om GKE and others on premise, it seems to happen when A lot of data is being inserted.

Our SQL servers hosted by a different hosting provider have no such issues while under the same load with similar specs.

We've already tried using another connection and increasing the max_allowed_packet flag.

The server is running with a failover and is using the binlog.

example errors from the log:

E  2017-08-18T12:09:41.883032Z 344745 [Note] Aborted connection 344745 to db: '***' user: '***' host: '***.***.***.***' (Got an error reading communication packets) 
E  2017-08-18T12:11:12.061786Z 362326 [Note] Aborted connection 362326 to db: '***' user: '***' host: '***.***.***.***' (Got timeout reading communication packets) 



E  2017-08-18T13:22:07.443999Z 10400 [Note] Aborted connection 10400 to db: 'cms' user: '***' host: 'cloudsqlproxy~***' (Got an error reading communication packets) 
E  2017-08-18T13:22:07.447055Z 10459 [Note] Aborted connection 10459 to db: 'cms' user: '***' host: 'cloudsqlproxy~***' (Got an error reading communication packets) 
E  2017-08-18T13:22:11.617081Z 10509 [Note] Aborted connection 10509 to db: 'cms' user: '***' host: 'cloudsqlproxy~***' (Got an error reading communication packets) 
E  2017-08-18T13:22:12.841405Z 9892 [Note] Aborted connection 9892 to db: 'cms' user: '***' host: 'cloudsqlproxy~***' (Got timeout reading communication packets) 

[Shivam(Google Cloud Support)]

Are you using CloudSQL second generation instance? The intermittent error ‘Got an error reading communication packets’ is a known issue and the suggested workaround is to retry the connection with exponential back-off.

‘Net packets out of order’ seems like a MySQL error and without more context, it is not easy to find the root cause. 

[Stenn Kool]

Yes we are using a second generation instance, throttling the inserts is hardly a sollution because it would be very hard to keep our data up to date.

Do you know if this bug is going to be resolved in the near future?

Op vrijdag 18 augustus 2017 22:28:51 UTC+2 schreef Shivam(Google Cloud Support):

[Shivam(Google Cloud Support)]

The CloudSQL team is aware of it, though there is no eta. I suggest to star the issue to get notified for any future updates.

[Pontus Lundin]

Hi,

I have the same problem when doing lots of updates in a for loop, i write on it here as well:
https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/126

But is there no quick fix to at least make it work even if throttling ? How does people get around this issue which is i guess a very common scenario to do lots of inserts/updates ?

It has never failed to do LOAD DATA INFILE it seems to happen when i do a lot of update queries.

Thanks.

[Thomas Weyn]

Same issue here, no apparent reason except for having many concurrent queries.

Op vrijdag 18 augustus 2017 15:24:56 UTC+2 schreef Stenn Kool:

[davidc...@google.com]

Hello Thomas,

Please note that aborted connection error is caused mainly by the following events:

- A client attempts to access a database but has no privileges for it.

- A client uses an incorrect password.

- A connection packet does not contain the right information.

- It takes more than connect_timeout seconds to obtain a connect packet.”

- The client slept for longer than the defined wait_timeout or interactive_timeout seconds

I suggest you review this document that contains a list of the most frequent issues you might run into when working with Google Cloud SQL instances as well as steps you can take to address them. I would also recommend to read this third party document to identify the possible reasons for MySQL “Got an error reading communication packet” errors, and how to address them.

If you still are seeing this error after reviewing the above information, and you believe the issue is on our end, please open a private Issue Tracker and provide the following information:

- Project ID and Cloud SQL instance ID

- Timestamp of the last occurence

- Have you performed any change to the code accessing the Mysql since the "timeouts" or the "error reading" was not happening?

- Do you close the connections properly during the normal behaviour?

- Do you keep many connections idle and open for a long time?

- Screenshot of the error encountered on your end

Please make sure to open it using the provided link as it will be a private Issue Tracker where you will be able to share your project ID.

[Alejo gomez]

Same issue here with App engine (python3 - standard environment) and cloud sql -2nd Gen Mysql instance

 

I think that it's not a problem with sql cloud, or with the configuration of my application, but that it's an app engine problem. I came to this conclusion in the following way:

 

if I connect to the sql cloud instance through Mysql workbench, no connection is aborted

Similarly if I run my application on a local server, but connecting to cloud sql (through the cloud_sql_proxy), no error is generated and everything works perfect.

[Nicolas (Google Cloud Platform Support)]

Hi Alejo,

Thank you for reporting this issue.

Please note that Issue tracker would be the right channel to address these kind of technical issue as Google Groups hosts discussion forums where you're likely to find information like service status updates and release notes and high-level discussions on the platform.

Currently the Cloud SQL engineering team is aware of this issue and it's currently being worked on.

Further communications will be made in this thread, however we cannot promise an ETA for the resolution of this issue.

Also of my colleague answered here [2] with detailed resolution steps,

I hope it helps you!

[Melki Saputro]

I'm having the same issue. When Google will fix this or give us hint how to fix it. If not in 2 months i will change GCP with other Cloud Services.

[Amit Sinha]

Hello Melki,

Thank you for your reply.

As mentioned by Nicolas, this post has detailed description regarding the error. I would recommend to go through the link and see if it answers your concern.

[Albert Khasky]

Hi Amit,

I apparently don't have permission to view the contents of that post. Could you perhaps share it here? We have addressed all of the know possible causes for the error and yet it persists. Thanks!

[Olu]

Hi, Albert. 

Actually, the Link could no longer be accessible publicly as project-specific private information were shared on the link. However, the information shared on the link by the CloudSQL Specialist are details about how and why the Aborted Connection errors may be reported. Please view the comment below: 

=====

"Aborted connection nnnn to db:" error message is triggered when an existing connection is not terminated properly, which may look erroneous but is actually perfectly normal. Aborted connections happen because of unclean closure of connection or networking problem between the server and the client, but not because of the server [1].

You should note that this error does not mean that there are problems with your Cloud SQL instance and as far as I know these types of errors in the MySQL logs are harmless but seem to be useful to indicate client side behaviors because of the following:

1) It won't trigger "Aborted connection" error when mysql client closes cleanly.

2) If the proxy client is killed but didn't close mysql client --> The socket connection is still alive.

    a) manually close the socket connection at the client side

    b) OR wait until times out based "wait_timeout" in MySQL server and the connection closed --> "Aborted connection" error.

The official MySQL documentation suggests various reasons why this could happen and some actions to take [2]. And this blog goes into more details on fixes [3].

In conclusion, if you want to reduce such errors in the logs, I'd suggest to:

- cleanly close each connection and this should be fine when your app hits production.

- implement incremental backoff in your code to handle connections when the DB is temporarily unavailable.

Using exponential backoff prevents your application from sending an unhealthy number of connection requests when it can't connect to the database.

I hope this helps. 

[1] https://cloud.google.com/sql/docs/mysql/diagnose-issues#connection

[2] https://dev.mysql.com/doc/refman/8.0/en/communication-errors.html

[3] https://www.percona.com/blog/2016/05/16/mysql-got-an-error-reading-communication-packet-errors/