CloudSQL Proxy error: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp: i/o timeout

[Sean Dowd]

We have just migrated from a trial GCP account to a "real" one and are now unable to connect to our pre-existing MySQL CLoudSQL instance from GKE.

We have a pod (in a deployment) that has a cloudsql proxy container and a wordpress one (which I've replaced with a simple mysql container, running a while-true-sleep loop so we can exec in a test a command line mysql connection).  The client errors off with:

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 107

And the cloudsql proxy log shows this:

2018/08/22 15:41:57 New connection for "PROJECT:us-central1:MYSQL_INSTANCE_ID"

2018/08/22 15:42:27 couldn't connect to "PROJECT:us-central1:MYSQL_INSTANCE_ID": Post https://www.googleapis.com/sql/v1beta4/projects/PROJECT/instances/MYSQL_INSTANCE_ID/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp: i/o timeout

Note the 30 seconds elapsed time.

The cloudsql-proxy is invoked with:

        command: ["/cloud_sql_proxy"]

        args: ["-instances=$(MYSQL_INSTANCE)",

               "-credential_file=/secrets/cloudsql/XXX-mysql-proxy-access.json",

               "-verbose=true"]

Where mysql-proxy-access.json contains the JSON credentials of a service account assigned the Cloud SQL Client role and $MYSQL_INSTANCE is PROJECT:us-central1:MYSQL_INSTANCE_ID=tcp:3306

I've checked to ensure that the contents of XXX-mysql-proxy-access.json match the key on the service account.

The cluster nodes are on v1.10.6-gke.1 and we are using the latest cloudsql proxy image (gcr.io/cloudsql-docker/gce-proxy:1.11).  We tried making the mysql instance reside in the same zone as the nodes (us-central1b) but nothing changed.

[Sam (Google Cloud Support)]

Have you tried refreshing the access token? Some OAuth 2.0 flows require using refresh tokens to acquire new access tokens as they have limited lifetimes to enhance security [1]. A refresh token will allow your application access Cloud SQL beyond the access token’s lifetime [2].

Based on the error message, you can have a look at these documentations about troubleshooting Cloud SQL connection issues [3][4]. Then by means of ensuring proper configuration I would follow the guide in the fifth and sixth links [5][6]. The last link is an answer on StackOverflow that I found [7]. Hope this helps.

[1] https://cloud.google.com/storage/docs/json_api/v1/how-tos/authorizing#OAuth2Authorizing

[2] https://cloud.google.com/storage/docs/json_api/v1/how-tos/authorizing#OAuth2Authorizing

[3] https://cloud.google.com/sql/docs/mysql/diagnose-issues

[4] https://cloud.google.com/sql/faq#connections

[5] https://cloud.google.com/sql/docs/mysql/connect-admin-proxy

[6] https://cloud.google.com/sql/docs/mysql/sql-proxy

[7] https://stackoverflow.com/questions/5755819/lost-connection-to-mysql-server-at-reading-initial-communication-packet-syste

[Sean Dowd]

I ended up creating a new cluster that has sql in it's scope list (the other one did not).  Connections worked immediately (with the existing token).  If this is the answer, the documentation should probably point to this somewhere. I would have hoped that the cloudsql proxy would have a more descriptive error message.

I followed the list here (your link[5]):

• Enable the Cloud SQL API
  
• Install the proxy
  
• Create a service account
  
• Start the proxy
  
• Start the mysql session
  

but still could not connect.  Re-creating a cluster is really not a good solution (just re-creating the node pool did not work).  It also seems that the inability to update scope is a shortcoming in GCP/GKE.  Also note, link [6] following the kubernetes track, the documentation does not mention creating the cluster with the sql scope.

[Pia Chamberlain]

Thanks for the heads up, Sean. I'll make sure that feedback gets back to the Cloud SQL doc team.

Would a pointer here have helped?

https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes#service_account

[Sean Dowd]

Yes - it actually did.  Thanks.