Getting Google's Cloud SQL Proxy to work

[Traiano Welcome]

I'm following this example of how to get wordpress running on GKE connected to Google Cloud SQL via the Google Cloud SQL Proxy:

https://cloud.google.com/sql/docs/mysql/connect-container-engine

Unfortunately, my wordpress pod is failing with a crashloop error, and it's not clear from the documentation how to dig deeper into the reason for this. Here is a sample of the error:

bash-3.2$ kubectl get pods| egrep wordpress
wordpress-713960421-v4f49     0/2       CrashLoopBackOff   16         20m

(kubectl describe pod ...)

11m   22s     36      kubelet, gke-noon-staging-default-  pool-d500b601-dfb6                                    Warning FailedSync         Error syncing pod, skipping: [failed to "StartContainer" for "web" with   CrashLoopBackOff: "Back-off 5m0s restarting failed container=web   pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7-a565-42010a9a0023)"
 , failed to "StartContainer" for "cloudsql-proxy" with    CrashLoopBackOff: "Back-off 5m0s restarting failed container=cloudsql-proxy    pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7- a565-42010a9a0023)"  
 ]

My questions are:

• How to dig further into why the pod is failing to deploy with cloud sql proxy
• Has anyone got an example working for using cloud sql proxy with as simple mysql client pod?

Here is a description of the deployment (kubectl describe wordpress):

 https://pastebin.com/DjYM97R7

And a description of the pod (kubectl describe ):

 https://pastebin.com/pN7gUZg8

	I deployed the cloud sql proxy and wordpress container separately and found the cloud sql proxy runs fine, but the wordpress blog container fails to startup on kubernetes. Error syncing pod, skipping: failed to "StartContainer" for "wordpress" with CrashLoopBackOff
	Looking at the pod logs for wordpress, it seems wordpress is dying because it cannot connect to the MySQL DB: MySQL Connection Error: (2002) Connection refused – Traiano Welcome 3 hours ago

[Ahmet Alp Balkan]

Can you run "kubectl logs -l app=wordpress"? I am assuming there will be some logs from the crashing mysql container.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Traiano Welcome]

Hi Ahmet

On Thursday, 22 June 2017 02:25:05 UTC+4, Ahmet Alp Balkan wrote:

Can you run "kubectl logs -l app=wordpress"? I am assuming there will be some logs from the crashing mysql container.

Thanks for your response. I get no output from running that command (I suppose no logs are being generated). I tried both the command, and running it in a loop, just in case:

 for i in `seq 1 100`;do echo "checking $i " - $(kubectl logs -l app=web );done

No result.

Just to be clear, wordpress container is in a pod, and the cloud sql container is a sidecar container connected to pod.

Google's documentation seems pretty thin on getting this working properly with GKE containers, and non-existent on troubleshooting connectivity issues with cloud sql proxy.

Do you know of an alternative method of getting a container access to Cloud SQL from GKE ?

On Wed, Jun 21, 2017 at 7:58 AM, Traiano Welcome <tra...@gmail.com> wrote:

I'm following this example of how to get wordpress running on GKE connected to Google Cloud SQL via the Google Cloud SQL Proxy:

https://cloud.google.com/sql/docs/mysql/connect-container-engine

Unfortunately, my wordpress pod is failing with a crashloop error, and it's not clear from the documentation how to dig deeper into the reason for this. Here is a sample of the error:

bash-3.2$ kubectl get pods| egrep wordpress
wordpress-713960421-v4f49     0/2       CrashLoopBackOff   16         20m

(kubectl describe pod ...)

11m   22s     36      kubelet, gke-noon-staging-default-  pool-d500b601-dfb6                                    Warning FailedSync         Error syncing pod, skipping: [failed to "StartContainer" for "web" with   CrashLoopBackOff: "Back-off 5m0s restarting failed container=web   pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7-a565-42010a9a0023)"
 , failed to "StartContainer" for "cloudsql-proxy" with    CrashLoopBackOff: "Back-off 5m0s restarting failed container=cloudsql-proxy    pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7- a565-42010a9a0023)"  
 ]

My questions are:

• How to dig further into why the pod is failing to deploy with cloud sql proxy
• Has anyone got an example working for using cloud sql proxy with as simple mysql client pod?

Here is a description of the deployment (kubectl describe wordpress):

 https://pastebin.com/DjYM97R7

And a description of the pod (kubectl describe ):

 https://pastebin.com/pN7gUZg8

	I deployed the cloud sql proxy and wordpress container separately and found the cloud sql proxy runs fine, but the wordpress blog container fails to startup on kubernetes. Error syncing pod, skipping: failed to "StartContainer" for "wordpress" with CrashLoopBackOff
	Looking at the pod logs for wordpress, it seems wordpress is dying because it cannot connect to the MySQL DB: MySQL Connection Error: (2002) Connection refused – Traiano Welcome 3 hours ago

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Evan Jones]

I know nothing about wordpress, but for what it is worth, we are using this Cloud SQL Proxy container with success. A few notes about the config you posted:

* I'm assuming that where you have "-instances=[INSTANCE_CONNECTION_NAME]=tcp:[PORT]" you've replaced this with your Cloud SQL instance name (project:region:cloud_sql_name), and port with 3306, right?

* If you run kubectl logs pod cloudsql-proxy you should see something like the following:

2017/06/21 20:45:06 Listening on (whatever) for (your instance name)

2017/06/21 20:45:06 Ready for new connections

* If the cloudsql-proxy container is running, you can run kubectl exec (pod) -c cloudsql-proxy -ti /bin/sh to get a shell and try to verify the cloudsql-proxy configuration, or to start a new instance and make sure it is configured correctly.

* We are using this with local Unix socket connections in the /cloudsql directory, but localhost sockets should also work.

I'm surprised you don't see any logs from your wordpress container, but I can't help with that part.

Good luck!

[Traiano Welcome]

Hi Evan

On Thu, Jun 22, 2017 at 5:34 PM, Evan Jones <evan....@triggermail.io> wrote:

I know nothing about wordpress, but for what it is worth, we are using this Cloud SQL Proxy container with success. A few notes about the config you posted:

* I'm assuming that where you have "-instances=[INSTANCE_CONNECTION_NAME]=tcp:[PORT]" you've replaced this with your Cloud SQL instance name (project:region:cloud_sql_name), and port with 3306, right?

I have this:

          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=lol-staging:europe-west2:lol-staging-001=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]

I've redeployed and can see logs now, the wordpress container is failing because it's failing connection to SQL via the proxy:


     MySQL Connection Error: (2006) MySQL server has gone away
     Warning: mysqli::mysqli(): MySQL server has gone away in - on line 10
     Warning: mysqli::mysqli(): Error while reading greeting packet. PID=167 in - on line 10
     Warning: mysqli::mysqli(): (HY000/2006): MySQL server has gone away in - on line 10

The cloud sql proxy container complains about an account not having correct permissions (however  I have given it editor permissions):

    kubectl logs pods/wordpress-2608214628-9bc9b cloudsql-proxy

2017/06/22 13:40:15 Throttling refreshCfg(lol-staging:europe-west2:lol-staging-001): it was only called 24.005689746s ago
2017/06/22 13:40:15 couldn't connect to "lol-staging:europe-west2:lol-staging-001": ensure that the account has access to "lol-staging:europe-west2:lol-staging-001" (and make sure there's no typo in that name). Error during createEphemeral for lol-staging:europe-west2:lol-staging-001: googleapi: Error 403: The client is not authorized to make this request., notAuthorized
2017/06/22 13:40:18 New connection for "lol-staging:europe-west2:lol-staging-001"

So I'm now wondering how to better validate which account is being referred to here and whether the permissions are correct.

 

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/2kqHNT1Nbtc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Evan Jones]

The Cloud SQL Proxy logs suggest to me that it may not be using the right credentials? It is possible that it is trying to use the cluster's "default service account"?

If you use "kubectl exec ... -ti /bin/sh" you should be able to examine the contents of the credentials file that is being passed to ensure that the contents match what you expect. You can also try to interactively start /cloud_sql_proxy with different flags to see if you can get it to start and print the "expected" log message.

Good luck I hope that helps!

Hi Evan

To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.

[Traiano Welcome]

On Thursday, 22 June 2017 18:16:22 UTC+4, Evan Jones wrote:

The Cloud SQL Proxy logs suggest to me that it may not be using the right credentials? It is possible that it is trying to use the cluster's "default service account"?

It seems to have been a permissions issue. It started working when I gave the cloud-sql service account "Project Owner" rights. The problem with this is that it's too permissive though.

Another question though: Do I really have to use cloudsql proxy to access it from GKE? It seems that whitelisting the IP addresses of the GKE cluster works just as well - In which case I wonder why the documentation recommends the overcomplicated route of setting up cloud sql proxy.

 

[Evan Jones]

I believe we only grant the service account the "Cloud SQL Client" role.

I'm just a user of CloudSQL, but my understanding of the advantage of using the proxy is that it lets you control access via Google Cloud roles. Without it, you are trusting everything that uses some set of IP addresses with permission to connect to MySQL. If you have tools around managing Google Cloud permissions already, its nice to control access to MySQL using those some methods. The disadvantage, as you noted, is that you have to configure this weird Cloud SQL thing.

To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.