How can a private GKE cluster communicate with a Cloud SQL instance within the project.

[Shubhanan Bakre]

Hi,

I have tried the steps for connecting from a GKE cluster to CloudSQL that are given here: https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine

However, the same steps do not work for a private GKE cluster with master authorized networks (https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters)

Can someone shed some light on how can this be achieved?

Thanks!

Shubhanan

[Kamran (Google Cloud Support)]

Hello Shubhanan,

As the Cloud Proxy requires to establish connections to external IP address of the Cloud SQL instance, if you wish to run Cloud Proxy on your GKE node(s), the nodes should be able to talk with the Cloud SQL instance's external IP address. Due to this requirement, you can setup NAT gateway(s) to achieve your design. 

For information on how to configure NAT gateway(s) for your GKE cluster please visit the following articles:

1.  https://cloud.google.com/solutions/connecting-securely#natgateway

2. https://cloud.google.com/vpc/docs/special-configurations#natgateway

3. https://cloud.google.com/vpc/docs/special-configurations#multiple-natgateways

4. https://cloud.google.com/solutions/using-a-nat-gateway-with-kubernetes-engine

[Shubhanan Bakre]

Hi Kamran,

Thanks! I got this working. Is cloud proxy the only way? Would this work via authorized network setup?

- Shubhanan

[Dinesh (Google Platform Support)]

You can configure external IP address of NAT gateway to the Cloud SQL authorized network list that essentially authorize gateway to communicate with Cloud SQL.  Authorized network setup requires external IP address of MySQL client to be configured in SQL instance Authorization tab. You can find more information about authorized network setup in this article[1].

[1]: https://cloud.google.com/sql/docs/mysql/connect-admin-ip

[Shubhanan Bakre]

Dinesh,

Thanks for your response. I tried this and the connectivity from the NAT gateway to the MySQL instance works fine. However, the private GKE container is unable to connect to the mysql instance. To debug the pod config, I tried to see if a public cluster is able to communicate with the CloudSQL instance and it does not work either. So, my understanding is that the pod config is incorrect/incomplete. I am using this sample app https://github.com/GoogleCloudPlatform/kubernetes-engine-samples/blob/master/cloudsql/mysql_wordpress_deployment.yaml and removed the sidecar cloud sql proxy.

Following is my pod config:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
        - name: web
          image: wordpress
          ports:
            - containerPort: 80
          env:
            - name: WORDPRESS_DB_HOST
              value: <ip_of_cloudsql_instance>:3306
            # These secrets are required to start the pod.
            # [START cloudsql_secrets]
            - name: WORDPRESS_DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
            # [END cloudsql_secrets]
      # [START volumes]
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
# [END volumes]

[Larbi (Google Cloud Support)]

Hello Shubhanan,

I see that you removed the cloud proxy container from the deployment, the Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances . sure you do not need to use the proxy or configure SSL to connect to Cloud SQL from App Engine standard or flexible environment.

As you mention the connectivity from the NAT gateway to your Cloud SQL is working fine, What about the from your private or public GKE to this NAT gateway is it working fine ? 

[Shubhanan Bakre]

That is the part that doesn't seem to work.

Looking at the pod's logs, I see the following error:

Warning: mysqli::__construct(): (HY000/2002): Connection timed out in Standard input code on line 22

MySQL Connection Error: (2002) Connection timed out

[Larbi (Google Cloud Support)]

Hello Shubhanan,

For public cluster i did use this documentation [1] to use the NAT Gateway with Kubernetes Engine and i was able to Verify the NAT gateway routing with the kubctl command:

 

kubectl run example -i -t --rm --restart=Never --image centos:7 -- curl -s http://ipinfo.io/ip

 

I displayed the external IP address of the NAT gateway.

For private cluster you have to enable the Alias IPs [2] as it allow Pods to directly access hosted services without using a NAT gateway.

Cluster IPs for internal Services remain only available from within the cluster. If you want to access a Kubernetes Service from within the VPC, but from outside of the cluster (for example, from a Compute Engine instance), use an internal load balancer [3].

Did you enable Alias IPs?

[1]https://cloud.google.com/solutions/using-a-nat-gateway-with-kubernetes-engine

[2]https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips

[3]https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing

[Heather Young]

I'm facing a similar issue. According to this blog post from GCP it seems like I should be able to connect to the Private IP of a Cloud SQL instance simply by creating a service and endpoint  however I am not able to connect to the service via the Kubernetes Service Name <svc-name>.<namespace>.svc.cluster.local. With `dig` in a container on my cluster that DNS resolves to the correct private IP but I cannot connect. I have one deployment in my cluster which uses `hostNetwork: true` and I can connect with no problem from inside that container, but I need to be able to connect from a container that does not use the host network.

I understand that enabling Alias IPs may be what is needed, but do not currently have a cluster with that enabled and I would rather not recreate the whole cluster if I can help it. Does the blog post assume that Alias IPs is enabled already or am I missing something? So far I have not tried using a NAT gateway or proxy. Thanks

[Zeb Barnett]

I just ran into the same problem trying to use the private IP to connect to a Cloud SQL instance from a Kubernetes cluster and found out the hard way that the cluster has to be set up to be VPC-native. https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips

Once I created a cluster that was VPC-native, I was able to connect to the Cloud SQL instance from the pods without a problem.

Hope this helps!

[Sabyasachi Sengupta]

I have the same issue on VPC Native. My Clusters are private and I have tried connecting to the CloudSQL using the cloud-proxy side car. I see that the application is able to connect to the cloud-proxy. the cloud-proxy fails to connect to the Cloud SQL. I am using the --private-ip config of the cloud-proxy. I am not sure if I have to remove that or do something to get this working. I wish there were some documentation. Any help is deeply appreciated.