Connecting to Cloud SQL from Composer

[Terrence Szymanski]

We're currently running some PostgresOperator jobs on a Cloud SQL instance. We currently can do this using the external IP address and whitelisting the Composer worker nodes external IPs, but this isn't really sustainable. I'm wondering if there's a way to deploy the cloudsql proxy in Composer, of if there's another recommended way to connect to Cloud SQL instances from Airflow?

Terry

[Trevor Edwards]

Static worker IP addresses is a feature we are considering, but this is not planned yet.

If you are comfortable editing kubernetes spec, it is possible for you to add a sql proxy service to your existing Composer kubernetes spec. Composer already has a sql proxy running to connect to the Airflow database. If you experience issues with doing this, though, it would not be something we can support.

[Terrence Szymanski]

Thanks Trevor, we've deployed a cloud sql proxy pod to the Composer GKE cluster after it's created, and this seems to work well.

Terry

[Pravallika Talabattula]

I set up cloudproxy in kubernetes cluster by following this document - https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/Kubernetes.md

but I my cloud composer dag is not able to access cloud sql data. Can you please guide or provide me with a document which i can follow to set up Cloud composer and Cloud SQL set up

[Arik Liber]

This short article I've written might help you:

https://medium.com/@ariklevliber/connecting-to-gcp-composer-tasks-to-cloud-sql-7566350c5f53

Arik

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/faae9711-b543-480a-9d0d-ad26d175bc08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sree Iyer]

Hi All,

I'm in a similar situation. I need to connect to CloudSQL Postgresql From Google Cloud Composer.  Unfortunately I am not a kubernetes expert plus very limited knowledge on GCP. Just by reading some documentation I tried out the following -

This is what I have tried so far -

1) Set up VPC Network Peering. The CloudSQL Postgresql and Google Cloud Composer are in separate projects.

    The VPC Network Peering Status shows "ACTIVE" on both the sides, so I am guessing my setup is correct.

2) Set up the Firewall  - Here is where I am confused. Anyway this is what I did

     on the CloudSQL Postgresql - I set the Direction = Ingress
      Protocols and ports = all

      Targets = All Instances in the Network (I am guessing Target is the CloudSQL Postgresql)

      Source = Is the google composer  (Service account)

3) Testing - On the google cloud composer (Gcloud Active Shell)

     I wanted to see whether I can connect to the CloudSQL Postgresql database

     psql -U master -h [ DATABASE PRIVATE IP ADDRESS]

     Got the Below Error -

      psql: error: connection to server at "10.XX.X.X", port 5432 failed: Connection timed out

        Is the server running on that host and accepting TCP/IP connections?

Don't know whether there is some issue on the Firewall setup ? Please help.

If I have to take the route of Cloud SQL Proxy. Can someone please give me the detailed steps on how to do all the complete setup.

any help is highly appreciated !!!