Cloud SQL Hook

[thibault clement]

Hi everybody,

I'm currently evaluating Cloud Composer and I got some issue to connect to my Google Cloud Postgres Database. I'm currently using the PostgresOperator and I see two options to be able to access to my DB from Cloud Composer:

1. Whitelisting the Composer worker nodes external IPs on my Google Cloud conf. Yes, yes I know it's just a bad temporary solution... but allow me to easily test my first DAGs.

 2. The Cloud SQL proxy solution following this doc: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine. Between us... it's a pain in the ass to set up for a guys like me that don't really know Kubernetes. Also I'm currently evaluating using Cloud Composer especially to avoid all this configuration/operation cost and be able to focus on the development of operators and DAGs. 

So I was wondering if there is any plan to provide a google cloud SQL hook ? 

I think it's clearly a missing part today. It's really nice and easy to use all the GCP hooks or operators to move, transform and load my data across the GCP services except Google SQL.

It's easier for me today to access my Google Cloud SQL database from my Airflow installed on a server outside GCP (I just whitelist the IP on Cloud SQL) than to access it from Cloud Composer.

Anyway thanks guy and keep going, 

What you do is going in the good direction and help us a lot,

Thibault

[Wilson Lian]

Hi Thibault,

Thanks for your feedback, I've noted it in our internal feature request tracker. I'm including some instructions to help you deploy a Cloud SQL proxy service in the Kubernetes Engine cluster and make use of it in Airflow.

1. Run additional Cloud SQL proxy containers, one for each SQL DB (limitation of the SQL proxy container) 

2. Expose these SQL proxies as services, for example, 

​

3. You may also want to inject a different service account to each SQL proxy in case your CloudSQL instance lives in a different project. 

4. Finally, add the CloudSQL DB connection in Airflow and use it in your DAGs. Example below:

best,

Wilson

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5acbb057-d9df-4c8c-b123-b25ed57d24d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[thibault clement]

Hi Wislon,

Thanks for added this in your internal feature request tracker (let me know if you have any planning) and thanks for the useful additional instruction.

Thibault

On Thu, 28 Jun 2018 at 09:20, Wilson Lian <wwl...@google.com> wrote:

Hi Thibault,

Thanks for your feedback, I've noted it in our internal feature request tracker. I'm including some instructions to help you deploy a Cloud SQL proxy service in the Kubernetes Engine cluster and make use of it in Airflow.

1. Run additional Cloud SQL proxy containers, one for each SQL DB (limitation of the SQL proxy container) 

2. Expose these SQL proxies as services, for example, 

​

3. You may also want to inject a different service account to each SQL proxy in case your CloudSQL instance lives in a different project. 

4. Finally, add the CloudSQL DB connection in Airflow and use it in your DAGs. Example below:

best,

Wilson

On Wed, Jun 27, 2018 at 12:34 AM, thibault clement <thibau...@gmail.com> wrote:

Hi everybody,

I'm currently evaluating Cloud Composer and I got some issue to connect to my Google Cloud Postgres Database. I'm currently using the PostgresOperator and I see two options to be able to access to my DB from Cloud Composer:

1. Whitelisting the Composer worker nodes external IPs on my Google Cloud conf. Yes, yes I know it's just a bad temporary solution... but allow me to easily test my first DAGs.

 2. The Cloud SQL proxy solution following this doc: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine. Between us... it's a pain in the ass to set up for a guys like me that don't really know Kubernetes. Also I'm currently evaluating using Cloud Composer especially to avoid all this configuration/operation cost and be able to focus on the development of operators and DAGs. 

So I was wondering if there is any plan to provide a google cloud SQL hook ? 

I think it's clearly a missing part today. It's really nice and easy to use all the GCP hooks or operators to move, transform and load my data across the GCP services except Google SQL.

It's easier for me today to access my Google Cloud SQL database from my Airflow installed on a server outside GCP (I just whitelist the IP on Cloud SQL) than to access it from Cloud Composer.

Anyway thanks guy and keep going, 

What you do is going in the good direction and help us a lot,

Thibault

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5acbb057-d9df-4c8c-b123-b25ed57d24d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

[al...@ifmrestoration.com]

Wislon,

  Do you have more of a step by step example?  This is exactly what we need but my skills are not as far along.  We are using App Maker to create an entry app.  The data is stored in Cloud SQL.  I need to get the results into Big Query.  We have composer up and running and are familiar with moving data around different Big Query instances.  

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5acbb057-d9df-4c8c-b123-b25ed57d24d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wilson Lian]

+Maria, our support oncall for this week, explicitly for visibility

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5acbb057-d9df-4c8c-b123-b25ed57d24d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/4d228653-23d2-4098-a0b7-20ec4cd6fd64%40googlegroups.com.

[carlos...@farfetch.com]

Hi Wilson,

I did what you mentioned, i follow this instructions https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/Kubernetes.md . 

I can confirm that the workers have access to my cloud sql but i am not able to access to the database in the Ui. Maybe some firewall rule is missing?

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5acbb057-d9df-4c8c-b123-b25ed57d24d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.