Reading data in another storage bucket in Composer

[Jocelyn]

Hi,

There is a daily data transfer that comes in to Cloud Storage from an external party. These files are zipped, so i'm trying to automate and unzip them into another folder in the same bucket.

I am getting this error  Broken DAG: [/home/airflow/gcs/dags/file_unzip.py] name 'gcs' is not defined

Which I presume is because the zipped files are not located in the data folder of composer but in another cloud storage bucket. Is there a way to read the files in the external bucket in airflow or does Composer only read data from it's own bucket?

Jocelyn

[Feng Lu]

Hi Jocelyn,

DAGs in Cloud Composer can read/write data to any GCS bucket as long as:

- if you don't specify a connection id, please make sure the service account of your Composer environment is configured with read/write access to the desired bucket.

- if you do specify a GCP connection, similarly you'll need to ensure the service account in the GCP connection has access to the bucket. 

Looking at the error message you shared, it appears that your DAG is broken (i.e., python code problem) and ran into undefined variable 'gcs'.  

Feng 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/97b08fd6-ae9a-46b6-9b14-57a4f3aec977%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.