Using Composer to Move a file from one Google Cloud Storage location to another

[al...@ifmrestoration.com]

Hello,

  I am creating a file drop location to process a CSV file, then archive it once imported.  I can read the CSV from a google cloud storage location, and load into Big Query.  However, I am struggling to move the CSV from the drop folder to an archive folder.  Ideally I would use GoogleCloudStorageToGoogleCloudStorageOperator but from research online,  v1.9.0 doesnt support this.

From the link below, I attempted to get the hook and operator from github, and placed in the 

plugins/ folder

https://stackoverflow.com/questions/50478765/broken-dag-no-module-named-airflow-contrib-gsc-to-gcs

When I attemp to load the DAG, getting the following error message.  No module named gcs_to_gcs

The main parts to my dag

from airflow import DAG

from airflow.operators.bash_operator import BashOperator

from datetime import datetime, timedelta

from airflow.contrib.operators.bigquery_operator import BigQueryOperator

from airflow.contrib.operators.bigquery_to_gcs import BigQueryToCloudStorageOperator

from airflow.contrib.operators.gcs_to_bq import GoogleCloudStorageToBigQueryOperator

from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook

from airflow.contrib.operators.gcs_to_gcs import GoogleCloudStorageToGoogleCloudStorageOperator

from airflow.models import BaseOperator

So three questions, is my attempt to load GoogleCloudStorageToGoogleCloudStorageOperator correct?  Is their a better way?  And Can airflow / composer be upgraded?

Thanks!

Aron 

[Bikram Sisodia]

Another option: gcsfs from Python operator https://gcsfs.readthedocs.io/en/latest/api.html 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/155a9b1b-18ec-4081-a89b-5d9db6f0ed12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bikram Sisodia

Analytics Director

130 Sutter St • 7th Floor • San Francisco • CA • 94104

-------------------------------------------------------

essenceglobal.com
Facebook • Twitter • YouTube • Instagram

[Tim Swast]

The gcs_to_gcs operator is not yet available in Cloud Composer. This blog post : https://cloud.google.com/blog/products/data-analytics/how-to-transfer-bigquery-tables-between-locations-with-cloud-composer describes how to add the operator to your plugins directory and use it in a sample pipeline.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CABUrjvAPib4PbtOmLKf-NvrTjhY6HeuVGk3mzpKp5jgFqD8EKw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

  •  Tim Swast
  •  Software Friendliness Engineer
  •  Google Cloud Developer Relations

  •  Seattle, WA, USA