Run a Python Dataflow Job from Google Cloud Composer

aniello...@intarget.net

unread,

Jul 26, 2018, 5:50:29 AM7/26/18

to cloud-composer-discuss

Hi to everyone,

I'm a newbie on Google Cloud Composer, and i'm trying to run a Google Data Flow Pipeline (developed with python-sdk) from Google Cloud Composer.

Everything working fine if my Dataflow file is a single-file script that contain all Pipeline and Transforms.

But if i try to run a Dataflow packages like this (located in a GCS):

dataflow/

-- __init__.py

-- run.py (this is the main file)

-- setup.py

-- cloud/

----- __init__.py

----- pipeline.py

----- transform.py

----- util.py

with the following definition of DAG in my dag.py file:

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.now(),
    'dataflow_default_options': {
        'project': 'PROJECT_NAME', 
        'region': 'REGION', 
        'temp_location': 'gs://FLOW-BUCKET/flow/tmp',
        'staging_location': 'gs://FLOW-BUCKET/flow/staging',
        'setup_file': 'gs://FLOW-BUCKET/dataflow/setup.py',
        'runner':  'DataFlowRunner',
        'job_name': 'job_name_'+ str(time.time()), 
    },
}

dag = DAG('workflow', schedule_interval=None, default_args=default_args)

task = DataFlowPythonOperator(
    py_file='gs://FLOW-BUCKET/dataflow/run.py',
    task_id='data-processing',
    dag=dag)

the Dataflow fail with this error:

WARNING - File "/tmp/dataflow3275c1f6-run.py", line 18, in <module

{gcp_dataflow_hook.py:120} WARNING - from cloud.pipeline import run

WARNING - ImportError: No module named cloud.pipeline

If I try to run it on a local with airflow everything working fine, but if I try to run directly from composer the process fail.

I've tried to move my Dataflow package in the gs://COMPOSER-BUCKET/airflow/dags/ folder but the problem persist as mentioned here

Anyone has a solution for this topic? I've tried to search online, but i was unable to find more information about Dataflow + Composer integration.

Thanks in advance.

Trevor Edwards

unread,

Jul 26, 2018, 12:01:57 PM7/26/18

to cloud-composer-discuss

It sounds like you would need to import dataflow.cloud.pipline instead of cloud.pipeline, if your directory structure is

dags/

dataflow/

cloud/

pipeline.py

rjai...@datalicious.com

unread,

Sep 3, 2018, 1:21:36 PM9/3/18

to cloud-composer-discuss

@ aniello have you found the solution to the above issue ? i'm facing the same issue.

Bikram Sisodia

unread,

Sep 3, 2018, 3:16:14 PM9/3/18

to rjai...@datalicious.com, cloud-composer-discuss

Try putting your cloud folder in GCS dag bucket and just run.py in Dag folder.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1e8db907-4532-462b-9e10-f1e645dbd808%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bikram Sisodia

Analytics Director

130 Sutter St • 7th Floor • San Francisco • CA • 94104

-------------------------------------------------------

essenceglobal.com
Facebook • Twitter • YouTube • Instagram

Ramanand Jaiswal

unread,

Sep 4, 2018, 10:03:00 AM9/4/18

to Bikram Sisodia, cloud-composer-discuss

Tried but still the same issue.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1e8db907-4532-462b-9e10-f1e645dbd808%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bikram Sisodia
Analytics Director

130 Sutter St • 7th Floor • San Francisco • CA • 94104

-------------------------------------------------------
essenceglobal.com
Facebook • Twitter • YouTube • Instagram

--

Ramanand Jaiswal | Senior Data Engineer | Datalicious
e: rjai...@datalicious.com | m: +91-888411391
a: L77, 15th Cross Rd, Sector 6, HSR Layout, Bengaluru, Karnataka 560102
w: www.datalicious.com

Contact sup...@datalicious.com anytime, we're keen to help!

Crystal Qian

unread,

Sep 10, 2018, 5:39:45 PM9/10/18

to rjai...@datalicious.com, bikram....@essenceglobal.com, cloud-compo...@googlegroups.com

I'd try configuring your Dataflow directory as a local Python dependency:

https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-local

In this case, your run.py would be the use_local_deps.py, and everything else would be in a dependencies/ directory. So, you'd be moving all of this to the dags/ folder of the GCS bucket corresponding your environment.

Hope this helps!

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1e8db907-4532-462b-9e10-f1e645dbd808%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bikram Sisodia
Analytics Director

130 Sutter St • 7th Floor • San Francisco • CA • 94104

-------------------------------------------------------
essenceglobal.com
Facebook • Twitter • YouTube • Instagram

--

Ramanand Jaiswal | Senior Data Engineer | Datalicious
e: rjai...@datalicious.com | m: +91-888411391
a: L77, 15th Cross Rd, Sector 6, HSR Layout, Bengaluru, Karnataka 560102
w: www.datalicious.com

Contact sup...@datalicious.com anytime, we're keen to help!

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CABFdCr-9kB3jxruTuhAzxoda9MVEs6f16P4mLg1t6A4P-aP-Ow%40mail.gmail.com.

Incredible Budget

unread,

Dec 22, 2024, 2:56:07 AM12/22/24

to cloud-composer-discuss

Incredible Budget

unread,

Dec 22, 2024, 2:57:24 AM12/22/24

to cloud-composer-discuss

Reply all

Reply to author

Forward