Running Dataflow pipelines from Cloud Composer

[Norbert Kremer]

I'm just getting started with running some Dataflow pipelines from Cloud Composer. I have a working java dataflow pipeline packaged as a jar file, and I can't get it to run from CC.  Still reviewing this code with another developer, so not ready to post details yet.

In the meanwhile, a general question:

Is there a well-documented demo that shows how to run DataFlow pipelines from Cloud Composer? I have not been able to find any so far.  I'd be looking for one demo for a java pipeline and one demo for a python pipeline. I'm looking for documentation of the process end-to-end, showing how to integrate my Dataflow code, build the environment (java or python, or if it is already built into CC, then demo that it's working right), install additional python dependencies that i may have, declare Airflow variables, etc.  You know, a self-contained document that has all the steps needed to actually get something to work with my code, not just code snippets and fragments and pointers to other web sites.  So a demo like the wordcount demo for Dataproc is not quite good enough (as it just runs pre-canned code, not my code.)

What's the best way to get started here?

[Feng Lu]

+Tim Swast seems totally reasonable, let's plan to have a dedicated tutorial/samples on Dataflow*Operators, wdyt? 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/7bb26069-2cbb-42fb-b8bf-71982e5b3e56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Madhurranjan Mohaan]

Have we published these tutorials to run dataflow from Cloud Composer?   

On Thursday, August 9, 2018 at 12:01:54 AM UTC-7, Feng Lu wrote:

+Tim Swast seems totally reasonable, let's plan to have a dedicated tutorial/samples on Dataflow*Operators, wdyt? 

On Tue, Jul 31, 2018 at 10:20 AM Norbert Kremer <norbert....@gmail.com> wrote:

I'm just getting started with running some Dataflow pipelines from Cloud Composer. I have a working java dataflow pipeline packaged as a jar file, and I can't get it to run from CC.  Still reviewing this code with another developer, so not ready to post details yet.

In the meanwhile, a general question:

Is there a well-documented demo that shows how to run DataFlow pipelines from Cloud Composer? I have not been able to find any so far.  I'd be looking for one demo for a java pipeline and one demo for a python pipeline. I'm looking for documentation of the process end-to-end, showing how to integrate my Dataflow code, build the environment (java or python, or if it is already built into CC, then demo that it's working right), install additional python dependencies that i may have, declare Airflow variables, etc.  You know, a self-contained document that has all the steps needed to actually get something to work with my code, not just code snippets and fragments and pointers to other web sites.  So a demo like the wordcount demo for Dataproc is not quite good enough (as it just runs pre-canned code, not my code.)

What's the best way to get started here?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.