Is appengine pipeline still supported?

Michal Kozak

unread,

Nov 30, 2016, 8:32:20 PM11/30/16

to Google App Engine Pipeline API

Hi,

I'm considering building app engine app that is using appengine pipeline.

What is the status of this product?

Is it still supported or is it deprecated?

What is it's relation in regards to cloud dataflow / apache_beam ?

Thank you

Michal

Arie Ozarov

unread,

Dec 1, 2016, 2:29:36 PM12/1/16

to app-engine-...@googlegroups.com

The library was transition to the open source community.

You should favor cloud dataflow if possible.

Arie | Ozarov | oza...@google.com | 415-624-6429

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Michal Kozak

unread,

Dec 20, 2016, 5:45:17 PM12/20/16

to Google App Engine Pipeline API

Thank you for reply. I can use dataflow for many use-cases but not all - and clearly cannot easily run python dataflow from app engine standard. Does the pipelines library still work on appengine and prometeus? I'm asking as it doesn't seem to work for me and would be great if someone could help me understanding why.

Arie Ozarov

unread,

Dec 21, 2016, 11:50:08 AM12/21/16

to app-engine-...@googlegroups.com

Yes, it should work and in fact it is still used by many application.

If you describe the issue that you are experiencing maybe someone in the group can help.

Arie | Ozarov | oza...@google.com | 415-624-6429

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michal Kozak

unread,

Dec 21, 2016, 12:35:45 PM12/21/16

to app-engine-...@googlegroups.com

Thank you for reply, I managed to make it work.

On Wed, Dec 21, 2016 at 8:49 AM, 'Arie Ozarov' via Google App Engine Pipeline API <app-engine-...@googlegroups.com> wrote:

Yes, it should work and in fact it is still used by many application.
If you describe the issue that you are experiencing maybe someone in the group can help.

On Tue, Dec 20, 2016 at 2:45 PM, 'Michal Kozak' via Google App Engine Pipeline API <app-engine-pipeline-api@googlegroups.com> wrote:
Thank you for reply. I can use dataflow for many use-cases but not all - and clearly cannot easily run python dataflow from app engine standard. Does the pipelines library still work on appengine and prometeus? I'm asking as it doesn't seem to work for me and would be great if someone could help me understanding why.

On Thursday, December 1, 2016 at 11:29:36 AM UTC-8, Arie Ozarov wrote:
The library was transition to the open source community.
You should favor cloud dataflow if possible.

Arie | Ozarov | oza...@google.com | 415-624-6429

On Wed, Nov 30, 2016 at 5:32 PM, 'Michal Kozak' via Google App Engine Pipeline API <app-engine-...@googlegroups.com> wrote:
Hi,

I'm considering building app engine app that is using appengine pipeline.

What is the status of this product?
Is it still supported or is it deprecated?
What is it's relation in regards to cloud dataflow / apache_beam ?

Thank you
Michal

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/app-engine-pipeline-api/VOzw_FUGxIM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to app-engine-pipeline-api+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- Michal

er...@worklytics.co

unread,

Jan 15, 2019, 4:32:36 PM1/15/19

to Google App Engine Pipeline API

By "transition to the open source community", what do you mean? There are ~dozen pending PRs that no one seems to be dealing with; who is maintaining it? Anyone?

Cloud Dataflow seems like a great replacement for MapReduce, but after playing for a couple days I don't find it as nice for orchestration equivalent to the Pipelines framework. We're Java-based and typically invoke Pipeline jobs on the fly from our application code, in response to user actions. Canonical use-case would be provisioning an account for a new customer: lots of little things that need to happen in lots of places. Generally, these can be parallelized but some have depedencies between them.

Is Cloud Composer the preferred solution for this now? Seems like that's little more than "managed Airflow", with only Python supported.

On Thursday, December 1, 2016 at 11:29:36 AM UTC-8, Arie Ozarov wrote:

Rajesh Gupta

unread,

Jan 15, 2019, 7:46:23 PM1/15/19

to Google App Engine Pipeline API

We are heavily datastore based. It is sad that suddenly important projects is transitioned to opensource with pending PR's and very less documentation.

GCP is adding more load to developers by adding new technologies. Suddenly, sometimes GCP team works against the existing old projects and creates panic to the small teams.
There are no simple pipepline technology to manage/migrate data on datastore. :(.

Reply all

Reply to author

Forward