Groups

Google App Engine Pipeline API

1–30 of 179

0 selected

Michal Kozak, … Rajesh Gupta7

1/15/19

Is appengine pipeline still supported?

We are heavily datastore based. It is sad that suddenly important projects is transitioned to

unread,

Is appengine pipeline still supported?

We are heavily datastore based. It is sad that suddenly important projects is transitioned to

1/15/19

Luke Zulauf, wTyeRogers3

9/27/17

Running a pipeline in a single process

Basic scaling is an interesting option that I hadn't considered before. Going through the code I

unread,

Running a pipeline in a single process

Basic scaling is an interesting option that I hadn't considered before. Going through the code I

9/27/17

Ronoaldo Pereira

4/6/16

Implementing Pipelines in Go programming language

Hi all! App Engine Pipelines and MapReduce is a somewhat must-have tool in order to build some more

unread,

Implementing Pipelines in Go programming language

Hi all! App Engine Pipelines and MapReduce is a somewhat must-have tool in order to build some more

4/6/16

Mayank Bhagya, … Arie Ozarov5

3/1/16

Abort only a child pipeline

Hello Arie. Yes, this was regarding python. On Tuesday, 1 March 2016 22:18:10 UTC+5:30, Arie Ozarov

unread,

Abort only a child pipeline

Hello Arie. Yes, this was regarding python. On Tuesday, 1 March 2016 22:18:10 UTC+5:30, Arie Ozarov

3/1/16

Frank Wilson, … Arie Ozarov10

2/29/16

Guidance on how to cleanup old pipeline entities safely?

Thanks for posting that Nickolas! I definitely want to check out Dataflow for Python! Frank On 27

unread,

Guidance on how to cleanup old pipeline entities safely?

Thanks for posting that Nickolas! I definitely want to check out Dataflow for Python! Frank On 27

2/29/16

Moisés Belchín, Erwan Macé2

9/9/15

mapreduce.yaml and NDB filters

I have found however than in my case, I had to write the filters as a String containing a list of

unread,

mapreduce.yaml and NDB filters

I have found however than in my case, I had to write the filters as a String containing a list of

9/9/15

Mayank Bhagya, Brett Slatkin4

8/26/15

Controlling queue name of yielded pipelines

I figure you could add it there? On Wed, Aug 26, 2015 at 11:28 AM Mayank Bhagya <mayankbhagya@

unread,

Controlling queue name of yielded pipelines

I figure you could add it there? On Wed, Aug 26, 2015 at 11:28 AM Mayank Bhagya <mayankbhagya@

8/26/15

8/21/15

Increased memory footprint after pipelining

Okay I figured. It is this line: "db.put (entities_to_put)" in the 'evaluate'

unread,

Increased memory footprint after pipelining

Okay I figured. It is this line: "db.put (entities_to_put)" in the 'evaluate'

8/21/15

Dmitry Sadovnychyi, … Santiago L. Valdarrama4

4/1/15

How to run tests?

I'm having exactly the same problem and I don't seem to find a solution for it. On Thursday,

unread,

How to run tests?

I'm having exactly the same problem and I don't seem to find a solution for it. On Thursday,

4/1/15

Julien Piquot, Arie Ozarov3

3/20/15

import Datastore entities inside BigQuery : problem while reading csv files (regression issue ?)

Hello, Thanks for your answer. I just published the same message on Github. Julien Le vendredi 20

unread,

import Datastore entities inside BigQuery : problem while reading csv files (regression issue ?)

Hello, Thanks for your answer. I just published the same message on Github. Julien Le vendredi 20

3/20/15

Bardia D., … Tom Kaitchuck6

10/20/14

Execution Ordering - Newbie Question

I am a bit confused. In your earlier posts you indicated that D was spawned from within C and E

unread,

Execution Ordering - Newbie Question

I am a bit confused. In your earlier posts you indicated that D was spawned from within C and E

10/20/14

David Fuelling, Tom Kaitchuck2

10/17/14

Relation Between Mapper Context and Reducer Context?

The context objects are separate. On Tue, Sep 30, 2014 at 8:37 AM, David Fuelling <sappenin@gmail.

unread,

Relation Between Mapper Context and Reducer Context?

The context objects are separate. On Tue, Sep 30, 2014 at 8:37 AM, David Fuelling <sappenin@gmail.

10/17/14

Patrice De Saint Steban, … Tom Kaitchuck10

10/2/14

Save all results in one google cloud file

Sorry my math is screwy above. A more exact formula is that in P passes it can process data as large

unread,

Save all results in one google cloud file

Sorry my math is screwy above. A more exact formula is that in P passes it can process data as large

10/2/14

David Fuelling, Tom Kaitchuck4

9/29/14

Release Notes for 0.8 appengine-mapreduce release?

Hey, this is great! It's a feature I was just thinking about requesting, so glad to hear it's

unread,

Release Notes for 0.8 appengine-mapreduce release?

Hey, this is great! It's a feature I was just thinking about requesting, so glad to hear it's

9/29/14

Jeffrey Tratner, Tom Kaitchuck3

9/22/14

Using cloudstorage library instead of Files API for shuffle in Python?

Great! Looking forward to it! On Thursday, September 18, 2014 12:57:39 PM UTC-7, Tom Kaitchuck wrote:

unread,

Using cloudstorage library instead of Files API for shuffle in Python?

Great! Looking forward to it! On Thursday, September 18, 2014 12:57:39 PM UTC-7, Tom Kaitchuck wrote:

9/22/14

jwor...@sportsrisq.com, Arie Ozarov2

9/17/14

Retrying a pipeline

You should follow the advice given in your SO thread http://stackoverflow.com/questions/25883652/

unread,

Retrying a pipeline

You should follow the advice given in your SO thread http://stackoverflow.com/questions/25883652/

9/17/14

Bill Speirs, Tom Kaitchuck2

9/12/14

Tasks Continually Retried because of 429 "App Error"

This is normal behaviour for a MapReduce that has more shards than workers, It is simply the library

unread,

Tasks Continually Retried because of 429 "App Error"

This is normal behaviour for a MapReduce that has more shards than workers, It is simply the library

9/12/14

8/18/14

Using mapreduce with PIP?

The "official" python docs (https://developers.google.com/appengine/docs/python/

unread,

Using mapreduce with PIP?

The "official" python docs (https://developers.google.com/appengine/docs/python/

8/18/14

Tim Jones, Tom Kaitchuck7

8/4/14

3 Proposed Enhancements

Sorry, no. Not a lot is new in Pipelines. I don't know if I mentioned this, but the syntax I

unread,

3 Proposed Enhancements

Sorry, no. Not a lot is new in Pipelines. I don't know if I mentioned this, but the syntax I

8/4/14

Dragoslav Mitrinovic

7/25/14

sharding improvement for equality filters (DatastoreInputReader, Python)

We've recently started using equality filters in mapper jobs configured with DatastoreInputReader

unread,

sharding improvement for equality filters (DatastoreInputReader, Python)

We've recently started using equality filters in mapper jobs configured with DatastoreInputReader

7/25/14

Caio Iglesias, David Hardwick4

7/18/14

Expensive DataStore writes on _AE_Pipeline_Barrier

By running child pipelines I'm saving on some writes. Down to 208, from which 156 are on

unread,

Expensive DataStore writes on _AE_Pipeline_Barrier

By running child pipelines I'm saving on some writes. Down to 208, from which 156 are on

7/18/14

Jesús Valladares Solis

7/9/14

Creating a FutureValue from an instance of MapJob [ futureCall(new MapJob<>(...)) ] produce the wrong return type (AxisTypesParameter)

just filed a bug: https://code.google.com/p/appengine-mapreduce/issues/detail?id=218 public class

unread,

Creating a FutureValue from an instance of MapJob [ futureCall(new MapJob<>(...)) ] produce the wrong return type (AxisTypesParameter)

just filed a bug: https://code.google.com/p/appengine-mapreduce/issues/detail?id=218 public class

7/9/14

7/8/14

Threading Heisenbug in mapreduce- style financial simulation

Hi, I'm using the Pipeline API plus cloudstorage to parallelise a large financial simulation,

unread,

Threading Heisenbug in mapreduce- style financial simulation

Hi, I'm using the Pipeline API plus cloudstorage to parallelise a large financial simulation,

7/8/14

Ricardo Diaz, Brett Slatkin3

6/26/14

Reducer shard with too many values

Thanks Brett, that help me. On Wednesday, June 25, 2014 5:02:34 PM UTC-5, Brett wrote: Use a combiner

unread,

Reducer shard with too many values

Thanks Brett, that help me. On Wednesday, June 25, 2014 5:02:34 PM UTC-5, Brett wrote: Use a combiner

6/26/14

Caio Iglesias, … Brett Slatkin4

6/16/14

UnexpectedPipelineError: Yielded a disallowed value: handlers.importpipeline.AddOne(*(15,), **{})

Great! Happy to hear it. And thanks. On Mon, Jun 16, 2014 at 1:53 PM, Caio Iglesias <caioiglesias@

unread,

UnexpectedPipelineError: Yielded a disallowed value: handlers.importpipeline.AddOne(*(15,), **{})

Great! Happy to hear it. And thanks. On Mon, Jun 16, 2014 at 1:53 PM, Caio Iglesias <caioiglesias@

6/16/14

Mark Larimer, … Josh Whelchel (loudr.fm)4

6/5/14

Retrying a generator pipeline from a child stage?

You have the option of yielding a duplicate of pipeline A if AC finds errors. Then you have a

unread,

Retrying a generator pipeline from a child stage?

You have the option of yielding a duplicate of pipeline A if AC finds errors. Then you have a

6/5/14

aswath, … Arie Ozarov4

5/7/14

Using namespace

No, you can only provide one namespace. The Datastore does not provide a way to use one query for

unread,

Using namespace

No, you can only provide one namespace. The Datastore does not provide a way to use one query for

5/7/14

Aswath Satrasala2

4/29/14

Mapreduce example not completing the job

Sorry about the email. There was issue with deploying to the remote server. The mapreduce job is

unread,

Mapreduce example not completing the job

Sorry about the email. There was issue with deploying to the remote server. The mapreduce job is

4/29/14

Andrew Ducker, Tom Kaitchuck5

3/29/14

Handling a job "giving up"

Aaaah. Thank you - that makes perfect sense, but I misunderstood that bit of the docs! On Monday, 24

unread,

Handling a job "giving up"

Aaaah. Thank you - that makes perfect sense, but I misunderstood that bit of the docs! On Monday, 24

3/29/14

Gilberto Torrezan Filho, Tom Kaitchuck5

3/24/14

Best practices on sorting values received by the reducer?

To clarify in this case you could have a single reducer write to 3 files, for each of A > B, A = B

unread,

Best practices on sorting values received by the reducer?

To clarify in this case you could have a single reducer write to 3 files, for each of A > B, A = B

3/24/14

Search

Clear search

Close search

Google apps

Main menu