Guidance on how to cleanup old pipeline entities safely?

[Frank Wilson]

Hi, 

Is there any guidance on how to safely cleanup entities left behind from previous pipeline runs? I'm referring to entities with kinds that start _AE_* . I'm worried about having a continually growing set of data I no longer need.

Thanks,

Frank

[Arie Ozarov]

1. You can use the Pipeline UI to delete old jobs.

2. You can use the Pipeline API (if you need a listing functionality like the UI has, have a look at this).

Arie | Ozarov | oza...@google.com | 415-624-6429

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeli...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frank Wilson]

Hi,

Thanks for your reply Arie. I forgot to mention I am using the Python Pipeline/MapReduce APIs perhaps the Java implementation is significantly different. 

When I list jobs at ${baseurl}/_ah/pipeline/list the list seems to just grow and grow and there doesn't seem to be any facility in the UI to delete the old job data. I'm also worried that if I delete entities without examining and understanding the source code carefully I might disrupt jobs that maybe in progress.

The links you sent to me seem to require a google.com (i.e. google intranet) login, are these the links you meant to send me? (Seems I can't use my google.com customer credentials at any rate).

Thanks,

Frank

--

FRANK WILSON

VP ENGINEERING

Memberoo

The Complete Membership Management Toolkit

e: frank....@memberoo.net

t: 01225 581599

a: 30-32 Westgate Buildings, Bath, BA1 1EF

w: memberoo.net

Tw: @memberoohq

Fb: /Memberoo

[Arie Ozarov]

Apologies for the broken links. Here are the public links:

Pipeline API for deleting a job - https://github.com/GoogleCloudPlatform/appengine-pipelines/blob/master/java/src/main/java/com/google/appengine/tools/pipeline/PipelineService.java#L223

And the listing - https://github.com/GoogleCloudPlatform/appengine-pipelines/blob/master/java/src/main/java/com/google/appengine/tools/pipeline/impl/servlets/JsonListHandler.java#L42

I am not that familiar with the Python code but it looks like this is how you list the root pipelines - https://github.com/GoogleCloudPlatform/appengine-pipelines/blob/master/python/src/pipeline/pipeline.py#L3215 and this is how you would cleanup/delete a pipeline - https://github.com/GoogleCloudPlatform/appengine-pipelines/blob/master/python/src/pipeline/pipeline.py#L2741 (unfortunately it does not look like the pipeline API has a nice way to do that). 

Arie | Ozarov | oza...@google.com | 415-624-6429

[Nickolas Daskalou]

Slightly off topic - if one were looking at implementing a complex, async workflow on App Engine (Python), is this Pipeline API still the recommended way of doing things, or has it been superseded by something else?

Nick

[Arie Ozarov]

Using AE Pipelines is completely fine and though control/maintenance was transferred to the open source community it is very useful and widely used.

The AE Pipelines is also used by the AE Mapreduce library but If the workflow is around data processing I would suggest looking at Google Cloud DataFlow instead.

Arie | Ozarov | oza...@google.com | 415-624-6429

[Nickolas Daskalou]

Thanks Arie.

Nick

[Nickolas Daskalou]

Arie,

Looks like the Dataflow Python SDK was just released - that's great!

How long do you think it will take for it to become "official" (like the Java SDK is)?

Nick

[Arie Ozarov]

I think the goal is to get some more feedback/experience to polish the API but you will get a

better answer if you ask this question to someone from that team.

[Frank Wilson]

Thanks for posting that Nickolas! I definitely want to check out Dataflow for Python!

Frank