3 Proposed Enhancements

[Tim Jones]

I am using the Pipeline API in a production environment and generally speaking it does what I need it to, so many thanks for all the hard work you guys have put into it.  I do have two feature requests though that I think would make a large number of situations easier and more clear.

1. Branching

At present a pipeline always flows from beginning to end, but there are a number of situations where you want to change which jobs you spin off on the results on the basis of some intermediate value. 

I am currently getting around that problem by passing my conditional result down both paths and Excepting on the opposite of the value I need, something like:

ParentJob::run(int a, int b)

{

  FutureValue<Boolean> isGreaterThan = futureCall(new DoComparisonJob(), a, b);

  futureCall(new GreaterThanJob(),isGreaterThan);

  futureCall(new LessThanJob(), isLessThan);

}

GreaterThanJob::run(Boolean isGreaterThan)

{

    if (!isGreaterThan)

        throw new Exception("Wasn't Greater than");

    doStuff();

}

LessThanJob::run(Boolean isGreaterThan)

{

    if (isGreaterThan)

        throw new Exception("Wasn't Less than");

    doStuff();

}

This works, but it has issues, one of which is that it is hard to tell when a job is failed because I'm doing pseudo-branching vs when it has failed because something went wrong. It is also quite verbose and spreads the comparison operator across multiple files, thus making it more difficult to read the flow of the pipeline.  

Another workaround would be to simply to a comparison of one value vs another inside a comparison job and start one of two new pipelines on the basis of the result. The best version of that solution would also echo the status URL of the new pipeline to the console.  That has the disadvantage of course of having a less readable pipeline UI.

Ideally, there would be a way to simply execute one result or the other on the basis of the result.  Something like: 

BranchingParentJob::run(int a, int b)

{

  FutureValue<Boolean> isGreaterThan = futureCall(new DoComparisonJob(), a, b);

  if (isGreaterThan.getValue())

      futureCall(new GreaterThanJob(),isGreaterThan);

  else

      futureCall(new LessThanJob(), isLessThan);

}

I am aware that there would be drawbacks, and the Pipeline system would then not be able to lay out all jobs at a particular level immediately upon initial execution, but I would propose that there is a strong value to be gained from such a feature.

2. Failing a Job

Currently the only way to fail a job is to throw an exception.  There are at least two problems with that that I have run into.  First, it involves throwing an exception during normal execution.  I know this is a style question and can spark a religious war, so I won't belabor the point.

The second issue is more significant - you cannot throw a generic exception in a Job without deriving from the untyped Job class and then calling that job with a futureCallUnchecked.  This means of course that you no longer have the benefit of having the compiler check your types (which is significant as typing around Jobs can be tricky). It also means that you have to rework every futureCall which is also time consuming since the parameter ordering is different between futureCall and futureCallUnchecked.  

3. Unrecoverable Exception

Some errors are recoverable and some are not.  If you are using a job to make an API RPC call and the RPC is failing, then you should probably retry it.  On the other hand, if the job is able to detect that the input is invalid, then the Job should never be called again.  It would be great to be able to throw an UnrecoverableError and the Pipeline would never retry the Job again.  

I would love any feedback you guys have on these proposals, and thanks again for your time,

Tim

[Tom Kaitchuck]

See Arie's response in the other thread for the first two. 

3. Yes, we really should rethink how the automated retries are implemented now that we have error handling. (It seems odd to have both features and for them to be totally unrelated).

Obviously we could have retry built on exception handling, but it might lose some of it's convenient syntax. What would you like to see for the syntax for specifying what should/should not be retried?

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeli...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Tim Jones]

Hi Tom,

I've been pondering this since you posted the reply, and I'm note really sure I have the perfect answer to your question, but I also didn't want to leave you hanging.  What I would propose would simply be a new exception in the pipeline package: UnrecoverableJobException. If the Job throws an UnrecoverableJobException, it will not be retried.  If a parent Job has a handleException function, it is called immediately after the child class throws the Exception.  

Hope that helps!

Cheers,

Tim

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsub...@googlegroups.com.

[Tom Kaitchuck]

What are your thoughts on something like:

futureCall(new MyJob(), retrySettings(1000, 10).retryOn(FooException.class).abortOn(RuntimeException.class));

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeli...@googlegroups.com.

[Tim Jones]

Hey Tom, sorry, I missed this reply in December somehow. I like the syntax.  One of those two function may not be necessary though since the default behavior must be to do one or the other.  So, if you guys leave default behavior as always retrying, only abortOn would be necessary unless I'm missing something.

Peace,

Tim

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Tim Jones]

Just thought I'd check in and see if anything has been implemented?

Thanks,

Tim

[Tom Kaitchuck]

Sorry, no. Not a lot is new in Pipelines. 

I don't know if I mentioned this, but the syntax I mentioned comes from here: 

https://code.google.com/p/appengine-gcs-client/source/browse/trunk/java/src/main/java/com/google/appengine/tools/cloudstorage/RetryHelper.java
Which can totally be used to do the same thing within the scope of a single request / job.

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeli...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.