Job coordination in Java with futureList

[Ronoaldo José de Lana Pereira]

I'm trying to wait for all child jobs before returning the value for this job:

class GenerateReport extends Job3<List<Void>, Set<String>, Long, Date> {

private static final long serialVersionUID = 1L;

@Override

public Value<List<Void>> run(Set<String> reservas, Long fileId, Date data) {

List<Value<Void>> results = new ArrayList<Value<Void>>();

for (String reservaKey : reservas) {

FutureValue<Long> sales = futureCall(new DaylySalesCount(), immediate(reservaKey), immediate(data));

results.add(futureCall(new WriteReportLine(), sales, immediate(reservaKey), immediate(fileId)));

}

return futureList(results);

}

}

.. but this raises an InvocationTargetException when handling this task. Am I doing a improper use of futureList()?

[Ronoaldo José de Lana Pereira]

Sorry, I forgot the stack trace:

2011-07-20 14:41:52.418:WARN::/_ah/pipeline/handleTask

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.appengine.tools.pipeline.impl.PipelineManager.invokePrivateJobMethod(PipelineManager.java:194)

at com.google.appengine.tools.pipeline.impl.PipelineManager.registerReturnValue(PipelineManager.java:225)

at com.google.appengine.tools.pipeline.impl.PipelineManager.runJob(PipelineManager.java:294)

at com.google.appengine.tools.pipeline.impl.PipelineManager.processTask(PipelineManager.java:149)

at com.google.appengine.tools.pipeline.impl.servlets.TaskHandler.doPost(TaskHandler.java:41)

at com.google.appengine.tools.pipeline.impl.servlets.PipelineServlet.doPost(PipelineServlet.java:58)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)

at com.ofertaunica.webapplication.server.servlets.TemplateDetectionFilter.doFilter(TemplateDetectionFilter.java:34)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.ofertaunica.webapplication.server.servlets.RegiaoFilter.doFilter(RegiaoFilter.java:149)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.ofertaunica.webapplication.server.servlets.VersionFilter.doFilter(VersionFilter.java:38)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:35)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:58)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:122)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:97)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

at com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:70)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:351)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[Mitch Rudominer]

Hi Ronoaldo,

Thanks for trying out Java Pipleine. Sorry for the trouble. Looking at your code I don't see that you have done anything wrong. Your use of futureList() seems correct to me. I will investigate more carefully tonight and get back to you tomorrow. In the mean time, do you happen to have any more of the stack trace? Usually an InvocationTargetException has a cause.

thanks,

Mitch

[Ronoaldo José de Lana Pereira]

Hi Mitch,

Thanks for your response. When running the standard local devserver under Eclipse Java EE, some FINE and other log levels don't show up. I'm not sure why, but after hacking some settings in my Eclipse setup I got the full stacktrace:

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:100)

at com.google.appengine.tools.pipeline.impl.PipelineManager.invokePrivateJobMethod(PipelineManager.java:190)

... 64 more

Caused by: java.lang.RuntimeException: Unrecognized type of Value: class com.google.appengine.tools.pipeline.FutureList

at com.google.appengine.tools.pipeline.Job.registerReturnValue(Job.java:99)

... 70 more

If I change the value of the return type to immediate(null) for example, it works fine. But I need to wait the child to be finished, otherwise the closeFinally() of my report will prevent the child jobs to write data to it. Looking in the source code here looks like there must be a way to handle a FutureList<?>. My current workaround is to return a futureCall() for a dummy job that just receives the future list and return a Value<Void>:

if (results.size() > 0) {

   return futureCall(new WaitForChildJobs(), futureList(results));

} else {

   return immediate(null);

}

Let me know if you need some help testing. I'll try to do a testCase for this. I also opened a issue to track it here: http://code.google.com/p/appengine-pipeline/issues/detail?id=37

Best Regards,

[Mitch Rudominer]

Hi Ronoaldo,

Thank you for that extra information. That helped a lot. This is a bug in Pipeline. I will fix it this morning. I'll let you know when it is ready.

- Mitch

[Mitch Rudominer]

Hi Ronoaldo,

I filed issue number 38 for the issue you found. I then fixed the problem with revision 50. Please get latest code and make sure your issue is resolved.

thanks,

Mitch

[Ronoaldo José de Lana Pereira]

Hi Mitch,

Thanks for your quick update on this. It works perfectly fine now!

I'm just curious about returning empty lists, because I got an error when the list is with zero size... Since it is not documented, I'm trying returning null. Is this the right to do it? Any best practice?

Best Regards,

- Ronoaldo

[Ronoaldo José de Lana Pereira]

By the way, you may want to merge Issue 37 with 38. I opened it before you, but it is a duplicate.

- Ronoaldo

[Mitch Rudominer]

You should be able to return an empty list. This is a bug. I may not be able to get to it for a day or two. If you would like, feel free to fix and submit a patch.

[Mitch Rudominer]

OK, thanks, I will.

[Ronoaldo José de Lana Pereira]

Hi Mitch,

I played around with the API, but no much luck yet ... I couldn't find a good way to represent a slot for an empty list, since the grouping attribute with value 0 means a single value returned in the internals (my understood from javadoc here). This may require more changes than I wold expect ... can you point me some directions, so I can contribute?

I attached a patch with the test case that fails when returning the empty FutureList, and also a small patch to configure the queue name to the backend via a system property. I needed that feature and it works fine in my production tests. This is not as good as we can do in Python, maybe we can add a JobSetting for this instead?

Hope to be able to help with the project, it is a very handy tool!

Best Regards,

- Ronoaldo

[Mitch Rudominer]

Hi Ronaldo,

Sorry for the delay in responding to this. I was out on vacation. Thanks a lot for your patch! I'll have a chance to look at it in the next day or two.

- Mitch

[Mitch Rudominer]

Ronaldo,

I have filed issue 34 for the futureList() not being able to handle an
empty list issue. I have also coded a fix for the problem and it
should be released in a day or two.

thanks,
Mitch

p.s. Next I will work on adding the task queue as a JobSetting.

On Aug 24, 10:50 am, Mitch Rudominer <rudomi...@google.com> wrote:
> Hi Ronaldo,
>
> Sorry for the delay in responding to this. I was out on vacation. Thanks a
> lot for your patch! I'll have a chance to look at it in the next day or two.
>
> - Mitch
>
> On Fri, Aug 5, 2011 at 8:01 AM, Ronoaldo José de Lana Pereira <
>
>
>
>
>
>
>
> rpere...@beneficiofacil.com.br> wrote:
> > Hi Mitch,
>
> > I played around with the API, but no much luck yet ... I couldn't find a
> > good way to represent a slot for an empty list, since the grouping attribute
> > with value 0 means a single value returned in the internals (my understood

> > from javadoc here<http://code.google.com/p/appengine-pipeline/source/browse/trunk/java/...>).

[Mitch Rudominer]

Ronoaldo,

My apologies. I just realized I have been spelling your name
incorrectly.

sorry,
Mitch

[Ronoaldo José de Lana Pereira]

Hello Mitch,

Thanks a lot for your feedback on this! Let me know if you need more testing. I have lots of pipelines where I can test it out on a live app. I'll take a look on the new r53 and give some feedback.

About my name: no problem, sometimes I make mistakes with my name too :) In fact, I didn't even noted ...

Best Regards,

- Ronoaldo