Task Guarantee?

[Richard Arrano]

Hello,
I have been designing my app with the notion in mind that even named
tasks may execute more than once, but I only recently came to realize
that a task may not execute at all. I have a task that operates on a
subset of my entities and it's absolutely imperative that all members
of this subset get processed and saved. I originally thought named
tasks would help accomplish this, but this does not seem to be the
case. Is there any way to guarantee that I process these entities? I
also considered a cron job that checks every couple of minutes to
check for unprocessed entities(since a cron job will kick off the
initial task) but I was hoping for a slightly more elegant solution.

Thanks,
Richard

[Per]

If you need to be super-certain, then you will always want to have a status flag that tells you if you have already processed some entity or not. It's also useful for cases where you cannot use transactions (a task might die in the middle as well of course)

Well, at least that's how we do it :) Whenever something is important, we want to track some kind of status anyhow. One of these statuses would be STATUS_UNPROCESSED or so.

Cheers,
Per

[pdknsk]

> tasks may execute more than once, but I only recently came to realize
> that a task may not execute at all.

How so? I've noticed the former, but not the latter.

[Michael Hermus]

Same here; I would be very concerned if this were the case. I use transactional task en-queues; once successfully on a queue, I have never seen a task fail to attempt execution. In cases where the execution is not successful, the task system correctly retries the task as specified by configuration.

I don't expect any guarantees around timing of execution, although for the most part the ETA settings work pretty well.

[stevep]

Having no insight about you logic... Agree with other responses that while the timing of a task queue task may be unpredictable, the final execution of said task is pretty certain. If you want more control, then perhaps a pull queue may be an option since you should easily find a means to record/log what is in the queue, and delete the queue items only after your logic has ascertained its final disposition is proven (e.g. completed a put() operation without exception). HTH -stevep

[Richard Arrano]

I also have not seen tasks fail to run, but I found this thread:

 

http://stackoverflow.com/questions/5583813/google-app-engine-added-task-goes-missing

 

Specifically, the part that says: "Tasks are not guaranteed to be executed in the order they arrive, and they are not guaranteed to be executed exactly once. In some cases, a single task may be executed more than once or not at all."

 

I haven't seen the behavior myself, and perhaps the commenter is not correct, but it occurred to me that I need to account for the possibility. I believe I will have a status flag a la Per's suggestion. Thanks!

 

-Richard 

[pdknsk]

Well this Travis Webb isn't a Googler, so I wouldn't put much stock
into his claim. I'm quite sure Google already has logic in place for
this, so any additional measures seem reduntant.

[Michael Hermus]

I think that is from an old version of the documentation when the Task Queue was an experimental feature. The current version of documentation does not match (https://developers.google.com/appengine/docs/java/taskqueue/overview-push). It says:

When implementing the code for tasks (as worker URLs within your app), it is important to consider whether the task is idempotent. App Engine's Task Queue API is designed to only invoke a given task once; however, it is possible in exceptional circumstances that a task may execute multiple times (such as in the unlikely case of major system failure). Thus, your code must ensure that there are no harmful side-effects of repeated execution

I do not think you have to worry about tasks being lost or dropped (if used correctly).