Looping task in Taskqueue

[Iap]

Hi,

In my application, I have to pool the datestore every 3 seconds for
updating states.
I implement this pooling into taskqueue as a task.
For every execution, the task gets the job done and add itself to the
taskqueue for next execution.
Something like recursive execution.

The problem is that after a certain period, from one hour to several hours,
the number of tasks grows up. I understand that it is because the
auto-retry of the taskqueue.
Because of the TombStonedError, I can not assign name to task for
preventing the duplication.
Also, I have no idea to handle the TransientError. The document says
to retry again later.
Because this is a background running process, the question is "how to
be later?",
Put it in taskqueue! Then, I am like a python which swallow its own tail.

In javascript, ActionScript, there are "setTimeout" and "setInterval"
for one-time tasks and looping tasks.
In Twisted/Python, there are reactor.callLater and task.LoopingCall
for the both too.

How to do looping task in the taskqueue correctly?

Thanks

[Andrew Chilton]

Hi Iap,

On 23 February 2010 04:23, Iap <iap...@gmail.com> wrote:
> How to do looping task in the taskqueue correctly?

I think what you really need is to run a cron every so often. If your
requirements are more than once per minute, why not run a cron every
minute which adds a specific numberof tasks which are set to run
throughout that next minute.

If you're still set on using the task queue to spawn itself, there is
a header you can use called 'X-AppEngine-TaskRetryCount'. You can read
this value and then decide whether to spawn yourself or not.

See: http://code.google.com/appengine/docs/python/taskqueue/overview.html#Task_Request_Headers

For example, if that value is 1, then respawn. If it is greater than
1, then just end the current task (successfully) and do not respawn.
Have a play and see what works.

Hope that helps.

Cheers,
Andy

--
contact: Andrew Chilton
website: http://www.chilts.org/blog/

[Brandon Thomson]

cron does not run at 3 second intervals so that would not work.

if TransientError occurs the task worker will return an error and be
retried. as long as your task is idempotent or you are using a
transactional task you don't need to do anything. The bad news is it
sometimes takes 15-30s for this error to be raised and you will exceed
your 3s timeout.

on the other hand if you always return a non-error code from your
handler, even in the event of exception, then you would not need to
worry about the auto-retry adding duplicate tasks. maybe that is the
best solution for you.

[Eli Jones]

You just need to add a try: except: (if you're using Python) to handle TombstonedTaskError and TaskAlreadyExistsError.

I use a taskadd function that looks like this:

def taskAdd(name,counter,delay=0,wait=.5):

    try:

        taskqueue.add(name = name,url = '/mytasks/doTask',

                      countdown = delay, params = {'counter' : counter} )

    except (taskqueue.TaskAlreadyExistsError, taskqueue.TombstonedTaskError), e:

        pass

    except:

        from time import sleep

        sleep(wait)

        taskAdd(name,counter,delay,2*wait)

Once a task is done.. it adds a new one back to the queue like so:

counter = int(self.request.get('counter')) + 1

taskAdd(name+str(counter),counter)

That way.. if you get any error besides one indicating the Task is already there.. it just tries to add the task again a little later.

Of course.. you give it whatever delay you want.. in case you don't want it to start immediately.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Iap]

Hi,

I have tried the work-around same as the one provided by Eli below, I think that it works.

I set a serial number (the counter) for every generated tasks.

For every 3 seconds there is an execution.

Now, the serial number goes to 57470 counts. (almost 2 days)

2010/2/25 Eli Jones <eli....@gmail.com>