Interaction between ndb and taskqueue.Queue's add_async

[Michael Olson]

I'm currently using the asynchronous facilities provided by the ndb library, including the ndb context functions for accessing memcache asynchronously.  I'm trying to integrate those functions with the taskqueue.Queue's add_async function described here:

https://developers.google.com/appengine/docs/python/taskqueue/queues#Queue_add_async

It seems that if I am not interested in the result of the add operation, I can call:

    task_rpc = queue.add_async(...)

    ndb.eventloop.queue_rpc(task_rpc)

and an ndb.toplevel function will guarantee that the rpc completes.  If I am interested in the result of the operation, however, such as within a transaction - to guarantee it has succeeded - or for catching and acting on generated exceptions, there is not a clear way to deal with the generated rpc.  My understanding of both sets of code is that if I call task_rpc.check_success() before the rpc has completed, it will cause a busy wait that will prevent any other ndb tasklets from executing.  My hack thus far has been:

    ndb.eventloop.run()

    do_something(task_rpc)

So I attempt to guarantee that all rpcs have completed before I check the result of the taskqueue rpc.

Is there a more elegant way to integrate these two sets of functionality, or are there any plans to add an ndb context function for adding taskqueue jobs?

Thanks in advance for any insight.

[James A. Morrison]

On Wed, Jun 12, 2013 at 11:46 AM, Michael Olson <michae...@gmail.com> wrote:

I'm currently using the asynchronous facilities provided by the ndb library, including the ndb context functions for accessing memcache asynchronously.  I'm trying to integrate those functions with the taskqueue.Queue's add_async function described here:

https://developers.google.com/appengine/docs/python/taskqueue/queues#Queue_add_async

It seems that if I am not interested in the result of the add operation, I can call:

    task_rpc = queue.add_async(...)

    ndb.eventloop.queue_rpc(task_rpc)

 Why don't you have a tasklet for this:

@ndb.tasklet

def add_task_async(...):

  rpc = queue.add_async(...)

  yield rpc

  raise ndb.Return(rpc.get_result())

 

and an ndb.toplevel function will guarantee that the rpc completes.  If I am interested in the result of the operation, however, such as within a transaction - to guarantee it has succeeded - or for catching and acting on generated exceptions, there is not a clear way to deal with the generated rpc.  My understanding of both sets of code is that if I call task_rpc.check_success() before the rpc has completed, it will cause a busy wait that will prevent any other ndb tasklets from executing.  My hack thus far has been:

    ndb.eventloop.run()

    do_something(task_rpc)

So I attempt to guarantee that all rpcs have completed before I check the result of the taskqueue rpc.

Is there a more elegant way to integrate these two sets of functionality, or are there any plans to add an ndb context function for adding taskqueue jobs?

Thanks in advance for any insight.

--
You received this message because you are subscribed to the Google Groups "appengine-ndb-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appengine-ndb-di...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Thanks,
Jim
http://phython.blogspot.com

[Michael Olson]

 Why don't you have a tasklet for this:

@ndb.tasklet

def add_task_async(...):

  rpc = queue.add_async(...)

  yield rpc

  raise ndb.Return(rpc.get_result())

To be honest, it wasn't clear to me that the ndb eventloop functionality supported yielding to anything other than the google.appengine.ext.ndb.tasklets.Future objects; I clearly didn't read the documentation closely enough, as I now see that spelled out in the main docs page.

Thanks for the help.