Async actions and adisp

85 views
Skip to first unread message

Frank Smit

unread,
Sep 2, 2011, 6:24:58 AM9/2/11
to python-...@googlegroups.com
I'm working on a small project to check if Momoko is really usable in
a real project. And I bumped into some limitations with adisp.

- When you `yield` a non-callable in your request handler (or any
other function decorated with `process`) an exception is raised,
because anything that's yielded is used as a callback. I fixed this in
Momoko's adisp.

You can now use an empty yield to stop the request and redirect for
example. Using `return` didn't work and an exception was raised when I
tried to use a return statement.

- When I put a query inside `prepare` (to fetch user data) and
decorate `prepare` with `process` the query is executed after the
request is finished.

So it goes like this:
1: prepare function starts
2: request handler is started
3: request does stuff and ends (it's done)
4: query is executed in the prepare function
5: prepare functions ends

This is not good, because that means you can only use async functions
inside the request handler get/post/... methods. This affects all code
that uses adisp (I think).

Ben Darnell

unread,
Sep 2, 2011, 1:08:47 PM9/2/11
to python-...@googlegroups.com
On Fri, Sep 2, 2011 at 3:24 AM, Frank Smit <fr...@61924.nl> wrote:
 - When I put a query inside `prepare` (to fetch user data) and
decorate `prepare` with `process` the query is executed after the
request is finished.

So it goes like this:
 1: prepare function starts
 2: request handler is started
 3: request does stuff and ends (it's done)
 4: query is executed in the prepare function
 5: prepare functions ends

This is not good, because that means you can only use async functions
inside the request handler get/post/... methods. This affects all code
that uses adisp (I think).

More generally, you can only use async functions from functions that are themselves asynchronous, and get/post/etc are the only parts of the RequestHandler interface that can be async (they are also atypical examples of async functions since they don't take a callback object directly, the final callback is implicitly available as self.finish()).  This contagious aspect of async interfaces is one reason Friendfeed used synchronous calls to mysql and memcache.  adisp lets the inside of your async methods look more like synchronous code, but as far as the caller is concerned the interface is still asynchronous.

-Ben

Ovidiu Predescu

unread,
Sep 2, 2011, 3:27:47 PM9/2/11
to python-...@googlegroups.com
Another limitation of the asynchronous model with adisp is that very
invasive. If you want to use a synchronous framework with adisp you
need to either modify the framework itself to make it asynchronous or
build an async wrapper around it.

In some cases it becomes impractical. For my app I started by writing
entity classes for each table in the database. This soon becomes
repetitive to the point I started thinking about an ORM layer to
replace the hand-written code. That is a lot of work I don't want to
do at the moment, so I decided to wrap Storm in an async layer. I
wrote a simple database wrapper that executes an entire function in a
thread on a thread pool. When done it invokes a callback on the main
thread. I wrapped this with adisp to make it easy to write code. The
result provides a nice separation between the sync and async code.

Ovidiu

Frank Smit

unread,
Sep 2, 2011, 4:40:50 PM9/2/11
to python-...@googlegroups.com, Ben Darnell
On Fri, Sep 2, 2011 at 7:08 PM, Ben Darnell <b...@bendarnell.com> wrote:
> More generally, you can only use async functions from functions that are
> themselves asynchronous, and get/post/etc are the only parts of the
> RequestHandler interface that can be async (they are also atypical examples
> of async functions since they don't take a callback object directly, the
> final callback is implicitly available as self.finish()).  This contagious
> aspect of async interfaces is one reason Friendfeed used synchronous calls
> to mysql and memcache.  adisp lets the inside of your async methods look
> more like synchronous code, but as far as the caller is concerned the
> interface is still asynchronous.
> -Ben

I see. Async queries would, in that case, only be useful if the query
takes really long (if it's not cached). I think. Right?

I guess Momoko isn't really of any use, unless I add support for
blocking connections since DB queries don't tend to be very long.

Ben Darnell

unread,
Sep 3, 2011, 2:41:19 PM9/3/11
to Frank Smit, python-...@googlegroups.com
Async queries are more useful the longer the query takes.  They're never completely useless even for fast queries (finer-grained IOLoop callbacks improve scheduling and smooth latency), but at some point they're not worth the complexity they introduce (that point will vary depending on how comfortable you are with the asynchronous model and details of your servers' resource utilization).  Personally, I'd stick with a synchronous interface for memcache, but I'd consider an async database interface. 

-Ben
Reply all
Reply to author
Forward
0 new messages