How hard would it be to make bits of Django compatible with async servers?

[Simon Willison]

I've been getting very excited about Node.js recently:

http://simonwillison.net/2009/Nov/23/node/

It's basically Twisted / Tornado but in JavaScript, and with the huge
advantage that, because it's brand new, it doesn't have any legacy
blocking APIs. Database access looks like this:

c.query("select * from articles").addCallback(function(rows) {
pretty_print(rows);
});

Even filesystem access is non-blocking:

posix.cat("/etc/passwd").addCallback(function(content) {
puts(content);
});

This made me think about non-blocking IO and Django. Theoretically,
many of the services that Django provides could be used in a non-
blocking environment such as that provided by Twisted or Tornado:

* URL routing shouldn't need to be blocking - it's just dispatching to
a function based on a regex (though this is stymied by thread locals)
* Form handling shouldn't need to block on I/O
* If we cache templates in memory instead of reading from disk each
time, rendering them shouldn't need to block on IO either
* SQL statement generation with the ORM shouldn't need to block either

That last one is particularly interesting to me. My understanding is
that much of the ORM refactoring for multi-db consisted of improving
the separation of SQL generation from actual database interaction.
Being able to do something like this would still be incredibly useful:

def my_async_view(request, response):
query = Blog.objects.filter(pubdate__year = 2009).exclude
(tag__slug = 'sport')
def done(rows):
response.render('blog.html', {'rows': rows})
db.execute_async(query, done)

The db.execute_async method would take a queryset and a callback,
execute the query asynchronously and pass the result to that callback.
I've invented an imaginary potential async replacement for the
blocking response=view(request) idiom.

I don't think async support should be baked in to Django core, but
it's a very interesting thing to think about. Should we eventually
break Django up in to smaller reusable libraries this kind of thing
would be an exciting use case.

[Russell Keith-Magee]

Multi-db doesn't change anything here. Query execution has always been
separate from query construction (at least, it has been since the
QuerySet refactor). In your example, the assignment of 'query'
constructs a query, but doesn't execute anything until you try to
iterate over the results.

> I don't think async support should be baked in to Django core, but
> it's a very interesting thing to think about. Should we eventually
> break Django up in to smaller reusable libraries this kind of thing
> would be an exciting use case.

Depends what you mean by 'baked into the core'. Given the emerging
importance of COMET et al, I think we would be foolish to say that
Django shouldn't develop capabilities to accommodate this sort of
development. This may mean making it easier to use Django components
(like the Forms library or URL dispatcher) outside of a Django
settings environment, or it may mean adding things to Django - the
COMET analog of mod_wsgi, for example. Constructive suggestions are
welcome, patches even more welcome.

Yours
Russ Magee %-)