Integrating Django with Tornado's web server
Bret Taylor
web server/framework we built at FriendFeed that we open sourced last
week (see http://bret.appspot.com/entry/tornado-web-server).
I just checked in change to Tornado that enables you to run any WSGI-
compatible framework on Tornado's HTTP server so that Django apps
could run on top of Tornado's HTTP server and benefit from some of the
performance work we have done. (I just sent a message to django-users@
with getting started instructions as well, but if you are interested,
take a look at http://github.com/facebook/tornado/blob/master/tornado/wsgi.py#L188).
I chose the WSGI approach because it is generic and applies to all
frameworks, but Django is obviously the most widely used. I am curious
if there is any benefit to implementing more "native" support in
django.core.handlers or if WSGI is the preferred way of adding support
for new servers. If there is any performance or usability benefit, let
me know, because we would be happy to contribute our time to make it
happen.
In the meantime, if you find any issues with our WSGI support, let me
know so we can fix problems.
Bret
Russell Keith-Magee
>
> I am one of the authors of Tornado (http://www.tornadoweb.org/), the
> web server/framework we built at FriendFeed that we open sourced last
> week (see http://bret.appspot.com/entry/tornado-web-server).
>
> I just checked in change to Tornado that enables you to run any WSGI-
> compatible framework on Tornado's HTTP server so that Django apps
> could run on top of Tornado's HTTP server and benefit from some of the
> performance work we have done. (I just sent a message to django-users@
> with getting started instructions as well, but if you are interested,
> take a look at http://github.com/facebook/tornado/blob/master/tornado/wsgi.py#L188).
This is awesome news, Bret! I've only taken a brief look at Tornado so
far, but it certainly looks interesting; having easy support for WSGI
(and therefore Django) makes it even more compelling.
> I chose the WSGI approach because it is generic and applies to all
> frameworks, but Django is obviously the most widely used. I am curious
> if there is any benefit to implementing more "native" support in
> django.core.handlers or if WSGI is the preferred way of adding support
> for new servers. If there is any performance or usability benefit, let
> me know, because we would be happy to contribute our time to make it
> happen.
I'd be surprised if there aren't some friction points that could be
optimized by using a Tornado-native interface, especially if you're
looking to exploit some of the specific capabilities of Tornado.
However, this leads us to an interesting chicken-and-egg situation. No
offense intended to yourself and the great work you have done, but
Tornado isn't a major market leader (not yet, anyway). The Django core
team is very sensitive to the fact that adding something to the Django
Core is effectively seen as "blessing" something as stable and worth
using, and Tornado hasn't been around long enough in the public
sphere to warrant that kind of recommendation.
At least initially, I'd prefer to see Django-Tornado support as a
project external to the Django core. It's all just Python code, after
all - having a Tornado handler in the Django tree rather than
somewhere else PYTHONPATH doesn't make the underlying functionality
any more or less accessible. The existing mod_python and mod_wsgi
interfaces could easily live outside the Django tree; they're just
present as part of the 'batteries included' strategy that covers the
most common mechanisms for deploying Python-based web applications at
present.
If, in the process of building and external Django support library for
Tornado, you find that it is necessary to modify the core in order to
support a particular feature of Tornado, raise a ticket in Django's
Trac instance, start a discussion on this mailing list, and we'll look
at making that change.
This externalized approach would also be in the interests of the
binding anyway. The rapid release/update schedules required by a new
and emerging project aren't really compatible with an established
framework like Django where the initial rapid development effort has
plateaued and been replaced with a slower process of incremental
updates. Long term, it may be appropriate to integrate Tornado support
into the Django core, but it would be better to wait until the
interface has matured before we look at integration.
Yours,
Russ Magee %-)
Bret Taylor
wrote:
> > frameworks, but Django is obviously the most widely used. I am curious
> > if there is any benefit to implementing more "native" support in
> > django.core.handlers or if WSGI is the preferred way of adding support
> > for new servers. If there is any performance or usability benefit, let
> > me know, because we would be happy to contribute our time to make it
> > happen.
>
> I'd be surprised if there aren't some friction points that could be
> optimized by using a Tornado-native interface, especially if you're
> looking to exploit some of the specific capabilities of Tornado.
>
> However, this leads us to an interesting chicken-and-egg situation. No
> offense intended to yourself and the great work you have done, but
> Tornado isn't a major market leader (not yet, anyway). The Django core
> team is very sensitive to the fact that adding something to the Django
> Core is effectively seen as "blessing" something as stable and worth
> using, and Tornado hasn't been around long enough in the public
> sphere to warrant that kind of recommendation.
Django "native" handler as a module in Tornado, not actually moving
the module to Django core.
It is a bit unclear to me at this point what the handler interface is
- some of the modules in that package subclass http.HttpRequest and
some subclass base.BaseHandler, but the assumptions about environment
variables seem to be spread between a bit spread between the modules.
I will just leave it as wsgi for now unless there is demand.
Bret
Graham Dumpleton
On Sep 14, 8:05 am, Russell Keith-Magee <freakboy3...@gmail.com>
wrote:
>
> > I am one of the authors of Tornado (http://www.tornadoweb.org/), the
> > web server/framework we built at FriendFeed that we open sourced last
> > I just checked in change to Tornado that enables you to run any WSGI-
> > compatible framework on Tornado's HTTP server so that Django apps
> > could run on top of Tornado's HTTP server and benefit from some of the
> > performance work we have done. (I just sent a message to django-users@
> > with getting started instructions as well, but if you are interested,
uses the standardised WSGI interface that Django provides and which is
the same interface that all WSGI capable hosting mechanisms would use.
Graham
Mike Malone
> compatible framework on Tornado's HTTP server so that Django apps
> could run on top of Tornado's HTTP server and benefit from some of the
> performance work we have done. (I just sent a message to django-users@
> with getting started instructions as well, but if you are interested,
> take a look at http://github.com/facebook/tornado/blob/master/tornado/wsgi.py#L188).
Great news, there was a lot of discussion at DjangoCon last week after
the Tornado launch about this possibility. There was a bit of hacking
on it during the Django sprints, but things seemed to stall as people
realized there were a few incompatibilities due to Tornado munging
some of the HTTP headers (not sure the details, as I wasn't working on
this stuff personally). Awesome to hear you made it work!
> I chose the WSGI approach because it is generic and applies to all
> frameworks, but Django is obviously the most widely used. I am curious
> if there is any benefit to implementing more "native" support in
> django.core.handlers or if WSGI is the preferred way of adding support
> for new servers. If there is any performance or usability benefit, let
> me know, because we would be happy to contribute our time to make it
> happen.
It would be interesting if you could formalize your issues with WSGI
(it's not a horrible solution, but there are certainly places where it
could be improved). Armin Ronacher was talking about updating the WSGI
PEP at DjangoCon and was soliciting ideas, so you may want to get in
touch with him and offer some suggestions for making WSGI work with
high performance containers like Tornado.
Mike
Graham Dumpleton
On Sep 15, 3:25 am, Mike Malone <mjmal...@gmail.com> wrote:
> > I just checked in change to Tornado that enables you to run any WSGI-
> > compatible framework on Tornado's HTTP server so that Django apps
> > could run on top of Tornado's HTTP server and benefit from some of the
> > performance work we have done. (I just sent a message to django-users@
> > with getting started instructions as well, but if you are interested,
> Great news, there was a lot of discussion at DjangoCon last week after
> the Tornado launch about this possibility. There was a bit of hacking
> on it during the Django sprints, but things seemed to stall as people
> realized there were a few incompatibilities due to Tornado munging
> some of the HTTP headers (not sure the details, as I wasn't working on
> this stuff personally). Awesome to hear you made it work!
>
> > I chose the WSGI approach because it is generic and applies to all
> > frameworks, but Django is obviously the most widely used. I am curious
> > if there is any benefit to implementing more "native" support in
> > django.core.handlers or if WSGI is the preferred way of adding support
> > for new servers. If there is any performance or usability benefit, let
> > me know, because we would be happy to contribute our time to make it
> > happen.
>
> It would be interesting if you could formalize your issues with WSGI
> (it's not a horrible solution, but there are certainly places where it
> could be improved). Armin Ronacher was talking about updating the WSGI
> PEP at DjangoCon and was soliciting ideas, so you may want to get in
> touch with him and offer some suggestions for making WSGI work with
> high performance containers like Tornado.
WSGI in a thread pool on top of async is a bad idea for a multiprocess
web server because you can't prevent the process accepting more
requests than the WSGI application can handle via the thread pool at
any one time. End result is that you can unnecessarily block requests
in one process which could otherwise have been handled by another
process which has available resources to handle them. WSGI on top of
non async multiprocess servers like Apache don't have this issue as
Apache will only accept connections when it knows that it has
available threads to handle the request.
Part of the reason that any update to WSGI specification has not come
out, is because the async crowd has derailed the discussion a couple
of times. The core people around WSGI do not believe that async
support should be a part of it, so please, if you want a standardised
async interface, do it independently.
Graham
