pure-HTTP deployment?
Chad Whitacre
I'm interested in deploying Django as a pure-Python HTTP daemon,
much like Zope and CherryPy are deployed. Is this done at all w/
Django?
Thanks.
chad
Jacob Kaplan-Moss
> I'm interested in deploying Django as a pure-Python HTTP daemon, much
> like Zope and CherryPy are deployed. Is this done at all w/ Django?
I'm pretty sure that's how Simon is serving his new site -- see
http://simonwillison.net/2006/Dec/20/nginx/#comments. Simon, if you're
reading this, want to chime in?
Jacob
Simon Willison
This is a long term point of frustration for me. I /was/ using CherryPy
- for all of 8 hours - but it just didn't seem stable enough. What I
really want is a pure-Python server that can serve up a WSGI
application and doesn't need to be baby-sat. For whatever reason,
CherryPy was giving me "bad gateway" errors to my nginx front-end. I
could have set up something like supervisord, daemontools or monit to
monitor and automatically restart it but I'd rather avoid the
administration overhead.
So, I'm now serving simonwillison.net using a stripped down
mod_python/Apache server (mod_python is pretty much the only module
loaded) that's sat behind nginx, which is configured to serve up my
static files and proxy the dynamic requests through to Apache. I'm
using Apache as a python application server basically because I can
rely on it staying up. It's been working great so far.
nginx is fantastic as a front-end static file server / proxy /
load-balancer (when I was running CherryPy I had two CherryPys
load-balanced by nginx). Well worth checking out.
One alternative I haven't explored fully yet is fastcgi, in particular
the fastcgi variant which is managed by the web server (so you don't
have to manage the fastcgi processes yourself).
So... if anyone knows of (or wants to write) a robust pure-python WSGI
server that comes with tools for properly managing it as a daemon, I'm
all ears. Until then, I'll probably stick with mod_python/Apache.
Cheers,
Simon
Chad Whitacre
Thanks for chiming in.
> So... if anyone knows of (or wants to write) a robust pure-python WSGI
> server that comes with tools for properly managing it as a daemon, I'm
> all ears. Until then, I'll probably stick with mod_python/Apache.
Well, after that I have to tip my hand. :^)
I've started such a project, called Aspen:
http://www.zetadev.com/software/aspen/
I show in the screencast how one could use Aspen to serve a
Django app. I'm asking on this list because I think Aspen could
possibly be a good fit for Django HTTP deployments.
The catch is that it's got the CherryPy module at its core, so
I'm interested to hear more about the problems you had with that.
Did the process die? Did it spin? What exactly did your glue code
look like?
I'm all ears, as I'm sure are the CherryPy folks. :^)
chad
Jacob Kaplan-Moss
> So, I'm now serving simonwillison.net using a stripped down
> mod_python/Apache server (mod_python is pretty much the only module
> loaded) that's sat behind nginx, which is configured to serve up my
> static files and proxy the dynamic requests through to Apache. I'm
> using Apache as a python application server basically because I can
> rely on it staying up. It's been working great so far.
FYI, this is pretty similar to the most stable architecture we've so far found
at World Online. We use perlbal in front of Apache for Django, and a
dedicated lighttpd server/instance for serving static media.
The added bonus of a load balancer is that we can move sites between different
physical machines and/or pool them between multiple machines with almost no
fuss. It's pretty nice.
Shameless plug: I'm going to be covering Django deployment in my "Advanced
Django" tutorial at PyCon...
Jacob
Jeremy Dunck
> Shameless plug: I'm going to be covering Django deployment in my "Advanced
> Django" tutorial at PyCon...
Pluggedy-plug plug plug
http://us.pycon.org/TX2007/TutorialsPM#PM1
Jacob Kaplan-Moss
> http://us.pycon.org/TX2007/TutorialsPM#PM1
Oh, God -- who the hell put *that* picture there? I have got to change that...
Jacob
Jeremy Dunck
Aw, I thought it was cute, what with the smiling pig shirt. ;-)
Peter Ma
--
mawe...@gmail.com
13585201588
Simon Willison
> Simon,
> Well, after that I have to tip my hand. :^)
>
> I've started such a project, called Aspen:
>
> http://www.zetadev.com/software/aspen/
I've got a bit of a philosophical problem with Aspen - the fact that it
supports "five different development patterns". I'm interested in one
and one-only: the ability to serve a WSGI application. All the other
stuff that Aspen does is neat, but could be done instead using WSGI
middleware. If Aspen was architected as a simple
just-serves-up-WSGI-robustly server and an optional set of middleware
for the five different patterns it would be a much more attractive
option to me.
> The catch is that it's got the CherryPy module at its core, so
> I'm interested to hear more about the problems you had with that.
> Did the process die? Did it spin? What exactly did your glue code
> look like?
I wish I could give a more detailed answer, but I was asleep for most
of the time I was running CherryPy! Since my site was throwing "bad
gateway" errors I didn't spend much (any) time trying to figure out
what was wrong - I just threw mod_python/Apache back in there as
quickly as possible. This could well be a case of user error - you
certainly shouldn't be put off running CherryPy through its paces.
The process monitoring thing is a really big issue for me though. I
want tools which have the ability to recover from errors built-in as
much as possible. Process monitoring software is generally a royal pain
to set up - take a look at this tutorial:
http://pylonshq.com/project/pylonshq/wiki/DaemonTools
It's certainly not impossible to do, but if I can avoid all of that
sysadmin work then all the better.
As an aside, I've just started using monit for process monitoring and
it's been by far the least hassle to set up of all the options I looked
at. It's still not perfect but it's a pretty nice tool - I'm using it
to watch over nginx, mysql and apache at the moment:
http://www.tildeslash.com/monit/
I'm genuinely extremely interested in finding a robust pure-Python
application server that I can use for Django stuff - but at the moment
my priority is keeping my site up. Python needs an answer to Mongrel!
Jacob Kaplan-Moss
> I'm genuinely extremely interested in finding a robust pure-Python
> application server that I can use for Django stuff - but at the moment
> my priority is keeping my site up. Python needs an answer to Mongrel!
A number of people have made this same assertion, and I'm a bit curious about
it. Is there a reason that using Apache/mod_python as the app server and
nginx/perlbal as a distributor is somehow less effective than a pure-Python
solution?
My gut tells me that having Apache handle all the transport layer stuff in C
is bound to be faster, and perlbal/mod_python has worked *fabulously* for me.
Of course, more options are always better, but I'm very curious about what
improvements you envision getting out of a pure-Python web stack.
Jacob
Chad Whitacre
> Is there a reason that using Apache/mod_python as the app
> server and nginx/perlbal as a distributor is somehow less
> effective than a pure-Python solution?
Apache/mod_python is heavy. A more nimble server would simplify
production deployment, especially in heterogeneous (multi-site)
environments. A nimble server could also be used for dev, further
simplifying things.
> My gut tells me that having Apache handle all the transport
> layer stuff in C is bound to be faster
Well, yeah. Mongrel is written in C though, so with a true Python
equivalent speed wouldn't be an issue.
chad
Antonio Cavedoni
> Well, yeah. Mongrel is written in C though, so with a true Python
> equivalent speed wouldn't be an issue.
So why not Mongrel itself, if it's written in C?
I must say I've been pretty well served by Apache, myself, so I'm not
sure I see a need.
Also, didn't GvR mention in his Mondrian Google Tech Talk that he's
using the wsgiref library (which should have a lightweight WSGI web
serve if I'm not mistaken)?
Cheers.
--
Antonio
Chad Whitacre
> So why not Mongrel itself, if it's written in C?
Mongrel is for Ruby, although it could certainly be ported I
suppose. Unfortunately that's too far over my head at the moment.
> I must say I've been pretty well served by Apache, myself, so I'm not
> sure I see a need.
Fair enough.
> Also, didn't GvR mention in his Mondrian Google Tech Talk that he's
> using the wsgiref library (which should have a lightweight WSGI web
> serve if I'm not mistaken)?
Yeah, but see the Web-SIG for recent discussion on wsgiref's
limited use in production: Guido needs HTTP/1.1, e.g.
chad
Chad Whitacre
Thanks for the reply.
> I've got a bit of a philosophical problem with Aspen - the fact that it
> supports "five different development patterns". I'm interested in one
> and one-only: the ability to serve a WSGI application. All the other
> stuff that Aspen does is neat, but could be done instead using WSGI
> middleware. If Aspen was architected as a simple
> just-serves-up-WSGI-robustly server and an optional set of middleware
> for the five different patterns it would be a much more attractive
> option to me.
Hmmm, ok. Helpful feedback, thanks. Since I gave you a bad first
impression w/ a screencast, I thought it might take another to
convince you that Aspen isn't really that complicated:
http://tech.whit537.org/2006/12/screencast-django-aspen-stephane.html
Let me know your thoughts.
> The process monitoring thing is a really big issue for me though.
Now that's another story. I use daemontools some, mostly for the
consistent interface it provides (and you're right that it's a
pain to set up). However, the auto-restarting feature strikes me
mostly as a band-aid for buggy daemons. :-)
> I'm genuinely extremely interested in finding a robust pure-Python
> application server that I can use for Django stuff - but at the moment
> my priority is keeping my site up. Python needs an answer to Mongrel!
Others have indeed compared Aspen to Mongrel, although Aspen
isn't written in C (yet?).
chad
Chad Whitacre
> what's the pure-HTTP deployment mean???
Sorry, by pure-HTTP deployment I mean that the Django server
process speaks HTTP instead of FastCGI or mod_pythonese.
chad
Sylvain Hellegouarch
They are indeed.
I'd be interested to know which version of CP Simon was running. CP3 is
way more stable, efficient and comprehensive that CP2 (and its WSGI
server is like ten steps ahead).
- Sylvain
Jeremy Dunck
>
> Let me know your thoughts.
Where does one find this tiny shim?
> > The process monitoring thing is a really big issue for me though.
>
> However, the auto-restarting feature strikes me
> mostly as a band-aid for buggy daemons. :-)
mod_python does a good job of isolating errors on one request from
another request.
I think it's reasonable to expect that from any WSGI server, no?
> > I'm genuinely extremely interested in finding a robust pure-Python
> > application server that I can use for Django stuff - but at the moment
> > my priority is keeping my site up. Python needs an answer to Mongrel!
>
> Others have indeed compared Aspen to Mongrel, although Aspen
> isn't written in C (yet?).
Is it possible for a mis-behaving request handler to take down the
Aspen process? :)
Don Arbow
> Where does one find this tiny shim?
Well, the url is in the screencast:
http://aspen-commons.googlecode.com/svn/stephane/
Don
Waylan Limberg
Generally, in my experience retrieving that information from a
screencast is a pain. If there isn't any supporting documentation, I
won't bother taking the time to extract it all from the screencast.
Aspen's documentation seems rather sparse. If there was some written
documentation giving more details on the info covered in either
screencast, I might look more closely. The way I see it, the
screencast is just to wet our appetite for more. Unfortunately there
isn't more AFAICT.
--
----
Waylan Limberg
way...@gmail.com
Simon Willison
> A number of people have made this same assertion, and I'm a bit curious about
> it. Is there a reason that using Apache/mod_python as the app server and
> nginx/perlbal as a distributor is somehow less effective than a pure-Python
> solution?
I'm interested in this for a couple of reasons. Firstly, it's less bits
of software - if we could tell people "Install Django, install
Reinhardt[1], run a couple of instances of Reinhardt on high ports and
point nginx / perlbal / your load-balancer of choice at it" we'd have a
really nice story for deployment; the Rails guys have this right now
with Mongrel. Talking people through configuring mod_python is one more
step where they can get stuck.
Secondly, a reliable stand-alone Python web server would be a fantastic
thing to have for low traffic applications and intranets. Loads of web
development takes place behind the firewall after all - Guido has
stated that Mondrian runs on a pure Python web server (the wsgiutils
one) and handles a couple of thousand users just fine.
The bigger vision is this: it's clear to me that application servers
configured behind a fast reverse proxy is the Right Way to build
websites. Imagine if you could trivially set up something like this:
example.com/blog/ -> proxies to WordPress, running on PHP somewhere
example.com/drag-n-drop-shopping-cart/ -> proxies to a Rails app
example.com/sudoku/ -> proxies to a Django app
example.com/catwalk/ -> proxies to a TurboGears app
Etc. Sure, you can do this today with some careful setup, but having a
good pure-Python web server that you can just start and then leave
alone would make deploying micro-apps a heck of a lot easier.
Check this out: http://www.hackdiary.com/archives/000099.html . Matt
Biddulph wrote a tiny-app in Camping (a Rails micro-framework), set it
running under Mongrel, pointed a reverse proxy at it and released it to
the world. If he builds something else it can have its own Mongrel.
That's the ease of configuration and setup I'm interested in.
Cheers,
Simon
Simon Willison
> I'm interested in this for a couple of reasons. Firstly, it's less bits
> of software - if we could tell people "Install Django, install
That [1] was meant to be accompanied with a footnote saying that
Reinhardt would be an awesome name for a robust pure-python application
server.
Chad Whitacre
Thanks for the feedback.
>> Well, the url is in the screencast:
>>
>> http://aspen-commons.googlecode.com/svn/stephane/
>
> Generally, in my experience retrieving that information from a
> screencast is a pain.
Yes, sorry. Omitting the link was an oversight on my part.
> If there isn't any supporting documentation, I
> won't bother taking the time to extract it all from the screencast.
I threw Stephane together this morning. It's so tiny I didn't
think to give it it's own project infrastructure (doc, tests,
etc.), especially since I wanted to get it out to you all quick.
> Aspen's documentation seems rather sparse.
Really? Have you seen this:
http://www.zetadev.com/software/aspen/0.6/doc/html/
I guess I'd consider that pretty good documentation for a project
that's only 1 month old. No?
chad
Chad Whitacre
> If Aspen was architected as a simple
> just-serves-up-WSGI-robustly server and an optional set of
> middleware for the five different patterns it would be a much
> more attractive option to me.
As mentioned those five patterns are illustrative, not
definitive. Aspen really is just a WSGI server.
What do you envision such an architecture looking like? Are you
thinking of just a library (that's what CherryPy gives us), or an
actual server? If the latter, what would your site look like on
the filesystem? Would this server be Django-specific, or would it
be general to Python?
chad
Chad Whitacre
> I'm interested in this for a couple of reasons. ...
These are among the reasons I started Aspen.
I think there are three layers to look at here:
1. core HTTP library -- wsgiref, cherrypy.wsgiserver, etc.
2. server -- UI for the library (cli, config files, etc.)
3. web framework -- 'nuff said
Seems to me that the frameworks could/should cooperate at least
on #1, and maybe on #2.
Aspen is really about #2. I tried to design Aspen to be usable
for all kinds of Python web frameworks, whether Python-centric or
filesystem-centric. I'd be willing to talk about implementing the
filesystem-centric parts separately if that might mean a wider
audience for Aspen. But it may also truly be the case that
there's not enough value in cooperating on #2.
chad
Chad Whitacre
> mod_python does a good job of isolating errors on one request from
> another request.
Is that mod_python or is that the prefork MPM?
> I think it's reasonable to expect that from any WSGI server, no?
Oh, probably. :)
> Is it possible for a mis-behaving request handler to take down the
> Aspen process? :)
Oh, probably. :)
Good food for thought, thanks. As mentioned, Aspen wraps
cherrypy.wsgiserver, which uses a pool of threads to handle
requests. Exceptions will only affect a single request, but I
haven't looked at other ways a thread could crash/hang.
chad
Simon Willison
> I'd be interested to know which version of CP Simon was running. CP3 is
> way more stable, efficient and comprehensive that CP2 (and its WSGI
> server is like ten steps ahead).
__version__ = '3.0.0beta2'
I'm really interested as to what kind of failures can be expected from
CherryPy. Two that I were getting were "bad gateway" (actually an nginx
error message, caused through some sort of error in connecting to the
server) and occasionally a CherryPy formatted error message saying
something along the lines of "CherryPy server is stopped".
Chad Whitacre
I've posted documentation for Stephane using Google Code's new
Wiki feature:
http://code.google.com/p/aspen-commons/wiki/Stephane
chad
Jeremy Dunck
>
> > mod_python does a good job of isolating errors on one request from
> > another request.
>
> Is that mod_python or is that the prefork MPM?
I've never had to care. :) A quick poke through mod_python.c shows
mod_py tries to be resilient:
...
resultobject = PyObject_CallMethod(idata->obcallback,
"HandlerDispatch", "O",
request_obj);
/* release the lock and destroy tstate*/
release_interpreter();
if (! resultobject) {
ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, req,
"python_handler: Dispatch() returned nothing.");
return HTTP_INTERNAL_SERVER_ERROR;
}
...
I fear I've wandered off-topic, but I guess my philosophy in writing
server software is that no bug in a handling a single request should
affect the handling in other requests. That and $1 would get me a cup
of coffee, since I have yet to write any HTTP server software. ;-)
Chad Whitacre
> A quick poke through mod_python.c shows mod_py tries to be resilient:
Yeah, cherrypy.wsgiserver provides this level of isolation. There
are still cases where a rogue thread could take down the server,
e.g., with an infinite loop, but I think that's to be expected at
some level.
chad
Chad Whitacre
> error message, caused through some sort of error in connecting to the
> server)
HTTP 502 from nginx probably just means that the backend server
died rather than hung. In which case there would probably have
been a Python core dump, or maybe something in your app's error log.
> occasionally a CherryPy formatted error message saying
> something along the lines of "CherryPy server is stopped".
I'm not sure you'd see anything CherryPy-formatted if you were
only using the wsgiserver.py module.
chad
Chris Lee-Messer
twisted. I seem to remember that zope actually nows uses twisted for
its server, and there is a twisted.web2.wsgi module that I've seen
mentioned elsewhere as a functional wsgi server that's being used in
production for some sites.
It might be possible to put together a functional wsgi web server quite
quickly for django based upon this.
-chris
ps. I do not actually know very much about twisted but have had it on
my list of things to explore for a long time.
Jeremy Dunck
> ps. I do not actually know very much about twisted but have had it on
> my list of things to explore for a long time.
I'm starting a django-related twisted thingie soon. Hopefully I can
report something back.
G.J. Souverein
(note: use the 2nd example on the page the 1st example is outdated !)
You also need Twisted Web2. You can download it here : http://twistedmatrix.com/projects/web2/.
Do something like sys.path.append('the foldername above the folder of your Djangoprojekt ') at the
start of the code of your Twisted-Web2-Django-driver and you are all set.
This works on Windows XP with Python 4 and also with Python 5. It serves up pages faster then Django's
dev server and it's almost as fast as mod-python.
Sylvain Hellegouarch
we expected failures we'd fix them :)
Anyway, if you can have a try at the final CP 3 release. If you still
experiment issues it'd be useful to report them on the CP users list
directly. we could then help you out.
- Sylvain