Daemon processes - am I "getting" this?
David Preece
So, I'm running Django on Apache as MPM prefork in a slice. Not
running with WSGIDaemonProcess causes it to stomp all over the memory.
However, I just added WSGIDaemonProcess and per-project process groups
and can *see* ... here, look
root 14796 0.0 1.8 89352 4908 ? Ss 00:10 0:00 /usr/
sbin/apache2 -k start
davep 14797 0.0 6.3 333936 16552 ? Sl 00:10 0:00 \_ /
usr/sbin/apache2 -k start
davep 14799 0.0 8.6 279352 22576 ? Sl 00:10 0:00 \_ /
usr/sbin/apache2 -k start
www-data 14832 0.0 1.6 89488 4368 ? S 00:10 0:00 \_ /
usr/sbin/apache2 -k start
www-data 14833 0.0 1.6 89488 4368 ? S 00:10 0:00 \_ /
usr/sbin/apache2 -k start
www-data 14839 0.0 1.6 89488 4360 ? S 00:11 0:00 \_ /
usr/sbin/apache2 -k start
www-data 14843 0.0 1.6 89488 4348 ? S 00:12 0:00 \_ /
usr/sbin/apache2 -k start
www-data 14844 0.0 1.6 89488 4356 ? S 00:12 0:00 \_ /
usr/sbin/apache2 -k start
So we have a small django project and a bigger django project. And by
using the defaults (except for the value of user, clearly) I now have
15 threads running within each of these - a little corner of MPM
threaded, as it were. Does this imply that the forked apache process
itself is now running 15 threads? Can handle 15 concurrent
connections? Keep-alive's? Have I, with the exception of the threads
that get stuck in the GIL, more or less spawned 15 apache processes
for the price of one? If so, this is great.
BTW, I did read a lot of the various disclaimers about django being
probably thread safe, but no more than probably.
Also, can I make a suggestion? Would it be possible to change the
displayed name of the forked process to represent the
WSGIDaemonProcess it represents - much like sshd and postgres can?
Would make life even clearer for me and, I suspect, a thing of rare
beauty for the mass hosting crowd....
Dave
Graham Dumpleton
The normal Apache child process are still prefork mode and are single
threaded. It is only the mod_wsgi daemon mode processes that are
multithreaded.
> Can handle 15 concurrent
> connections?
Because Apache child processes still run in prefork mode and are
single threaded, if more concurrent requests come in than Apache child
processes then Apache will start up extras to meet demand. This will
be the case even if those requests end up then being proxied through
to the mod_wsgi daemon process. Thus, If 15 concurrent requests came
in for same WSGI application running in daemon mode, you would end up
with 15 Apache child processes accepting those requests with each then
proxying through to the same mod_wsgi daemon process that that
application runs in.
If you don't have a requirement for using prefork, ie., don't need to
run PHP, then some further benefit can be gained by running with
Apache worker MPM as then cut down on the number of Apache child
processes as well and thus memory.
> Keep-alive's? Have I, with the exception of the threads
> that get stuck in the GIL, more or less spawned 15 apache processes
> for the price of one? If so, this is great.
Don't be overly concerned about the GIL. The fact that Apache is multi
process and that there is a lot going on which is not Python code
means that using up those multiples cores and/or processes is not
usually a problem. For my prior commentary about the GIL see:
http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html
> BTW, I did read a lot of the various disclaimers about django being
> probably thread safe, but no more than probably.
>
> Also, can I make a suggestion? Would it be possible to change the
> displayed name of the forked process to represent the
> WSGIDaemonProcess it represents - much like sshd and postgres can?
> Would make life even clearer for me and, I suspect, a thing of rare
> beauty for the mass hosting crowd....
I don't know of any standard way of doing it. If one does a fork/exec
one can change the name of the process at the point of the exec call,
but the mod_wsgi daemon processes are a straight fork of the main
Apache parent process and not exec is involved.
This page:
http://www.uofr.net/~greg/processname.html
Since gone dead in the few minutes since I checked it, explains a hack
for doing it on Linux but would seem to be Linux specific and doesn't
give you ability to change name displayed to arbitrary value as bound
by length of existing argv0 value.
If you know a way of doing it which doesn't involve an exec, would
love to hear about it. I somewhat doubt there is a standard way of
doing it though as it would seem a bit like a back door for an
application to hide its true identity.
Graham
da...@zedkep.com
to clients, and then proxies the calls over to the the WSGI daemon
process which in this case is actually running the django app ... on
multiple threads. It might say "apache" but it's not actually serving
port 80. More memory footprint saving could therefore be had by
switching to the worker MPM which, as best I can tell, is
traditionally shunned because of the non-threadsafe nature of third
party modules (PHP?) and because memory is "practically free". But if
I'm not running any (additional) mods other than wsgi then I should be
good to go.
Yeah, here we go, simply a question of typing "apt-get install apache2-
mpm-worker" and a few more tens of megs are saved - actually a big
deal if one is using a VPS slice.
Presumably this approach differs from FastCGI in that the daemon
processes are multi-threaded?
So, really, mod_wsgi is a lot more than a modern replacement for
mod_python. It's more some cunning middleware for gluing wsgi
applications onto Apache (because web serving is solved now, mmmkay)
and providing some dead easy scalability solutions. Holy crap. I feel
a jolly afternoon playing around and benchmarking is probably called
for. After that I feel my calling to start evangelising this to the
wider django community (not that I'm like big and important or
anything) because it's not a particularly simple framework to deploy
and this is *way* the best way to do it.
Dave
Graham Dumpleton
>
> Oh I *see*, so apache serves the "front end" - ie actually connecting
> to clients, and then proxies the calls over to the the WSGI daemon
> process which in this case is actually running the django app ... on
> multiple threads.
Yes.
Have a look at thread:
http://groups.google.com/group/modwsgi/browse_frm/thread/17f9659b3c50bf27
My last post in that has some PDFs attached which give some pictures
of how embedded, daemon, mod_proxy and fastcgi are different. I still
need to add these as part of proper documentation. :-)
> It might say "apache" but it's not actually serving
> port 80.
Not daemon processes no. They use a UNIX socket for receiving proxied
request from Apache child processes. Those diagrams in that thread
explain it.
> More memory footprint saving could therefore be had by
> switching to the worker MPM which, as best I can tell, is
> traditionally shunned because of the non-threadsafe nature of third
> party modules (PHP?) and because memory is "practically free". But if
> I'm not running any (additional) mods other than wsgi then I should be
> good to go.
>
> Yeah, here we go, simply a question of typing "apt-get install apache2-
> mpm-worker" and a few more tens of megs are saved - actually a big
> deal if one is using a VPS slice.
>
> Presumably this approach differs from FastCGI in that the daemon
> processes are multi-threaded?
It isn't too much different from FastCGI except that in FastCGI the
daemon is actually a fork/exec'd process. Ie., it runs a foreign
process such as Python which inside uses flup to talk FastCGI. In
mod_wsgi the daemon is a fork only and as such can control better
interaction with Apache that FastCGI. For example, in FastCGI the web
application could add its own signal handlers and screw up Apache
being able to shutdown the process properly. In mod_wsgi it prevents
application from doing things like that.
> So, really, mod_wsgi is a lot more than a modern replacement for
> mod_python. It's more some cunning middleware for gluing wsgi
> applications onto Apache (because web serving is solved now, mmmkay)
> and providing some dead easy scalability solutions.
The goal is to have one package that can with very very little
configuration change and with no need to change the WSGI script file,
be able to handled embedded for speed, daemon for isolation, and maybe
down the track a third mode I'll call exec mode or as that thread
calls the devilish plan. This would do a fork/exec for daemon but not
of arbitrary program but of Python only, with Python being told to
load up special Python mod_wsgi module to do daemon side stuff. The
idea of this last mode would be allow one to use different versions of
Python rather than be bound to the one that mod_wsgi was specially
compiled for. May be of benefit in ISPs where users want to use
different versions of Python.
> Holy crap. I feel
> a jolly afternoon playing around and benchmarking is probably called
> for.
You might look at:
http://code.google.com/p/modwsgi/wiki/PerformanceEstimates
Remember that the real bottlenecks are always in the application and
any database access it does.
Graham