worker MPM + mod_wsgi questions
Andy
- I've seen blog posts recommending using worker MPM with mod_wsgi.
How does Python's GIL affect this?
Let's say I have a quad core machine. Do i need to make sure Apache
spawns at least 4 processes to take advantage of all 4 cores (assuming
I'm using the mod_wsgi embedded mode)?
For example, if I set:
MaxClients 100
ThreadsPerChild 50
In this case a max of only 2 processes are created - does it mean my
django app wouldn't be able to take advantage of the 3rd and 4th cores
because of Python's GIL?
Like wise, if using daemon mode, do I need to set
WSGIDaemonProcess example.com processes=4
to use all 4 cores?
- Between embedded mode & daemon mode, which uses less memory?
Assuming worker MPM is used.
I'll be using either shared hosting or VPS, so reducing memory use is
very important
- With mod_python, it's recommended to put a reverse proxy (eg. nginx)
in front of the fat Apache & serve static content from the reverse
proxy, does the same recommendation still applies to mod_wsgi?
If daemon mode is used, will the front end Apache process act as an
effective reverse proxy?
- What about mod_wsgi for nginx - how does that compared to Apache's
mod_wsgi? Would it be less memory intensive?
Thanks.
Graham Dumpleton
>
> A few questions about mod_wsgi:
>
> - I've seen blog posts recommending using worker MPM with mod_wsgi.
> How does Python's GIL affect this?
> Let's say I have a quad core machine. Do i need to make sure Apache
> spawns at least 4 processes to take advantage of all 4 cores (assuming
> I'm using the mod_wsgi embedded mode)?
> For example, if I set:
> MaxClients 100
> ThreadsPerChild 50
> In this case a max of only 2 processes are created - does it mean my
> django app wouldn't be able to take advantage of the 3rd and 4th cores
> because of Python's GIL?
> Like wise, if using daemon mode, do I need to set
> WSGIDaemonProcess example.com processes=4
> to use all 4 cores?
I wouldn't be overly concerned about the GIL and try to guess what
configuration may be better than another. I have talked about this a
bit before in:
http://blog.dscpl.com.au/2007/07/web-hosting-landscape-and-modwsgi.html
In that I said:
"""In addition to the low overhead, there are also other positive
benefits deriving from how Apache works when using this mode. The
first is that Apache uses multiple child processes to handle requests.
As a result, any contention for the Python GIL within the context of a
single process is not an issue, as each process will be independent.
Thus there is no impediment when using multi processor systems.
That said, the GIL is not as big a deal as some people make out, even
when using Apache with only one multi-threaded child process for
accepting requests. This is because the code which handles accepting
of requests, determines which Apache handler should process the
request, along with the code for reading the request content and
writing out the response content, is all written in C and is in no way
linked to Python. As a consequence there are large sections of code
where the GIL is not being held. On top of that, the same web server
may also be serving up static files where again the GIL doesn't even
come into the picture. So, more than enough opportunity for making
good use of those multiple processors."""
I was actually talking about embedded mode there, but just as
pertinent to daemon mode. In particular, in daemon mode the Apache
server child processes are still doing work at the same time as they
are doing the proxying of the request to the daemon mode process. So,
you are going to have multiple processes trying to do stuff anyway.
There isn't much point trying to match number of daemon processes to
number of cores purely based on concerns about the GIL.
This doesn't mean you shouldn't try and tune the Apache MPM settings
and daemon mode settings to see what works best, but to do that you
really need to have your actual application running and be hitting it
with realistic traffic patterns. That is, no point just using 'ab' at
maximum throttle against a single URL as in practice your site is
never going to be pushed to the maximum. If you are running out of
grunt even for typical traffic, then you need to upgrade your system
to give it more headroom to deal with real spikes in traffic.
> - Between embedded mode & daemon mode, which uses less memory?
Worker MPM with daemon mode. See the dangers of using prefork and
especially embedded mode in:
http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
You sacrifice Apache's ability to create additional processes to
handle demand, but frankly for fat Python web applications that is
arguably a stupid feature as it compounds problems. Namely, just when
you start to get a spike in requests, Apache tries to create more of
your fat applications, which just load the machine even more and slow
things down. In worst case the slowing down makes Apache thinks it
needs even more processes and it can spiral out of control and choke
your whole server. So, for fat Python web applications you are just
better off using multithreading and making sure you have enough
processes/threads to handle expected demand to begin with.
> Assuming worker MPM is used.
> I'll be using either shared hosting or VPS, so reducing memory use is
> very important
>
> - With mod_python, it's recommended to put a reverse proxy (eg. nginx)
> in front of the fat Apache & serve static content from the reverse
> proxy, does the same recommendation still applies to mod_wsgi?
Yes. And turn keep alive off on Apache to ensure connections released
straight away. Keep alive is generally only effective for static media
files which nginx would then be handling. By disabling keep alive you
get better utilisation of available connections and lower memory usage
in Apache server child processes.
> If daemon mode is used, will the front end Apache process act as an
> effective reverse proxy?
Apache is effectively acting as an internal proxy for the daemon
processes. Even so, you are still better off pushing static media to a
nginx in front of that. The overhead of the extra internal hop within
Apache to get to the daemon processes is so small you would never see
it within context of typical request times for Python web
applications.
> - What about mod_wsgi for nginx - how does that compared to Apache's
> mod_wsgi? Would it be less memory intensive?
I can't really comment on that except to say that it doesn't matter
what WSGI hosting mechanism you use, be it Apache, nginx or a pure
Python web server such as Paste serve of CherryPy WSGI server. For
each process running your Python web application, each system is still
going to use about the same amount of memory. This is because the
underlying Python interpreter memory usage should always be the same
and your Python web applications is also going to always use about the
same amount of memory as well. Any small differences that there may be
would relate to how the web server aspect of the system uses memory
differently, but the differences aren't generally going to be
significant as long as you set the servers up properly. No particular
system provides some magic bullet that somehow nullifies how much
memory your actual Python web application uses and that is where the
bulk of your memory will be used.
Graham
Michael Schurter
OT, sorry
Thanks for the great explanations Graham. My talk proposal, Web
Server Shootout[1], for the Open Source Bridge Conference[2] was
accepted, so you may see me popping in with more questions similar to
the OPs. :-)
Would love to see you there! You'd definitely get a beer on me!
[1] http://opensourcebridge.org/proposals/119
[2] http://opensourcebridge.org/
Graham Dumpleton
It is in a foreign land over the big blue sea and a long way away from
where I am. So, I somewhat doubt you will see me there. :-)
Graham
Andy
addition to the Python code. My concern though is the amount of
concurrency allowed within Python.
Say I set up a Django site with mod_wsgi in daemon mode using 1
process & 30 threads. If a Python thread makes a database call or a
memcached call and blocks, can any other python threads run? Or would
they all block because they'll all be waiting for the GIL which the
thread that made the DB call never released?
I guess what I'm really trying to understand is - is a multithreaded
Django daemon process really possible given the Python GIL and all
those blocking DB calls?
By the way, does mod_wsgi work with event MPM? If yes, any particular
reasons you recommended using worker MPM instead of event MPM?
Thanks.
On Apr 17, 7:41 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/4/17 Andy <selforgani...@gmail.com>:
Aaron Watters
On Apr 17, 7:41 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/4/17 Andy <selforgani...@gmail.com>:
>
Yes, I'd suspect that in almost all real cases
the bottleneck will be disk I/O and database
access (which are also unaffected by the GIL).
These things are awfully tricky to philosophize
about from first principles -- you need to find
a slow application and run "top" etc to find out
what the real problem is. I'd even go so far as
to guess that the hypothesis about mod-python
may have been based on guesswork.
-- Aaron Watters
http://aaron.oirt.rutgers.edu/myapp/docs/W1500.whyIsWhiffCool
===
She was dirty, flirty, musta been about thirty
-- Stones, 60's
She was shifty, nifty, musta been about fifty
-- Stones, 80's
Ron Rothman
I'm a long-time lurker and happy mod_wsgi user. Saw this thread and
had to post.
Graham, I've noticed that you usually very wisely avoid making
assumptions about the particular environments in which people's
mod_wsgi instances are running. But here you have made several
assumptions; in particular, several which are not true in our
environment, so I wanted to point them out.
[Background: we're running mod_wsgi, embedded in Apache 2.2, worker
MPM. Machines in our server tier vary from 2-way to 8-way cpu. We
hacked--er, I mean customized--the mod_wsgi source code (because we
needed certain features) but for the most part it's a vanilla 3.0
installation. This tier currently handles on the order of 200 million
requests per day.]
1. Unfortunately, the GIL *is* a big deal, at least in our case. Our
application is completely CPU bound, and when we tested performance
with multiple threads, we found no increase in throughput (literally
0% increase) and an increase in latency that corresponded directly to
the number of threads per child. It was the GIL. Because we're so
cpu bound, the GIL effectively killed any hope we had of running multi-
threaded. (In our case, it turned out okay. We spent some time
optimizing for a smaller memory footprint and now we're running shared-
nothing: single-threaded, multiple processes.)
2. No, our servers do not serve up any static content (like images);
they are dedicated to serving requests from our application. Keep in
mind that not all mod_wsgi installations are in a shared environment.
3. Turn keepalives off for better performance? In some cases, maybe,
but in our case it would have highly negative consequences. That's
because our servers are behind a load balancer which maintains a pool
of open backend connections so that it can spare the expense of
unnecessary connection set-up/tear-down. My point is: you can't
advise to turn keepalives off without knowing certain important
details of the environment.
Just thought I'd share our experience with you. I know you know all
this--which is why it surprised me when you apparently gave the advice
without any mention of the underlying assumptions.
Best,
- ron
> single process is not an issue, as each process will be independent.
> Thus there is no impediment when using multi processor systems.
>
> That said, the GIL is not as big a deal as some people make out, even
> when using Apache with only one multi-threaded child process for
> come into the picture. So, more than enough opportunity for making
> good use of those multiple processors."""
>
Graham Dumpleton
> You made a good point that the C code in Apache will be running in
> addition to the Python code. My concern though is the amount of
> concurrency allowed within Python.
>
> Say I set up a Django site with mod_wsgi in daemon mode using 1
> process & 30 threads. If a Python thread makes a database call or a
> memcached call and blocks, can any other python threads run? Or would
> they all block because they'll all be waiting for the GIL which the
> thread that made the DB call never released?
extension module is meant to release the GIL before calling a
potentially blocking operation, or even a potentially time consuming
operation in foreign language, where no access is made to Python data
structures. If C extension modules do not do this, then they are
breaking the conventions regarding the GIL and could be argued to be
buggy.
> I guess what I'm really trying to understand is - is a multithreaded
> Django daemon process really possible given the Python GIL and all
> those blocking DB calls?
> By the way, does mod_wsgi work with event MPM? If yes, any particular
> reasons you recommended using worker MPM instead of event MPM?
tried it. It has been progressed to a supported MPM for Apache 2.4, so
I will need to look at it at one point. I believe though I have seen
people say they were already using it with mod_wsgi. So, even though
hasn't been tested, does supposedly work.
Graham
Graham Dumpleton
>
>
> On Apr 17, 7:41 am, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:
>> 2009/4/17 Andy <selforgani...@gmail.com>:
>>
>> That said, the GIL is not as big a deal as some people make out...
>
> Yes, I'd suspect that in almost all real cases
> the bottleneck will be disk I/O and database
> access (which are also unaffected by the GIL).
> These things are awfully tricky to philosophize
> about from first principles -- you need to find
> a slow application and run "top" etc to find out
> what the real problem is. I'd even go so far as
> to guess that the hypothesis about mod-python
> may have been based on guesswork.
http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
If that is what you mean then yes, I haven't experienced it first
hand, but logically knowing how Apache works and applying common sense
about characteristics of fat Python applications, it is a good as
hypothesis as any at the moment. :-)
Graham
Graham Dumpleton
> Greetings,
>
> I'm a long-time lurker and happy mod_wsgi user. Saw this thread and
> had to post.
>
> Graham, I've noticed that you usually very wisely avoid making
> assumptions about the particular environments in which people's
> mod_wsgi instances are running. But here you have made several
> assumptions; in particular, several which are not true in our
> environment, so I wanted to point them out.
"""I'll be using either shared hosting or VPS, so reducing memory use
The mention of shared hosting and reducing memory usage here rings
alarm bells that more than likely this is a case of someone trying to
prematurely optimise things, perhaps even before the application is
complete and before they know what sort of overheads exist in regard
memory use, application code, database etc.
In these cases I will always now steer them to the most conservative
configuration in respect of memory use, but also one where they are
least likely to hang themselves. Thus, am going to steer them well
clear of embedded mode and in particular prefork MPM because of the
potential problems which can arise from embedded mode if you do not
know how to configure Apache MPM settings.
That they would even be considering use of shared hosting does sort of
says that it is likely not even an web application that is likely to
attract much traffic. If they really thought it was going to get a lot
of traffic, they should be considering more powerful VPS systems or
dedicated hosting.
Now, I may well have been short on my replies this time and not gone
into as much detail as I would normally, but my patience was probably
wearing a bit thin at the time due to the ongoing parallel thread over
what the best number of processes/threads would be when using daemon
mode. There are some things I just can't give exact guidance on as I
don't know anything about the applications being run. I try by best,
but generalisations is about all one can do. A person really can only
work it out for themselves through proper testing and benchmarking. It
therefore gets a bit frustrating when they keep coming back again and
again expecting some answer.
> [Background: we're running mod_wsgi, embedded in Apache 2.2, worker
> MPM. Machines in our server tier vary from 2-way to 8-way cpu. We
> hacked--er, I mean customized--the mod_wsgi source code (because we
> needed certain features) but for the most part it's a vanilla 3.0
> installation. This tier currently handles on the order of 200 million
> requests per day.]
mod_wsgi. Are they the sort of things which may be useful to others? I
know some people want to allow preloading modules in Apache parent as
an example, but maybe there are other things I haven't thought of or
have put aside at least for the time being.
> 1. Unfortunately, the GIL *is* a big deal, at least in our case. Our
> application is completely CPU bound, and when we tested performance
> with multiple threads, we found no increase in throughput (literally
> 0% increase) and an increase in latency that corresponded directly to
> the number of threads per child. It was the GIL. Because we're so
> cpu bound, the GIL effectively killed any hope we had of running multi-
> threaded. (In our case, it turned out okay. We spent some time
> optimizing for a smaller memory footprint and now we're running shared-
> nothing: single-threaded, multiple processes.)
of use of GIL. That sort of situation is not one that most people are
likely to encounter.
I do at times mention CPU bound applications as changing things in
relation to the GIL. At this point in this case because their focus
was on memory usage, it didn't seem to be the more important issue.
After all, ensuring very low memory usage generally means sacrificing
performance.
> 2. No, our servers do not serve up any static content (like images);
> they are dedicated to serving requests from our application. Keep in
> mind that not all mod_wsgi installations are in a shared environment.
get away with as little memory in the VPS as possible as they don't
want to pay much money. In those cases I put emphasis on helping them
to lower memory usage, especially since the amount of traffic is
probably never going to be anything near what they may hope.
> 3. Turn keepalives off for better performance? In some cases, maybe,
> but in our case it would have highly negative consequences. That's
> because our servers are behind a load balancer which maintains a pool
> of open backend connections so that it can spare the expense of
> unnecessary connection set-up/tear-down. My point is: you can't
> advise to turn keepalives off without knowing certain important
> details of the environment.
so that you could reduce MaxClients for Apache so that not as many
processes/threads required in Apache server child worker processes and
so less base memory used.
> Just thought I'd share our experience with you. I know you know all
> this--which is why it surprised me when you apparently gave the advice
> without any mention of the underlying assumptions.
Overall though your points are quite valid and things aren't as simple
as sometimes I make them seem.
Graham
Andy
Really appreciate your detailed explanation.
On Apr 18, 11:51 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/4/18 Andy <selforgani...@gmail.com>:
Aaron Watters
On Apr 18, 2:08 pm, Ron Rothman <ron.roth...@gmail.com> wrote:
> 1. Unfortunately, the GIL *is* a big deal, at least in our case.
is an FUD excuse for switching to some other technology.
I'm happy to see you found a way around your difficulty.
If you are feeling cruel, ask an 8 year old if he
can count to 1000. Then ask him to prove it.
Unfortunately, the really smart ones may say
"1 2 skip a few a thousand" -- THIS DOES
NOT JUSTIFY A VIOLENT RESPONSE.
Ron Rothman
You asked about the modifications we made to mod_wsgi. I don't want
to hijack this thread, but I'll reply briefly so you have some idea
what we did.
Without getting too specific about our application: we needed to
modify the import script mechanism so that scripts could be run either
at module init (in the parent) or at child init. Just added a run-
at=parent|child|both flag to the WSGIImportScript directive.
While we were at it, we also added a call to pthread_atfork to ensure
that the mod_wsgi mutexes (wsgi_interp_lock, wsgi_module_lock) and the
GIL were not held across forks. I know you already call PyOS_AfterFork
(), which I believe does some thread state (and GIL?) cleanup, but our
import script kicks off a new thread and we wanted to be sure that no
locks were held by mod_wsgi on behalf of that thread whenever apache
decided to spawn a new child.
(Btw, we seriously considered sharing our modifications and opening up
discussions on how to best accomplish our goals, but (a) we were under
time pressure and had to get something live quickly, and (b) the code
we wrote is not pretty (see (a)) so we'd want to spend some time
cleaning it up before sharing it.)
Hope that helps clarify. Feel free to contact me if you'd like to
discuss further. Thanks again,
- ron
On Apr 19, 12:34 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/4/19 Ron Rothman <ron.roth...@gmail.com>: