Apache and memory starved VPS.
Graham Dumpleton
thought I might post a link to a discussion happening on Apache
developers list at the moment.
http://marc.info/?l=apache-httpd-dev&m=119296272027150&w=2
What this is about is how a Linux VPS can show absolutely huge amounts
of virtual memory being used. This appears to be significant as from
what I gather some VPS hosters somehow calculate the virtual memory
into your limit, rather than just looking at actual resident memory.
Result is that running prefork MPM with lots of processes, or even
worker MPM with few processes but lots of threads, can cause these
limits to be reached.
What this discussion is fingering as the culprit here is that on
Linux, if no action is taken, the global stack resource limit for a
user is applied to all threads that are created. By default this would
be 8MB per thread. Thus, multiply that by the number of threads and
that is a lot of virtual memory.
Suggestions given to reduce this are to use 'ulimit -s' to adjust the
stack size from Apache startup script, or if using Apache 2.2, then
perhaps use ThreadStackSize directive. Obviously, if fiddling with
these values, you would want to make sure that your web application
then runs okay for full range of requests to ensure you aren't
starving it of stack memory.
Anyway, could be an interesting area to experiment with. Do note that
at present using ThreadStackSize will only affect mod_wsgi embedded
mode and will not work for mod_wsgi daemon mode. Am going to look at
changing that though.
If anyone is using a VPS, will be interested to get some feedback if
you try changing the stack size as described.
Graham
Graham Dumpleton
WSGIDaemonProcess to allow per thread stack size to be customised on
systems where the default is too big. For example, to set per thread
stack size to 512KB, use:
WSGIDaemonProcess example processes=2 threads=25 stack-size=524288
This will counter the default per thread stack size on Linux boxes of 8MB.
Interestingly, on Mac OS X, the main application thread uses 8MB and
any subsequent threads created use 512KB as a default. In other words,
Mac OS X is much more sensible about its default values.
Note that this only applies to mod_wsgi daemon processes. If using
embedded mode and Apache 2.2, you should use Apache ThreadStackSize
directive. If using Apache 1.3/2.0, you would need to add 'ulimit -s'
command with appropriate value in the Apache 'envvars' file in same
directory as Apache httpd executable. If you don't use 'stack-size'
option, it will inherit the value from main Apache processes set by
one of these methods, although the ThreadStackSize is only inherited
if using mod_wsgi 2.0.
For more information on changes in mod_wsgi 2.0 see:
http://code.google.com/p/modwsgi/wiki/ChangesInVersion0200
Graham