Memory issues with Apache + mod_wsgi and 60+ process groups

[danjac]

We are running a number of fairly average volume Django sites (approx
20000 uniques a day).

Since moving to mod_wsgi from mod_python a month ago, we have been
experiencing occasional high memory usage (every couple of days or so)
leading to
serious unresponsiveness, which is only resolved by restarting Apache.

During these periods our traffic has been minimal as has been our
database load.

We are running the sites on two load-balanced servers (the database
and
media server are on different machines). There are approx 60 sites in
total, all running on the same codebase but in separate VirtualHosts
and WSGIDaemonProcesses.

I was wondering if the issue could be with the large number of child
processes, as we have so many.
We have tried tinkering with the configuration by adding maximum-
requests and inactivity-timeout (the
traffic varies from site to site a great deal) but the problem is
recurring.

I'd like to be able to eliminate mod_wsgi as the issue by ensuring
that
we have the optimum configuration for our case and we aren't doing
anything stupid.

If Apache/mod_wsgi isn't the problem then the issue is most likely our
own code, but I'd like to
avoid premature optimization.

Here is our Apache configuration:

Apache settings (prefork):

StartServers 8
MinSpareServers 5
MaxSpareServers 30
ServerLimit 30
MaxClients 30
MaxRequestsPerChild 150

and a typical WSGI configuration (60+ of them, identical configuration
except for names):

<VirtualHost *.80>
ServerName site1.com
ServerAlias site1.com
WSGIScriptAlias / /path/to/site1/site.wsgi
WSGIDaemonProcess site1.com threads=10 display-name=wsgi.site1
maximum-requests=3000 deadlock-timeout=30 inactivity-timeout=300
WSGIProcessGroup site1.com
</VirtualHost>

[Graham Dumpleton]

2010/1/22 danjac <danj...@gmail.com>:

> We are running a number of fairly average volume Django sites (approx
> 20000 uniques a day).
>
> Since moving to mod_wsgi from mod_python a month ago, we have been
> experiencing occasional high memory usage (every couple of days or so)
> leading to
> serious unresponsiveness, which is only resolved by restarting Apache.

Have you checked the obvious of ensure that DEBUG is set to False in
Django configuration?

Are you running PHP on the same site? If you aren't can you switch to
worker MPM for Apache. That will at least drop down the number of
Apache server child processes.

You also seem to have MaxRequestsPerChild set version low for where
mod_wsgi daemon mode is used and Apache server child processes
therefore only handling static files and proxying. Even if using PHP,
it wouldn't normally need to be that low and in fact could be 0 so
that processes don't even get recycled based on number of requests.

> and a typical WSGI configuration (60+ of them, identical configuration
> except for names):
>
> <VirtualHost *.80>
>    ServerName site1.com
>    ServerAlias site1.com
>    WSGIScriptAlias / /path/to/site1/site.wsgi
>    WSGIDaemonProcess site1.com threads=10 display-name=wsgi.site1
> maximum-requests=3000 deadlock-timeout=30 inactivity-timeout=300

You could drop inactivity-timeout to 60 seconds if want to be a bit
more aggressive about throwing out idle applications. Really depends a
bit on what application is used for

>  WSGIProcessGroup site1.com
> </VirtualHost>

Other than, can't see anything out of the ordinary that I would be
worried about.

If running on a VPS, you could try dropping down default per thread
stack size as documented in:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Memory_Constrained_VPS_Systems

Not sure this is really a good match for your problem though.

Would suggest perhaps looking at the size of results from database
queries in your application. Maybe there are specific URLs which have
a large transient memory usage for the data set or the resultant
objects from processing the data. Obviously once this memory is
allocated it stays allocated by the process.

Not sure whether you might be able to have a basic memory usage
monitor going using 'ps' and see if you can spot where memory jumps
and align that with specific URL requests occurring. Even if can't get
anything from that, monitoring would show whether memory usage is
incremental or sudden. If incremental, maybe you have a object
reference cycle with multiple __del__ methods and garbage collector
can't break the cycle and reclaim the Python objects and the memory
they use.

Graham

[Dan Jacob]

Thanks for your reply.

2010/1/22 Graham Dumpleton <graham.d...@gmail.com>:

> 2010/1/22 danjac <danj...@gmail.com>:
>> We are running a number of fairly average volume Django sites (approx
>> 20000 uniques a day).
>>
>> Since moving to mod_wsgi from mod_python a month ago, we have been
>> experiencing occasional high memory usage (every couple of days or so)
>> leading to
>> serious unresponsiveness, which is only resolved by restarting Apache.
>
> Have you checked the obvious of ensure that DEBUG is set to False in
> Django configuration?
>

No, all set to DEBUG=False by default. Might be worth double checking though.

No, just mod_wsgi, so that might be an option.

> You also seem to have MaxRequestsPerChild set version low for where
> mod_wsgi daemon mode is used and Apache server child processes
> therefore only handling static files and proxying. Even if using PHP,
> it wouldn't normally need to be that low and in fact could be 0 so
> that processes don't even get recycled based on number of requests.
>

OK.

>> and a typical WSGI configuration (60+ of them, identical configuration
>> except for names):
>>
>> <VirtualHost *.80>
>>    ServerName site1.com
>>    ServerAlias site1.com
>>    WSGIScriptAlias / /path/to/site1/site.wsgi
>>    WSGIDaemonProcess site1.com threads=10 display-name=wsgi.site1
>> maximum-requests=3000 deadlock-timeout=30 inactivity-timeout=300
>
> You could drop inactivity-timeout to 60 seconds if want to be a bit
> more aggressive about throwing out idle applications. Really depends a
> bit on what application is used for
>

There are some sites which are basically admin-only, and so don't see
much action over an average day.

>>  WSGIProcessGroup site1.com
>> </VirtualHost>
>
> Other than, can't see anything out of the ordinary that I would be
> worried about.
>
> If running on a VPS, you could try dropping down default per thread
> stack size as documented in:
>
>  http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Memory_Constrained_VPS_Systems
>
> Not sure this is really a good match for your problem though.
>
> Would suggest perhaps looking at the size of results from database
> queries in your application. Maybe there are specific URLs which have
> a large transient memory usage for the data set or the resultant
> objects from processing the data. Obviously once this memory is
> allocated it stays allocated by the process.
>
> Not sure whether you might be able to have a basic memory usage
> monitor going using 'ps' and see if you can spot where memory jumps
> and align that with specific URL requests occurring. Even if can't get
> anything from that, monitoring would show whether memory usage is
> incremental or sudden. If incremental, maybe you have a object
> reference cycle with multiple __del__ methods and garbage collector
> can't break the cycle and reclaim the Python objects and the memory
> they use.
>
> Graham
>

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>

--
Dan Jacob
Skype: danjac40
Mobile: (++44) (0)777 290 8352

[Graham Dumpleton]

To be doubly sure you are not accidentally running particular
applications in embedded mode instead of daemon mode, add to global
configuration in Apache:

WSGIRestrictEmbedded On

This will effectively prevent you from delegating application to run
in daemon mode, giving an error when it occurs.

This thereby protects you against configuration stuff ups when your
intention is to only run applications in daemon mode.

Graham

2010/1/22 Dan Jacob <danj...@gmail.com>:

[Dan Jacob]

Thanks again.

2010/1/22 Graham Dumpleton <graham.d...@gmail.com>: