runaway apache running on mod_wsgi consuming all swap

[Leo Shklovskii]

I've got a strange problem that I've been trying to figure out over the past
few days with my mod_wsgi apache suddenly taking up 100% of memory and swap, effectively
killing the server.

The setup is:
Ubuntu Jaunty (fully updated)
nginx/0.6.35 serving static and proxying to:
Apache/2.2.11 (Ubuntu)
mod_wsgi
Python 2.6.2
Django 1.1.1

Postgres/PostGIS DB on a different box

This is running on a slicehost 512 slice. The application is fairly simple
and only gets ~200 real hits a day + the usual bot traffic.

The workers config is set to:
<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers       5
    MaxClients           30
    MaxRequestsPerChild 300
    ServerLimit          30
</IfModule>

Several times over the past few days, I'll get an alarm that the time to
serve an http request has spiked (usually its .25 seconds) to 5 or 10
seconds and looking at the munin monitoring reveals that the
swap has been completely filled:

http://img687.imageshack.us/img687/7505/prod2memoryday.png

Jumping on the machine and looking at top shows the apache completely sucking
up all memory. Restarting it fixes the problem but I'd really love to be
able to figure out what's going on.

There's nothing useful or out of the ordinary in syslog, apache_access and
absolutely nothing in apache_error.

Has anyone else seen something like this before? or have suggestions on how
to figure out what the issue is?

My current plan is to try and get a gcore when its in this wedged state and
see if I can figure out something with gdb, but I'm not quite sure how to do
that.

Any thoughts or ideas from those who have more experience with this would be
very much appreciated!

--
--Leo

[Graham Dumpleton]

No time to discuss now. Read:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

In short, don't use embedded/prefork MPM for WSGI application.

Graham

2009/10/31 Leo Shklovskii <leo.shk...@gmail.com>:

[Graham Dumpleton]

For the record, there problem was in their code.

http://groups.google.com/group/django-users/browse_frm/thread/1fb8273d4c5b9526#

Graham

2009/10/31 Graham Dumpleton <graham.d...@gmail.com>: