runaway apache takes up all of swap

[Leo Shklovskii]

I've got a strange problem that I've been trying to figure out over the past few days with apache suddenly taking up 100% of memory and swap, effectively killing the server.

The setup is:
Ubuntu Jaunty (fully updated)
nginx/0.6.35 serving static and proxying to:
Apache/2.2.11 (Ubuntu)
mod_wsgi
Python 2.6.2
Django 1.1

Postgres/PostGIS DB on a different box

This is running on a slicehost 512 slice. The application is fairly simple and only gets ~200 real hits a day + the usual bot traffic.

The workers config is set to:
<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers       5
    MaxClients           30
    MaxRequestsPerChild 300
    ServerLimit          30
</IfModule>

Several times over the past few days, I'll get an alarm that the time to serve an http request has spiked (usually its .25 seconds) to 5 or 10 seconds and looking at the munin monitoring immediately reveals that the swap has been completely filled:

http://img687.imageshack.us/img687/7505/prod2memoryday.png

Jumping on the machine and looking at top show the apache completely sucking up all memory. Restarting it fixes the problem but I'd really love to be able to figure out what's going on.

There's nothing useful or out of the ordinary in syslog, apache_access and absolutely nothing in apache_error.

Has anyone else seen something like this before? or have suggestions on how to figure out what the issue is?

My current plan is to try and get a gcore when its in this wedged state and see if I can figure out something with gdb, but I'm not quite sure how to do that.

Any thoughts or ideas from those who have more experience with this would be very much appreciated!

--
--Leo

[Leo Shklovskii]

This problem turned out to be an application issue. I am using GeoDjango and calling buffer() with a negative value which caused problems on certain polygons for the underlying GEOS library. Thanks to everyone who responded off list.

--
--Leo