Hey Grant, All,
We've been running a live event with about 1,000 people and getting hit with up to hundreds of requests per second. I'm running 20 processes and 20 threads per process.
Every once in a while the memory across all processes spikes up to 200+MB and the load average skyrockets. I've seen it hit as high as 250 (vs. 0.7 normal)
service httpd graceful
fixes the issue (for a while)
Normal Example:
[deploy@daas7 DaaS-TMT-0]$ ps aux | grep 'apache' | grep 'TMT-0' | awk '{print $6/1024 " MB " $11}'; cat /proc/meminfo | grep -E '(MemFree|Avail)'; uptime
123.969 MB wsgi-DaaS-TMT-0
123.984 MB wsgi-DaaS-TMT-0
119.52 MB wsgi-DaaS-TMT-0
126.121 MB wsgi-DaaS-TMT-0
121.086 MB wsgi-DaaS-TMT-0
121.016 MB wsgi-DaaS-TMT-0
145.945 MB wsgi-DaaS-TMT-0
118.406 MB wsgi-DaaS-TMT-0
126.672 MB wsgi-DaaS-TMT-0
112.234 MB wsgi-DaaS-TMT-0
111.328 MB wsgi-DaaS-TMT-0
135.461 MB wsgi-DaaS-TMT-0
117.73 MB wsgi-DaaS-TMT-0
136.438 MB wsgi-DaaS-TMT-0
113.359 MB wsgi-DaaS-TMT-0
118.289 MB wsgi-DaaS-TMT-0
123.535 MB wsgi-DaaS-TMT-0
126.746 MB wsgi-DaaS-TMT-0
122.766 MB wsgi-DaaS-TMT-0
115.934 MB wsgi-DaaS-TMT-0
MemFree: 4993068 kB
MemAvailable: 25089688 kB
13:01:36 up 7 days, 9:27, 4 users, load average: 0.55, 0.82, 2.46
Server almost unresponsive:
[deploy@daas7 DaaS-TMT-0]$ ps aux | grep 'apache' | grep 'TMT-0' | awk '{print $6/1024 " MB " $11}'; cat /proc/meminfo | grep -E '(MemFree|Avail)'; uptime
275.457 MB wsgi-DaaS-TMT-0
277.633 MB wsgi-DaaS-TMT-0
274.633 MB wsgi-DaaS-TMT-0
285.215 MB wsgi-DaaS-TMT-0
278.156 MB wsgi-DaaS-TMT-0
272.445 MB wsgi-DaaS-TMT-0
277.543 MB wsgi-DaaS-TMT-0
274.371 MB wsgi-DaaS-TMT-0
277.699 MB wsgi-DaaS-TMT-0
273.18 MB wsgi-DaaS-TMT-0
273.363 MB wsgi-DaaS-TMT-0
278.094 MB wsgi-DaaS-TMT-0
276.719 MB wsgi-DaaS-TMT-0
277.074 MB wsgi-DaaS-TMT-0
274.324 MB wsgi-DaaS-TMT-0
275.32 MB wsgi-DaaS-TMT-0
273.684 MB wsgi-DaaS-TMT-0
271.797 MB wsgi-DaaS-TMT-0
283.133 MB wsgi-DaaS-TMT-0
255.16 MB wsgi-DaaS-TMT-0
28.8008 MB /usr/bin/convert
MemFree: 262352 kB
MemAvailable: 18945328 kB
13:18:50 up 7 days, 9:44, 4 users, load average: 253.79, 100.74, 40.20
After httpd graceful after a couple of minutes:
[deploy@daas7 DaaS-TMT-0]$ ~/stats.sh
100.383 MB wsgi-DaaS-TMT-0
110.719 MB wsgi-DaaS-TMT-0
101.176 MB wsgi-DaaS-TMT-0
128.449 MB wsgi-DaaS-TMT-0
112.527 MB wsgi-DaaS-TMT-0
109.465 MB wsgi-DaaS-TMT-0
103.875 MB wsgi-DaaS-TMT-0
98.8438 MB wsgi-DaaS-TMT-0
108.414 MB wsgi-DaaS-TMT-0
108.133 MB wsgi-DaaS-TMT-0
107.07 MB wsgi-DaaS-TMT-0
118.824 MB wsgi-DaaS-TMT-0
101.527 MB wsgi-DaaS-TMT-0
127.004 MB wsgi-DaaS-TMT-0
100.871 MB wsgi-DaaS-TMT-0
125.188 MB wsgi-DaaS-TMT-0
100.566 MB wsgi-DaaS-TMT-0
108.91 MB wsgi-DaaS-TMT-0
101.215 MB wsgi-DaaS-TMT-0
109.711 MB wsgi-DaaS-TMT-0
MemFree: 7607044 kB
MemAvailable: 25815540 kB
13:25:51 up 7 days, 9:51, 4 users, load average: 1.25, 38.56, 36.12
My main question is does anyone have any suggestions for seeing inside the daemon processes down to the python object level to see what is going on?
Thanks,
Jason