CPU and memory issues with apache

[Marin Pranjić]

Not sure if web2py has anything with this. Could be.

First issue is that CPU jumps up  to 100%. I am watching it with htop and it happens every few minutes, 'apache2' process is using it.
Site becomes unresponsive.

I tried to restart apache several times but it didn't help.
However, after server reboot CPU is back to normal and mostly below 2% usage.

Another issue is the memory. After server reboot memory consumption is <25%. After some time it goes up to 85%.
I have a controller that calls subprocess.Popen and sometimes it doesn't work because of "Cannot allocate memory".
This makes the memory consumption an issue.
It seems that only apache memory is growing (still using htop to check it).

I am not an apache expert and not sure what to do. I can provide additional details if needed.
I hope someone can help :)

Marin

[Richard Vézina]

Look like a memory leaks...

Maybe you could profile your app with web2py included profiler...

You just need to add "-F profileroutputfile.txt" at the end of the initialization web2py string

Richard

Marin

--
 
 
 

[Massimo Di Pierro]

What web2py version? What Python version? What apache version? Do you get the leak running with rocket?

[Marin Pranjić]

web2py 2.0.9
python 2.7.2

I tried apache, rocket and uwsgi.

I was stressing the server with many requests to monitor the memory usage.
Memory was slowly going up.
I think it is normal for memory to increase a bit but not sure how much should it go up, and should it decrease when I stop making traffic.

I don't know how to speed up the memory problem. I didn't try other servers much because this happens in production and I depend on apache config (rewrite rules).

After apache restart, memory is ~120Mb and the rest is free.
It goes up to 250Mb very fast and then it stabilizes (i think this is expected).

However this is how it looks like after some time (image attached).

I do not use any sort of caching in web2py. I do not host anything else in apache that might cause it.
I use default apache config with mpm_worker module.
It should create 2 only processes with many threads and it does.

--
 
 
 

[Massimo Di Pierro]

I think you are experiencing what is described in this post:

https://groups.google.com/forum/?fromgroups=#!searchin/web2py/leak/web2py/Go1TUyDm4ys/_2W7qnK8ZMEJ

and it was fixed in 2.1.1.

Massimo

[Niphlod]

one sec. I don't see in that htop 2 processes with "many threads". There are 27 different PIDs there! What is your apache config ?

[Marin Pranjić]

@Massimo, I upgraded to 2.2.1 but memory problems are still there. I will do some more testing.
Memory goes up +100Mb in less then a minute when I try to make a lot of traffic (hitting Ctrl + R )


@Niphlod

This is the config part. I think values are default ones that come with apache.

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

And htop is explained here:
http://serverfault.com/questions/24198/why-does-htop-show-lots-of-apache2-processes-by-ps-aux-doesnt?answertab=votes#tab-top

This is a list of apache modules:

Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgid_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 headers_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 proxy_module (shared)
 proxy_http_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 status_module (shared)
 wsgi_module (shared)

Marin

[Niphlod]

Disclaimer: really not trying to be the devil's advocate here, but it's really unclear to me why with uwsgi I can't see any memory leak (and it's not set to restart itself on high memory usage) with over 2millions requests.

You are confirming that configuring correctly htop you see only 2 apache processes and the rest are all threads ? How is it that they show different PIDs ?
Disclaimer2: No expert on apache here, but I've seen my fair share and I can assume I can read properly the docs (http://httpd.apache.org/docs/2.2/mod/worker.html).

You values means:
Start with 2 processes (StartServer), 25 threads each (ThreadsPerChild).
When there are less than 25 threads available (MinSpareThreads), spawn processes until at least 25 threads on total are free.
Reap a process when there are 75 threads doing nothing (MaxSpareThreads).
You can serve concurrently at most 150 clients (MaxClient), so you can end up with 150/25 processes (6) in normal statuses.
Apache should never recycle processes because of the MaxRequestsPerChild 0 directive.

It's unclear to me that the following behaviour (seems the one happening to you):
"""
if only one thread is terminating on one process serving a response, you may have more than one "terminating processes" (up to 150 (MaxClients)). 
"""
is avoided setting just MaxRequestsPerChild = 0 (like in your config) or setting also MaxSpareThreads = MaxClients.
 
Anyway, you don't have in place the ServerLimit directive that cuts off the maximum number of processes spawned by apache (you should set it to a value going from 6 to whatever you want, but just to try I'd set MaxSpareThreads to 150 and if the same behaviour happens, I'd set ServerLimit to 6)

[Marin Pranjić]

I have 4 apache processes running. One is run by a root and other 3 are owned by www-data.
I will try your suggestion.

Minor update:
hitting examples app does not cause any issues (CPU is low and memory is stable)

welcome app has slow memory increase
- I started with freshly restarted apache, 125Mb of memory used
- Reloading a page for one minute increased memory usage up to 190Mb.
- CPU was hitting the 100% limit but the hosting has a low CPU power available and I'm less worried about CPU

Welcome app is very simple. I have much worse scenario with a real app, as mentioned before.

Not sure what happens after a minute because everything (including shell and htop) becomes unresponsive because of a CPU issue.

I will have to do some more testing. This time on my local computer where I can use much more resources.
I will update this thread with the results.

Thank you for your help

--
 
 
 

[Massimo Di Pierro]

If you create an instance with an object with a __del__ method and a self reference (of if a module that you use creates an instance of such object), Python cannot collect it. Python uses reference counting and it cannot collect objects with self references and a del method. Depending on how big the object is, the memory would pile up. Even our explicit calls to gc does not help. Web2py opjects do have a __del__ method for this reason.