httpd processes use a HUGE amount of swap, even with ulimit -s set to 2MB
Clarke
fix this problem httpd's process end up using ~150MB of swap each over
time:
(this is swap)
161m
httpd
161m
httpd
158m
httpd
150m
httpd
141m
httpd
100m httpd
Obviously, this is a problem. Perhaps I'm not setting ulimit -s
correctly.
ulimit -s 2048
seems pretty simple, run as root .... and I'm also trying to run as
apache's user (daemon)
sudo su -c 'ulimit -s 2048' -s /bin/sh daemon
no love.
How can we fix this?
Hongli Lai
Hi.
Are you sure this is *actual memory usage* and not just VM size?
Passenger creates multiple threads. Because of the nature of threads,
memory usage for multi-threaded applications are usually misreported.
For example, a demo application that creates 100 threads (each thread
doing nothing) appears to be using 800 MB memory, according to ps's
"VSZ" column. But actual memory usage is only 532 KB - a significant
difference!
People usually read the "VM size" or "VSZ" column from process reporting
tools such as "ps" or "top", and it's likely that you've read that as
well. However, this column does not report memory usage accurately.
Unfortunately, "ps" and "top" do not provide facilities to report the
actual memory usage. GNOME System Monitor provides the most accurate
memory usage report, through the "Writable memory" column.
You probably don't have GNOME System Monitor installed. But you can also
find out the actual memory usage of a single process by measuring its
"private dirty RSS" with the following command (this only works on Linux):
grep Private_Dirty /proc/XXXX/smaps | awk '{ print $2 }' | xargs ruby -e
'puts ARGV.inject { |i, j| i.to_i + j.to_i }'
where 'XXXX' is the PID of a process. This reports memory usage in KB.
The best way for you to find out how much memory Apache (i.e. all Apache
processes combined) _really_ uses, is through 'free -m'. I suspect that
if you:
1. start Apache
2. measure the memory usage with 'free -m'
3. stop Apache
4. measure with 'free -m' again, and calculate the difference,
that you will find that the actual memory usage is significantly lower
than what is reported by 'ps'.
Many people have been fooled by this phenomenon, and indeed, I don't
understand why the authors of 'ps' and 'top' haven't updated their tools
to correctly report actual memory usage.
Wikipedia also elaborates this phenomena at
http://en.wikipedia.org/wiki/Virtual_memory.
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
Clarke
that apache doesn't seem to be taking ~800MB of swap space total.
That's good. Those numbers I listed were from the swap column of top,
FYI. The fact remains, however, that with passenger loaded swap says
~150MB and with it not loaded it says ~30MB (free -m confirms this)
so something is going on there. I'd be happy with at least knowing
why, even if its not a huge deal. Thanks for your time.
Clarke
errors start showing up with mod_rails, even with 2MB stack size -- do
you think it is doing that based on numbers pulled from ps or top
outputs and therefore freaked out when it shouldn't be?
Hongli Lai
Hi Clarke.
There may be all kinds of reasons why this is so. But I'll need more
information about your environment.
- How many Apache worker processes were running at the time?
- What are the PID, VM size and private dirty RSS for each Apache worker
process?
- What is the PID, VM size and private dirty RSS of the Apache control
process?
- What are the VM size and private dirty RSS of the spawned Rails processes?
- What are the VM size and private dirty RSS of the "Passenger
SpawnManager", "Passenger FrameworkSpawner" and "Passenger
ApplicationSpawner" processes?
I know this is quite a lot of information that one has to gather. But
it'll really help if you could provide them. :)
Hongli Lai
Could you post the exact error message?
maerzbow
hi hongli!
you commented on my post
http://blog.odeley.com/?p=53
today, where I was talking about some similar memory problems.
don't get me wrong! I really appreciate your work and do not blame you
for any memory leaks. it's really a strange situation that happened to
my server.
ok:
it might be that I have misinterpreted the memory outputs of 'ps', but
nevertheless all my processes I was trying to run (e.g. apache2ctl
stop) ended up in a "failed to allocate memory" at approx. 768MB.
and, the fun thing, this even happened to me after removing the
include for mod_passenger.so and restarting the server (no
SpawnManager was running after that)
unfortunately, for the next 2 weeks, I am in a bad situation to replay
this scenario. but then I will try to investigate what was going on!
cheers,
_________ mercy
Hongli Lai
>
> hi hongli!
>
> you commented on my post
>
> http://blog.odeley.com/?p=53
>
> today, where I was talking about some similar memory problems.
>
> don't get me wrong! I really appreciate your work and do not blame you
> for any memory leaks. it's really a strange situation that happened to
> my server.
>
> ok:
> it might be that I have misinterpreted the memory outputs of 'ps', but
> nevertheless all my processes I was trying to run (e.g. apache2ctl
> stop) ended up in a "failed to allocate memory" at approx. 768MB.
>
> and, the fun thing, this even happened to me after removing the
> include for mod_passenger.so and restarting the server (no
> SpawnManager was running after that)
>
>
> unfortunately, for the next 2 weeks, I am in a bad situation to replay
> this scenario. but then I will try to investigate what was going on!
>
> cheers,
>
> _________ mercy
Hi maerzbow.
No offense taken/intended. :) I just want people to be properly informed
about the truth.
FYI, I've written a memory statistics gathering tool. It's in the latest
development version (i.e. the git repository), as
'misc/memory_stats.rb'. If you have the time, please run this tool to
gather memory statistics. Note that this tool only works on Linux.
With kind regards,
Hongli Lai
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
jorrel
PID PPID Thrds VM Size Private Name
-------------------------------------------------------------------------------
20113 1 8 93.2 MB ? /usr/sbin/apache2 -k start
20404 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
20405 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
20406 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
20407 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
20408 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
### Processes: 6
### Total private dirty RSS: 0.00 MB (?)
PID PPID Thrds VM Size Private Name
-------------------------------------------------------------------------------
20401 20113 2 22.7 MB ? Passenger spawn server
### Processes: 1
### Total private dirty RSS: 0.00 MB (?)
can I ask what those 6 processes are?
I put
RailsMaxPoolSize 2
in my apache2.conf (where I pasted the LoadModule stuff)
are these 6 processes the ones that are managed/limited by
RailsMaxPoolSize or are
they completely different things?
Hongli Lai
> Hi! Using memory_stats.rb, this is my result:
>
> PID PPID Thrds VM Size Private Name
> -------------------------------------------------------------------------------
> 20113 1 8 93.2 MB ? /usr/sbin/apache2 -k start
> 20404 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
> 20405 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
> 20406 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
> 20407 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
> 20408 20113 1 53.2 MB ? /usr/sbin/apache2 -k start
> ### Processes: 6
> ### Total private dirty RSS: 0.00 MB (?)
>
> PID PPID Thrds VM Size Private Name
> -------------------------------------------------------------------------------
> 20401 20113 2 22.7 MB ? Passenger spawn server
> ### Processes: 1
> ### Total private dirty RSS: 0.00 MB (?)
Please run this tool as root, otherwise it can't determine the private
dirty RSS.
> can I ask what those 6 processes are?
Those are the Apache worker processes. They're explained in
http://httpd.apache.org/docs/2.2/mod/prefork.html
They're part of Apache, not Passenger.
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Jorrel
I just reduced the minimum number of those worker processes.
thanks for creating mod_rails!
--
chunky bacon!