Running out of memory, Apache locks up.

[Jason Hines]

I'm on a VPS (debian linux) with 1GB memory with 2 Rails applications.
(apache 2.2.3 / prefork) Passenger 1.0.5

My settings:
RailsAutoDetect off
RailsMaxPoolSize 4


Things runs smooth initially. However by the next day, Apache is
frozen. In my Apache error_log, I notice this:

(12)Cannot allocate memory: fork: Unable to fork new process terminate
called after throwing an instance of 'boost::thread_resource_error'
what(): Cannot create a thread: Cannot allocate memory (12)

And in my virtual host's error_log, I see:

*** Unexpected error in Passenger: write() failed: Broken pipe (32)

At this point, I cannot restart apache normally - RC init.d script
reports that it isn't running. I have to individually kill off each
apache2 instance before starting it back up.

If I can provide any more information, please let me know what would
be helpful.

Thanks, keep up the great work!

[Hongli Lai]

Can you try version 2.0 RC 1? It uses less virtual memory.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Jason Hines]

Another thing I've noticed too, is that passenger-memory-stats seems
to be reporting incorrect TOTAL DATA.



-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------
16030 1 13 161.4 MB ? /usr/sbin/apache2 -k start
16037 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16039 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16040 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16041 16030 1 42.1 MB ? /usr/sbin/apache2 -k start
16273 16030 1 106.3 MB ? /usr/sbin/apache2 -k start
16274 16030 1 106.0 MB ? /usr/sbin/apache2 -k start
16276 16030 1 106.0 MB ? /usr/sbin/apache2 -k start
21831 16030 1 138.0 MB ? /usr/sbin/apache2 -k start
26548 16030 1 137.9 MB ? /usr/sbin/apache2 -k start
3824 16030 1 162.2 MB ? /usr/sbin/apache2 -k start
### Processes: 11
### Total private dirty RSS: 0.00 MB (?)

--------- Passenger processes ---------
PID Threads VMSize Private Name
---------------------------------------
16034 2 19.0 MB ? Passenger spawn server
21887 1 34.6 MB ? Passenger FrameworkSpawner: 2.0.2
1423 1 34.7 MB ? Passenger ApplicationSpawner: /var/
www/apps/devtwo
1425 1 34.7 MB ? Rails: /var/www/apps/devtwo
### Processes: 4
### Total private dirty RSS: 0.00 MB (?)

[Hongli Lai]

Jason Hines wrote:
> Another thing I've noticed too, is that passenger-memory-stats seems
> to be reporting incorrect TOTAL DATA.

The private dirty RSS can only be obtained if you're running as root and
you're on Linux.

[Jason Hines]

See full output below. I am running as root on a Linux Debian VPS.


root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# uname
Linux
root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# bin/passenger-
memory-stats

-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------
16030 1 13 161.4 MB ? /usr/sbin/apache2 -k start
16037 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16039 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16040 16030 1 42.3 MB ? /usr/sbin/apache2 -k start
16041 16030 1 42.1 MB ? /usr/sbin/apache2 -k start
16273 16030 1 106.3 MB ? /usr/sbin/apache2 -k start
16274 16030 1 106.0 MB ? /usr/sbin/apache2 -k start
16276 16030 1 106.0 MB ? /usr/sbin/apache2 -k start
21831 16030 1 138.0 MB ? /usr/sbin/apache2 -k start
26548 16030 1 137.9 MB ? /usr/sbin/apache2 -k start
3824 16030 1 162.2 MB ? /usr/sbin/apache2 -k start
### Processes: 11
### Total private dirty RSS: 0.00 MB (?)

--------- Passenger processes ---------
PID Threads VMSize Private Name
---------------------------------------

16034 2 19.3 MB ? Passenger spawn server

21887 1 34.6 MB ? Passenger FrameworkSpawner: 2.0.2

20234 1 34.7 MB ? Passenger ApplicationSpawner: /var/
www/apps/devtwo
20242 1 34.7 MB ? Rails: /var/www/apps/devtwo

### Processes: 4
### Total private dirty RSS: 0.00 MB (?)

[Hongli Lai]

Jason Hines wrote:
> See full output below. I am running as root on a Linux Debian VPS.
>
>
> root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# uname
> Linux
> root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# bin/passenger-
> memory-stats
> -------------- Apache processes ---------------
> PID PPID Threads VMSize Private Name
> -----------------------------------------------
> 16030 1 13 161.4 MB ? /usr/sbin/apache2 -k start

Is /proc mounted? The private dirty RSS information is obtained from
/proc/(some pid)/smaps.

[Jason Hines]

Yes, /proc is mounted, though there is 'maps', but no 'smaps' in those
directories:

root@web3:/proc/16030# ls
auxv cwd exe maps mounts stat status wchan
cmdline environ fd mem root statm task


Kernel is:
2.6.9-023stab041.3-enterprise #1 SMP Wed Feb 14 13:36:44 MSK 2007 i686
GNU/Linux

> E-mail: i...@phusion.nl

[Jason Hines]

I just had this happen again. Apache begins consuming all available
memory. I was able to capture this output before manually restarting
Apache.

root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# bin/passenger-
memory-stats
-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------

19488 1 16 153.7 MB ? /usr/sbin/apache2 -k start
27911 19488 1 132.6 MB ? /usr/sbin/apache2 -k start
9921 19488 1 132.2 MB ? /usr/sbin/apache2 -k start
9922 19488 1 131.3 MB ? /usr/sbin/apache2 -k start
16092 19488 1 140.6 MB ? /usr/sbin/apache2 -k start
23600 19488 1 147.5 MB ? /usr/sbin/apache2 -k start
9229 19488 1 148.0 MB ? /usr/sbin/apache2 -k start
15899 19488 1 145.8 MB ? /usr/sbin/apache2 -k start
26134 19488 1 154.2 MB ? /usr/sbin/apache2 -k start
26514 19488 1 155.2 MB ? /usr/sbin/apache2 -k start
32357 19488 1 155.9 MB ? /usr/sbin/apache2 -k start
4085 19488 1 154.4 MB ? /usr/sbin/apache2 -k start
7542 19488 1 156.2 MB ? /usr/sbin/apache2 -k start
26295 19488 1 154.8 MB ? /usr/sbin/apache2 -k start
### Processes: 14

### Total private dirty RSS: 0.00 MB (?)

--------- Passenger processes ---------
PID Threads VMSize Private Name
---------------------------------------

19494 2 22.6 MB ? Passenger spawn server
22243 1 35.7 MB ? Passenger FrameworkSpawner: 2.0.2
16120 1 35.8 MB ? Passenger ApplicationSpawner: /var/
www/apps/devtwo
### Processes: 3

[Hongli Lai]

Jason Hines wrote:
> I just had this happen again. Apache begins consuming all available
> memory. I was able to capture this output before manually restarting
> Apache.
>
> root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# bin/passenger-
> memory-stats

It seems you're still on 1.0.x. Please upgrade to 2.0 RC 2.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Hamed Hashemi]

I've been running RC1 and I just encountered a similar issue too.

First received

[ pid=21317 file=ApplicationPoolServerExecutable.cpp:469 time=06/17/08
13:39:02.72 ]:

Cannot create a thread: Cannot allocate memory (12)

terminate called after throwing an instance of

'Passenger::SystemException'
what(): write() failed: Broken pipe (32)
[Tue Jun 17 13:40:05 2008] [notice] child pid 32418 exit signal
Aborted (6)

terminate called after throwing an instance of

'Passenger::SystemException'
what(): write() failed: Broken pipe (32)
[Tue Jun 17 13:40:31 2008] [notice] child pid 32727 exit signal
Aborted (6)
[ pid=1559 file=Hooks.cpp:624 time=06/17/08 13:48:01.348 ]:
Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restarting or shutting down Apache)

Then started receiving hundred of lines (probably 1 per request) of

Unexpected error in Passenger: The ApplicationPool server exited
unexpectedly.

To get everything back to normal, I had to do apachectl restart.

On Jun 17, 11:55 am, Hongli Lai <hon...@phusion.nl> wrote:
> Jason Hines wrote:
> > I just had this happen again.  Apache begins consuming all available
> > memory.  I was able to capture this output before manually restarting
> > Apache.
>
> > root@web3:/usr/lib/ruby/gems/1.8/gems/passenger-1.0.5# bin/passenger-
> > memory-stats
>
> It seems you're still on 1.0.x. Please upgrade to 2.0 RC 2.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Jason Hines]

I've had better success running Passenger since upgrading to the gem
version 1.9.1.

The apache2 processes are consuming around 27MB VMSize now. Hopefully
they stay this low.

[Jason Hines]

9kb RES

[Matthew Crenshaw]

I am experiencing problems similar to this. I am running Enterprise
Ruby w/ Passenger 2.0.3.

The machine has 512mb of ram

I am currently NOT hosting ANY Rails application; this server is setup
to hold a future Rails app.

Here is a bit from apache's error_log:

*snip*
[Sun Aug 10 01:27:28 2008] [notice] Apache/2.2.3 (CentOS) configured
-- resuming normal operations
[Sun Aug 10 14:51:48 2008] [error] (12)Cannot allocate memory: fork:

Unable to fork new process

[ pid=18100 file=Hooks.cpp:624 time=08/10/08 14:51:48.452 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=15403 file=ApplicationPoolServerExecutable.cpp:469 time=08/10/08
14:51:48.466 ]:
Cannot create an anonymous Unix socket: Cannot allocate memory (12)
[ pid=18254 file=Hooks.cpp:624 time=08/10/08 14:51:58.464 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=18255 file=Hooks.cpp:624 time=08/10/08 14:51:58.469 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=18256 file=Hooks.cpp:624 time=08/10/08 14:51:58.473 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=18257 file=Hooks.cpp:624 time=08/10/08 14:51:58.478 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=18258[ pid=[ pid=18259 file= file=Hooks.cpp:624 time=08/10/08
14:51:58.18260Hooks.cpp: file=Hooks.cpp:624 time=[ pid=08/10/08
14:51:58624 time=.18261 file=Hooks.cpp08/10/0
8 14:51:58489483 ]:: ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

.486 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)


Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[ pid=18262 file=Hooks.cpp:624 time=08/10/08 14:51:58.499 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

624 time=08/10/08 14:51:58.493 ]:

Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe (32) (this warning
is harmless if you're currently restart
ing or shutting down Apache)

[Sun Aug 10 14:51:58 2008] [notice] child pid 18092 exit signal
Segmentation fault (11)
[Sun Aug 10 14:51:58 2008] [notice] child pid 18100 exit signal
Aborted (6)



This is remedied by restarting apache.


I am using Passenger 2.0.3, why am I running out of memory so quickly?

[mr_anderson]

I had the SAME EXACT problem on my server.
Simple hello world app (one controller that renders :text => "hello
world" and returns false)
But after about a day or two, it would die and take apache with it,
with exactly those messages in your logs.

The only way I could fix it was by setting
RailsSpawnMethod to Conservative.

Still want to know why it dies like that though.
Runing passenger 2.0.2