mod_rails kills my Apache

[Carsten Gehling]

I have not been so lucky in my experiments with mod_rails...

I've installed it on a server previously running mongrel clusters.

The server is a FreeBSD 7.0
Apache is Apache 2.2 with MPM-prefork

I can setup mod_rails and run the apache with no problems. However
within 4-8 hours, the sites become unresponsive. All Apache processes
are now reported by "ps aux" as <defunct>. I cannot restart Apache
with "apachectl restart", I have to reboot the entire server.

I cannot find anything inside the logfiles that gives any hint to what
is happening. I am totally clueless. I've now gone back to mongrel
until I find out what is wrong.

Any hints? Suggestions to where I should look?

- Carsten

[John Trupiano]

I've been dealing with a similar plight. I've been testing out
passenger on a staging server, and every so often (not as frequently
as the 4-8 hours you reported), all of my apache processes become
defunct. I have to manually kill them (killall apache2), and start
apache again.

This has happened to me each of the last two weekends. Our staging
server gets substantially less traffic (perhaps none) on the
weekends. Could it be related to long periods of inactivity (client
requests)?

-John

[bensie]

I'm having the exact same issue. Specifically over the weekend when
traffic drops to zero.

[Hongli Lai]

Hi Carsten, John, Bensie.

Thanks for the feedback, and sorry for the stability issues. We've
received a number of other reports about similar stability problems. And
as such, we've increased our stress testing efforts.

In the past week, we've identified a number of stability issues, and
we've made great progress towards improving stability. This work is
available in the development version of Passenger (i.e. the git
repository). We're quite confident in that the development version is
more stable than the "stable" 1.0.x series. We've been stress testing
Passenger for hours now by bombarding it with 24 concurrent crawlers,
while doing things like gracefully restarting Apache and restarting the
application at random times, and so far Passenger has been very stable.

So we'd like to invite you to try the development version. This is
available at:
http://github.com/FooBarWidget/passenger/tarball/master

One can install the development version in 2 ways:
- by running 'bin/passenger-install-apache2-module' directly from the
source tarball.
- by extracting the source tarball, then generating a gem with 'rake
package'. After installing the gem, one can run
'passenger-install-apache2-module'.
In any case, please don't forget to copy & paste the new configuration
options provided by the installer.

We're looking forward to your feedback. Needless to say, we're very
interested in addressing whatever stability issues you might experience
in the development version.

With kind regards,
Hongli Lai
--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Tigris]

Hi Hongli,

On May 19, 2:36 am, Hongli Lai <hon...@phusion.nl> wrote:
> So we'd like to invite you to try the development version. This is
> available at:http://github.com/FooBarWidget/passenger/tarball/master
>
> One can install the development version in 2 ways:
> - by running 'bin/passenger-install-apache2-module' directly from the
> source tarball.
> - by extracting the source tarball, then generating a gem with 'rake
> package'. After installing the gem, one can run
> 'passenger-install-apache2-module'.
> In any case, please don't forget to copy & paste the new configuration
> options provided by the installer.
>
> We're looking forward to your feedback. Needless to say, we're very
> interested in addressing whatever stability issues you might experience
> in the development version.

I too am seeing apache issues like the other guys mentioned. I tried
to upgrade to the current 1.1.0 development gem hoping to fix my
multiple environment issue at the same time. The `rake package`
created the gem just fine (after I did a `yum install asciidoc`). But
after doing a gem install and trying to run the passenger-install-
apache2-module script, i get the following errors:

Checking for required software...

* GNU C++ compiler... found at /usr/bin/g++
* Ruby development headers... found
* OpenSSL support for Ruby... found
* RubyGems... found
* Rake... found at /usr/local/rubygems/gems/bin/rake
* Apache 2... found at /usr/sbin/httpd
* Apache 2 development headers... found at /usr/sbin/apxs
* Apache Portable Runtime (APR) development headers... found at /usr/
bin/apr-1-config
* fastthread... found

### In ext/boost/src:
g++ -g -DPASSENGER_DEBUG -fPIC -I../.. -D_REENTRANT -DNDEBUG -c *.cpp
pthread/*.cpp
In file included from ../../boost/date_time/gregorian/
greg_calendar.hpp:14,
from ../../boost/date_time/gregorian/
gregorian_types.hpp:19,
from ../../boost/date_time/posix_time/
posix_time_config.hpp:14,
from ../../boost/date_time/posix_time/
posix_time_system.hpp:13,
from ../../boost/date_time/posix_time/ptime.hpp:12,
from ../../boost/date_time/posix_time/
posix_time_types.hpp:12,
from ../../boost/thread/thread_time.hpp:10,
from ../../boost/thread/locks.hpp:11,
from ../../boost/thread/pthread/mutex.hpp:11,
from ../../boost/thread/mutex.hpp:16,
from ../../boost/thread/pthread/thread.hpp:14,
from ../../boost/thread/thread.hpp:17,
from pthread/thread.cpp:10:
../../boost/date_time/gregorian_calendar.hpp:63:50: error: boost/
date_time/gregorian_calendar.ipp: No such file or directory
rake aborted!
Command failed with status (1): [g++ -g -DPASSENGER_DEBUG -fPIC -
I../.. -D_...]
/usr/local/rubygems/gems/gems/passenger-1.1.0/Rakefile:97
(See full trace by running task with --trace)

--------------------------------------------
It looks like something went wrong


Any ideas? The file I downloaded from your above link just now was
named FooBarWidget-passenger-
c91301a114d1931f58d837b3b1dc4cf91bfc0291.tar.gz.


regards,
Danial

[Hongli Lai]

That was a little bug in the packaging Rake task. It has been fixed,
thanks for the report.

[Hongli Lai]

I'll also be in #passenger @ irc.freenode.net today, if there are issues.

[bensie]

Have you done any "no stress" testing for when there are no requests
for an extended period of time? That is the issue that we are having
with our Apache processes going defunct after somewhere beyond 24
hours of inactivity.

Thanks for your continued support and development.

On May 19, 6:32 am, Hongli Lai <hon...@phusion.nl> wrote:
> I'll also be in #passenger @ irc.freenode.net today, if there are issues.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

bensie wrote:
> Have you done any "no stress" testing for when there are no requests
> for an extended period of time? That is the issue that we are having
> with our Apache processes going defunct after somewhere beyond 24
> hours of inactivity.

Yes we have. We haven't found any problems in such cases.

With kind regards,
Hongli Lai

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Carsten Gehling]

Hi Hongli,

At the moment I have a theory about WHEN my Apache crashes to
<defunct>:

The timing match with a cron-job, that I have running approx every 4th
hour. The cron-job does the following:

1) SVN updates a development version of my site (also running
mod_rails)
2) Does a mongrel_rails stop/start on the development version (this
was an action, that I missed removing when switching to mod_rails)
3) Does an "apachectl graceful"

I removed no. 2) since mongrel isn't used with mod_rails. But the
problem persists. So I think that "apachectl graceful" might be the
culprit.

I read that you've included graceful restarts in your stress-test of
the development version, so I will gladly test this - although I
haven't got any time tonight. Tomorrow I will test it.

My site is serving approx. 25.000 page views each day (only inluding
rails-specific requests), so it is fairly well visited. The issue for
me is NOT because of inactivity, since the crashes happened during
normal periods of traffic.

But I'll test dev. version tomorrow and report back. Thank you for
this great support! :-)

- Carsten

[Danimal]

Carsten,

Why aren't you just doing the "touch tmp/restart.txt" in the rails
app? That's the mod_rails way of telling Apache to restart.

Would that make any difference?

-Danimal

On May 19, 11:53 am, Carsten Gehling <carsten.gehl...@gmail.com>
wrote:

[John Trupiano]

Hongli,

I've just updated our staging server with the latest from github. I
will certainly keep you posted on any issues/successes we have. I'm
already happy about the RailsEnv fix (one less cap task!).

btw, when do you expect the next version to be released?

Thanks.

-John

[Carsten Gehling]

On 19 Maj, 20:17, Danimal <fightonfightw...@gmail.com> wrote:
> Carsten,
>
> Why aren't you just doing the "touch tmp/restart.txt" in the rails
> app? That's the mod_rails way of telling Apache to restart.

I know - the above mentioned "graceful restart" was done because I had
forgot to remove it when switching from mongrel to mod_rails.

The reason I mentioned it here was, that maybe mod_rails has some
issues with "graceful".

- Carsten

[Hongli Lai]

Carsten Gehling wrote:
> I know - the above mentioned "graceful restart" was done because I had
> forgot to remove it when switching from mongrel to mod_rails.
>
> The reason I mentioned it here was, that maybe mod_rails has some
> issues with "graceful".

There were indeed some issues, but they had been fixed a while ago.

Regards,

[Carsten Gehling]

On 20 Maj, 13:41, Hongli Lai <hon...@phusion.nl> wrote:

> Carsten Gehling wrote:
>
> > The reason I mentioned it here was, that maybe mod_rails has some
> > issues with "graceful".
>
> There were indeed some issues, but they had been fixed a while ago.

But are they fixed in the stable version or only in the development
version?

I have now run the dev. version on my server for 9 hours, and so far
it runs without problems.

- Carsten

[Hongli Lai]

Carsten Gehling wrote:
> But are they fixed in the stable version or only in the development
> version?

In the development version.

---

[Carsten Gehling]

On 20 Maj, 21:41, Hongli Lai <hon...@phusion.nl> wrote:
> Carsten Gehling wrote:
> > But are they fixed in the stable version or only in the development
> > version?
>
> In the development version.

Then I believe, that my problems were because of these issues with
"graceful". The server has now run for 24 hours without any
breakdowns, and it is way better to handle heavy loads.

Brilliant work! Praise to all the good people at Phusion :-)

- Carsten

[bensie]

I have installed the development version as well, and will let you
know if apache stays alive. Thanks again for continuing development
on this incredible work!

[Kelly]

I don't know if this is related, but I'm getting similar Apache
slowdowns after installing the passenger module. The issue is
temporarily fixed by restarting Apache, but after an hour or so all
sites become unresponsive.

I'm seeing the following in my error_log:

apache2: ../boost/shared_ptr.hpp:315: T*
boost::shared_ptr<T>::operator->() const [with T =
Passenger::ApplicationPoolServer::ClientInfo]: Assertion `px != 0'
failed.

Interestingly load, cpu utilization, and available memory do not
spike.

In my Apache setup I have one virtual host pointing to a rails app.
This virtual host receives virtually no traffic (which is the reason I
used this particular virtual host). The remaining 50 virtual hosts
serve static content and Perl CGI. Nothing complicated.

Any ideas?

Thanks,

Kelly

P.S. Thanks for your hard work.

[Hongli Lai]

Kelly wrote:
> I don't know if this is related, but I'm getting similar Apache
> slowdowns after installing the passenger module. The issue is
> temporarily fixed by restarting Apache, but after an hour or so all
> sites become unresponsive.
>
> I'm seeing the following in my error_log:
>
> apache2: ../boost/shared_ptr.hpp:315: T*
> boost::shared_ptr<T>::operator->() const [with T =
> Passenger::ApplicationPoolServer::ClientInfo]: Assertion `px != 0'
> failed.
>
> Interestingly load, cpu utilization, and available memory do not
> spike.
>
> In my Apache setup I have one virtual host pointing to a rails app.
> This virtual host receives virtually no traffic (which is the reason I
> used this particular virtual host). The remaining 50 virtual hosts
> serve static content and Perl CGI. Nothing complicated.
>
> Any ideas?
>
> Thanks,
>
> Kelly
>
> P.S. Thanks for your hard work.

Does this happen with the development version as well?

--

[Kelly]

I'll try the devel version in off peak hours later tonight.

> E-mail: i...@phusion.nl

[bensie]

So far so good on the dev version....plus I am seeing noticeable
performance improvements in comparison to 1.0.5. Great work!

[Carsten Gehling]

Unfortunately my entusiasm was a little bit too early... ;-)

Apache no longer fails to <defunct> processes, but after some hours my
website still gets unresponsive. The only solution here is to
"apachectl restart" - then the site works again.

The last line in the logfile before the site became unresponsive is:

[Wed May 21 21:04:15 2008] [error] server reached MaxClients setting,
consider raising the MaxClients setting

- Carsten

[Hongli Lai]

How long did it take before this happens?

Could you also post the output of 'cat /tmp/passenger_status.*.fifo'
when it happens?

--

[Hongli Lai]

By the way, the latest development version provides a way to inspect
frozen Rails application instances:
http://www.modrails.com/documentation/Users%20guide%20latest.html#debugging_frozen

--

[Tigris]

> By the way, the latest development version provides a way to inspect

> frozen Rails application instances:http://www.modrails.com/documentation/Users%20guide%20latest.html#deb...

Awesome tool. I just upgraded to dev version
9dafc631270b045975f817966558f25f41483110. It did not install this
tool, but i was able to dig it out of the development bin folder. I
think maybe a simple tweak of the Rakefile or some config file should
fix this fact?

However, it seems inaccurate....

My configs show:

$ grep RailsMax /var/www/vhosts/*/conf/vhost.conf

/var/www/vhosts/my_site/conf/vhost.conf:RailsMaxPoolSize 7
/var/www/vhosts/other_site/conf/vhost.conf:RailsMaxPoolSize 2


$ ~/FooBarWidget-passenger-9dafc631270b045975f817966558f25f41483110/
pkg/passenger-1.1.0/bin/passenger-status
----------- General information -----------
max = 20
count = 5
active = 0
inactive = 5

----------- Applications -----------
/var/www/vhosts/my_site/rails/my_app/releases/20080526044941:
PID: 13709 Sessions: 0
PID: 1554 Sessions: 0
PID: 1556 Sessions: 0
PID: 1551 Sessions: 0

/var/www/vhosts/other_site/rails/my_app/releases/20080526031126:
PID: 1814 Sessions: 0


Where is it getting 20 as a max value from?

regards,
Danial

[Danial Pearce]

> Where is it getting 20 as a max value from?

Of course, I now see that i am a newb and need to read the new docs
when using a new version. RailsMaxPoolSize is server wide and not
configurable per host. So my setting inside the VirtualHost was just
being ignored. As was my RailsPoolIdleTime which would explain some of
my recent load issues.

regards,
Danial

[Carsten Gehling]

On 22 Maj, 11:00, Hongli Lai <hon...@phusion.nl> wrote:
> Carsten Gehling wrote:

> > Apache no longer fails to <defunct> processes, but after some hours my
> > website still gets unresponsive. The only solution here is to
> > "apachectl restart" - then the site works again.
>

> How long did it take before this happens?

It took around 24 hours before it happened (I am not quite sure, since
I was not the one who discovered it first time (the guy who did
restarted Apache, so I did not know exactly how long it had run).

> Could you also post the output of 'cat /tmp/passenger_status.*.fifo'
> when it happens?

There was no such file in my /tmp folder, when I looked.

I am a little reluctant to try again right now. This happened on our
production server, and my customers need some time with stability
without me fudging around. :-)

I will let you know, when I can try again. Then I will try with your
latest development version and find as much debug information as
possible.

- Carsten

[rogerdpack]

a couple of thoughts:
I assume if you do an apache graceful restart it graceful reloads
existing rails instances?
Also...what about mysql connections that timeout after 6 hours? Is
this accomodated for?
Thanks for the great stuff :)
-R

On May 22, 6:47 am, Hongli Lai <hon...@phusion.nl> wrote:
> Carsten Gehling wrote:
> > Unfortunately my entusiasm was a little bit too early... ;-)
>
> > Apache no longer fails to <defunct> processes, but after some hours my
> > website still gets unresponsive. The only solution here is to
> > "apachectl restart" - then the site works again.
>
> > The last line in the logfile before the site became unresponsive is:
>
> > [Wed May 21 21:04:15 2008] [error] server reached MaxClients setting,
> > consider raising the MaxClients setting
>
> > - Carsten
>
> By the way, the latest development version provides a way to inspect

> frozen Rails application instances:http://www.modrails.com/documentation/Users%20guide%20latest.html#deb...

>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

rogerdpack wrote:
> a couple of thoughts:
> I assume if you do an apache graceful restart it graceful reloads
> existing rails instances?

Yes.

> Also...what about mysql connections that timeout after 6 hours? Is
> this accomodated for?

That's handled by the Rails application and has got nothing to do with
Passenger.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[greg.newman]

I too am running into issues with Apache hanging with passenger 1.0.5.
Has anyone confirmed that the devel version at github is the fix?

[Tue Jun 03 18:32:42 2008] [notice] Graceful restart requested, doing
restart
[Tue Jun 03 18:32:42 2008] [error] (9)Bad file descriptor:
apr_socket_accept: (client socket)
[Tue Jun 03 18:32:42 2008] [error] (9)Bad file descriptor:
apr_socket_accept: (client socket)
[Tue Jun 03 18:32:42 2008] [error] (9)Bad file descriptor:
apr_socket_accept: (client socket)
[16124:StandardApplicationPool.h:249] Cleaning idle app /home/sitename/
rails (PID 10621)
[16124:Application.h:274] Application 0x96c9108: destroyed.
[16124:StandardApplicationPool.h:249] Cleaning idle app /home/sitename/
rails (PID 7240)
[16124:Application.h:274] Application 0x96c99b0: destroyed.
[16124:StandardApplicationPool.h:249] Cleaning idle app /home/sitename/
rails (PID 11011)
[16124:Application.h:274] Application 0x9737b08: destroyed.
[16124:StandardApplicationPool.h:249] Cleaning idle app /home/sitename/
rails (PID 10514)
[16124:Application.h:274] Application 0x9738898: destroyed.
*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)
(process 1959):

[Clipping backtrace to condense]

[Tue Jun 03 18:54:13 2008] [notice] suEXEC mechanism enabled
(wrapper: /usr/local/apache/bin/suexec)
[Tue Jun 03 18:54:13 2008] [warn] pid file /usr/local/apache/logs/
httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Jun 03 18:54:13 2008] [notice] Apache/2.2.8 (Unix) mod_ssl/2.2.8
OpenSSL/0.9.8b mod_bwlimited/1.4 PHP/5.2.6 Phusion_Passenger/1.0.5
configured -- resuming normal operations

[Dallas Kashuba]

The development versions have been much more stable than 1.0.5 for us.

Dallas

[greg.newman]

I installed the devel version this morning and will be watching it to
see if apache freezes again.

[greg.newman]

Wanted to swing by here and give an update.
I installed the devel version (1.10) almost 36 hours ago and haven't
had a freeze yet. (knock on veneer)