REE 2010.01 + Passenger 2.2.10 on Apache hangs
Steve Madsen
Passenger 2.2.10. Typical behavior for this server is 7-10 requests
per second, though only maybe 1 of those ends up in the Rails
application.
Over the next couple of days, I experienced symptoms similar to a SYN
flood attack, where I'd see 60-70 connections in TCP SYN_RECV state,
with all of the available Apache children running, many of them stuck
in W/"Sending Reply" state according to server-status. Fully stopping
and restarting Apache would fix the problem for a period of time, but
it'd come back within a couple of hours.
Last night I backed off to REE 2009.10 and Passenger 2.2.9 and the
site has been fine since. I can't do further testing at the moment
because I'm going out of town tomorrow and it's a production server.
When I come back I'd like to narrow down if it's Passenger (which I
suspect; Googling tells me that children stuck in W state is usually
due to a problematic module) or REE.
Has anyone else seen similar behavior after upgrading to 2.2.10 or
with REE 2010.01?
silent (e)
this week. After downgrading back to 2.2.8 I've had no problems.
This server is running ...
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-freebsd6.3]
Ruby Enterprise Edition 20090610
Server version: Apache/2.2.11 (FreeBSD)
I can't confirm that 2.2.10 was the problem but it started after
upgrading. I don't have enough apache-fu to diagnose the way you did
so i'll be interested to know what you find out.
matte
Phil Ross
> Over the next couple of days, I experienced symptoms similar to a SYN
> flood attack, where I'd see 60-70 connections in TCP SYN_RECV state,
> with all of the available Apache children running, many of them stuck
> in W/"Sending Reply" state according to server-status. Fully stopping
> and restarting Apache would fix the problem for a period of time, but
> it'd come back within a couple of hours.
>
> with REE 2010.01?
We're seeing almost exactly the same problem with Passenger 2.2.10.
The only difference is the problems start immediately after restarting
Apache - each request immediately gets stuck in the W (Sending Reply)
state for a long time. passenger-status reports no or few active
requests.
We are now running Passenger 2.2.9 again, which is working without any
problems.
We're running Apache 2.2.3 on Red Hat Enterprise Linux 5.4 with Ruby
1.8.6p369. Our Passenger config is as follows:
Global config:
PassengerMaxPoolSize 8
PassengerMaxInstancesPerApp 7
PassengerPoolIdleTime 0
RailsAppSpawnerIdleTime 1200
Virtual host (the only Passenger vhost):
PassengerUseGlobalQueue on
PassengerMaxRequests 30000
Regards,
Phil
Kamal Fariz
to 2.2.10 today after a mysterious Apache crash. Recalling that 2.2.10
fixed a file descriptor issue, I upgraded to it. After an hour,
haproxy took the app server out of rotation. I saw that there were
tons of httpd child processes while passenger-status itself showed no
active connections. A restart of Apache didn't help.
I downgraded to 2.2.9 and things are running again.
My config:
Ruby 1.8.6-p111 (non-REE)
Apache 2.2.3
Centos 5
Global config:
PassengerUseGlobalQueue On
PassengerMaxPoolSize 20
PassengerMaxInstancesPerApp 8
PassengerPoolIdleTime 0
vhost config:
PassengerHighPerformance on
Regards,
kamal
chuyeow
2010.01). We run 3 servers on load-balancing were wondering why only 1
particular server which we commissioned later and so had 2.2.10 had
lots of timeouts when restarting. We're using Passenger 2.2.9 on the
other 2 servers.
I'll downgrade to 2.2.9 and report back but I suspect our timeout
problems will be gone.
Cheers,
Chu Yeow
Hongli Lai
> I'm seeing the same problem too upgrading Passenger to 2.2.10 (REE is
> 2010.01). We run 3 servers on load-balancing were wondering why only 1
> particular server which we commissioned later and so had 2.2.10 had
> lots of timeouts when restarting. We're using Passenger 2.2.9 on the
> other 2 servers.
>
> I'll downgrade to 2.2.9 and report back but I suspect our timeout
> problems will be gone.
I guess the file descriptor fixes caused some regressions.
Can those who can reproduce the problem run 'passenger-status
--show=backtraces' and post the output?
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
Hongli Lai
http://github.com/FooBarWidget/passenger/commit/b8047d7567438d1d8a84ee1b453115d8fb95d8fe
Can anybody confirm?
Scottie35
> I think I've found the cause of the problem. This should fix the problem:http://github.com/FooBarWidget/passenger/commit/b8047d7567438d1d8a84e...
> Can anybody confirm?
So far, so good!
Before, I couldn't even get the site to load with 2.2.10 and the
latest REE. Now it's running just fine!
Thanks,
Scottie
Scottie35
On Mar 2, 12:44 pm, Scottie35 <sc...@sott.net> wrote:
>
> So far, so good!
>
Oops. Apache just bit the bullet. Back to 2.2.9!
From my Apache logs, looks like I got a ton of these:
[ pid=29278 file=ext/apache2/Hooks.cpp:727 time=2010-03-02
07:39:32.406 ]:
Unexpected error in mod_passenger: Could not connect to the
ApplicationPool server: Broken pipe (32)
Backtrace:
in 'Passenger::ApplicationPoolPtr
Passenger::ApplicationPoolServer::connect()' (ApplicationPoolServer.h:
746)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:523)
Hope it helps!
Scottie
Philip Ross
> I think I've found the cause of the problem. This should fix the problem:
> http://github.com/FooBarWidget/passenger/commit/b8047d7567438d1d8a84ee1b453115d8fb95d8fe
> Can anybody confirm?
This seems to have fixed it for us.
Previously, most requests were getting stuck in the Sending Reply
state. Now, everything is working fine.
Thanks,
Phil
Hongli Lai
> On 2 March 2010 11:13, Hongli Lai <hon...@phusion.nl> wrote:
>
> > I think I've found the cause of the problem. This should fix the problem:
> > Can anybody confirm?
>
> This seems to have fixed it for us.
>
> Previously, most requests were getting stuck in the Sending Reply
> state. Now, everything is working fine.
>
> Thanks,
>
> Phil
Can anybody else test whether my patch is effective?
Nash
about to update our production gems, but waiting on this to be
corrected.
Thanks.
-Nash
Hongli Lai
> Any ETA on when the next gem will be released with this fix? We're
> about to update our production gems, but waiting on this to be
> corrected.
I want to release it as soon as a few more people confirm that the fix works.
jonb
stress tests today, but the hanging problems I was seeing in the first
few requests have gone away.
Thanks,
-Jon
On Mar 3, 12:56 pm, Hongli Lai <hon...@phusion.nl> wrote:
> On Wed, Mar 3, 2010 at 6:46 PM, Nash <n...@kabbara.us> wrote:
> > Any ETA on when the next gem will be released with this fix? We're
> > about to update our production gems, but waiting on this to be
> > corrected.
>
> I want to release it as soon as a few more people confirm that the fix works.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Nash
running OK.
But the thing is, when I first installed the gem, I did get some
hanging which later went away after restarting the servers a couple of
times.
So it's a little difficult for me to say with 100% confidence that
this change has fixed the problem.
But I do know that things are working fine after applying the patch.
Thanks.
-Nash
On Mar 3, 2:56 pm, Hongli Lai <hon...@phusion.nl> wrote:
> On Wed, Mar 3, 2010 at 6:46 PM, Nash <n...@kabbara.us> wrote:
> > Any ETA on when the next gem will be released with this fix? We're
> > about to update our production gems, but waiting on this to be
> > corrected.
>
> I want to release it as soon as a few more people confirm that the fix works.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Leo baltus
On 2 mrt, 12:13, Hongli Lai <hon...@phusion.nl> wrote:
> I think I've found the cause of the problem. This should fix the problem:http://github.com/FooBarWidget/passenger/commit/b8047d7567438d1d8a84e...
> Can anybody confirm?
>
Yes, this fixed it for us as well.
Hongli Lai
upload the tarball and post the announcement later today.
John Leach
> Thanks for the confirmations. I've pushed 2.2.11 to Gemcutter. Will
> upload the tarball and post the announcement later today.
We've seen this too - you fixed it before we even reported it, nice
work :)
Ubuntu Hardy packages available now:
http://blog.brightbox.co.uk/posts/passenger-2-2-11-packages-for-ubuntu-8-04-hardy
John.