passenger respawning app problem.

[chposter]

Hi,

we have an app working on centos 5.6 x86_64, with Ruby Enterprise
Edition 1.8 7 2011-02-18 patchlevel 334 passsenger 3.0.7 and apache
2.2.17, defined as rack app, and rails 3.0.7.

The app works fine but unexpectedly the respawning fails:

First we start seeing errors like this (a lot):

[ pid=20143 thr=47969805936176 file=ext/apache2/Hooks.cpp:789
time=2011-05-13 08:52:15.11 ]: Either the vistor clicked on the 'Stop'
button in the web browser, or the visitor's connection has stalled and
couldn't receive the data that Apache is sending to it. As a result,
you will probably see a 'Broken Pipe' error in this log file. Please
ignore it, this is normal. You might also want to increase Apache's
TimeOut configuration option if you experience this problem often.


TimeOut is 300.
Second, the respawning fails (a lot):



[ pid=21681 thr=194380080 file=utils.rb:176 time=2011-05-13
09:01:54.429 ]: *** Exception LoadError in
PhusionPassenger::ClassicRails::ApplicationSpawner (No such file to
load -- application) (process 21681, thread #<Thread:0x172c0260>):



this app works fine normally and everything looks right . This is a
high visits app, so it tries respawning a lot, freezing the passenger
response. It looks like it goes railsbaseuri, instead of rackbaseuri

and then (maybe this is related with the apache restart) :



[ pid=22380 thr=47969805936176 file=ext/apache2/Hooks.cpp:865
time=2011-05-13 09:10:01.765 ]: Unexpected error in mod_passenger: The
ApplicationPool server unexpectedly closed the connection while we're
reading a response for the 'get' command.
Backtrace:
in 'void Passenger::ApplicationPool::Client::sendGetCommand(const
Passenger::PoolOptions&, std::vector<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<c
har>, std::allocator<char> > > >&)' (Client.h:382)
in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Client::get(const
Passenger::PoolOptions&)' (Client.h:699)
in 'Passenger::SessionPtr Hooks::getSession(const
Passenger::PoolOptions&)' (Hooks.cpp:297)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)

[ pid=22133 thr=47969805936176 file=ext/apache2/Hooks.cpp:865
time=2011-05-13 09:10:01.780 ]: Unexpected error in mod_passenger:
Cannot connect to Unix socket '/tmp/passenger.1.0.17025/generation-20/
socket': No such file or directory (2)
Backtrace:
in 'Passenger::ApplicationPool::Client*
Passenger::ApplicationPool::Client::connect(const std::string&, const
std::string&, const Passenger::StaticString&)' (Client.h:446)
in 'Passenger::ApplicationPool::Client*
Hooks::getApplicationPool()' (Hooks.cpp:271)
in 'Passenger::SessionPtr Hooks::getSession(const
Passenger::PoolOptions&)' (Hooks.cpp:297)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)

[ pid=22477 thr=47969805936176 file=ext/apache2/Hooks.cpp:865
time=2011-05-13 09:10:01.810 ]: Unexpected error in mod_passenger:
Could not read security response for the 'get' command from the
ApplicationPool server: read() failed: Connection reset by peer (104)
Backtrace:
in 'void Passenger::ApplicationPool::Client::sendGetCommand(const
Passenger::PoolOptions&, std::vector<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > > >&)' (Client.h:340)
in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Client::get(const
Passenger::PoolOptions&)' (Client.h:699)
in 'Passenger::SessionPtr Hooks::getSession(const
Passenger::PoolOptions&)' (Hooks.cpp:297)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)

[ pid=22477 thr=47969805936176 file=ext/apache2/Hooks.cpp:865
time=2011-05-13 09:10:01.810 ]: Unexpected error in mod_passenger:
Cannot connect to Unix socket '/tmp/passenger.1.0.17025/generation-20/
socket': No such file or directory (2)
Backtrace:
in 'Passenger::ApplicationPool::Client*
Passenger::ApplicationPool::Client::connect(const std::string&, const
std::string&, const Passenger::StaticString&)' (Client.h:446)
in 'Passenger::ApplicationPool::Client*
Hooks::getApplicationPool()' (Hooks.cpp:271)
in 'Passenger::SessionPtr Hooks::getSession(const
Passenger::PoolOptions&)' (Hooks.cpp:297)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)


And, Finally, restarting the mess

terminate called after throwing an instance of
'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error>
>'
what(): Invalid argument (22)
[ pid=21679, timestamp=1305270601 ] Process aborted! signo=SIGABRT(6),
reason=SI_TKILL, signal sent by PID 21679 with UID 0, backtrace
available.
--------------------------------------
[ pid=21679 ] Backtrace with 34 frames:
PassengerHelperAgent[0x5019a5]
/lib64/libpthread.so.0[0x2ad34eaabb10]
/lib64/libc.so.6(gsignal+0x35)[0x2ad34f47a265]
/lib64/libc.so.6(abort+0x110)[0x2ad34f47bd10]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv
+0x114)[0x2ad34ed76d14]
/usr/lib64/libstdc++.so.6[0x2ad34ed74e16]
/usr/lib64/libstdc++.so.6[0x2ad34ed741cb]
/usr/lib64/libstdc++.so.6(__gxx_personality_v0+0x353)[0x2ad34ed74ca3]
/lib64/libgcc_s.so.1[0x2ad34f244978]
/lib64/libgcc_s.so.1(_Unwind_Resume+0x9a)[0x2ad34f244cea]
PassengerHelperAgent(_ZN5boost5mutex4lockEv+0x58)[0x4c25d6]
PassengerHelperAgent(_ZN5boost10lock_guardINS_5mutexEEC1ERS1_+0x27)
[0x4c2605]
PassengerHelperAgent(_ZN9Passenger5Timer5startEv+0x1d)[0x4c2851]
PassengerHelperAgent(_ZN18TimerUpdateHandler18clientDisconnectedERN9Passenger13MessageServer19CommonClientContextERN5boost10shared_ptrINS1_13ClientContextEEE
+0x3d)[0x4c28bd]
PassengerHelperAgent(_ZN9Passenger13MessageServer29DisconnectEventBroadcastGuardD1Ev
+0x7c)[0x498674]
PassengerHelperAgent(_ZN9Passenger13MessageServer22clientHandlingMainLoopERNS_14FileDescriptorE
+0xd4e)[0x4f2cb2]
PassengerHelperAgent(_ZNK5boost4_mfi3mf1IvN9Passenger13MessageServerERNS2_14FileDescriptorEEclEPS3_S5_
+0x6a)[0x4a23c8]
PassengerHelperAgent(_ZN5boost3_bi5list2INS0_5valueIPN9Passenger13MessageServerEEENS2_INS3_14FileDescriptorEEEEclINS_4_mfi3mf1IvS4_RS7_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i
+0x5a)[0x4a2f7c]
PassengerHelperAgent(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf1IvN9Passenger13MessageServerERNS4_14FileDescriptorEEENS0_5list2INS0_5valueIPS5_EENSA_IS6_EEEEEclEv
+0x36)[0x4a2fbc]
PassengerHelperAgent(_ZN5boost6detail8function26void_function_obj_invoker0INS_3_bi6bind_tIvNS_4_mfi3mf1IvN9Passenger13MessageServerERNS7_14FileDescriptorEEENS3_5list2INS3_5valueIPS8_EENSD_IS9_EEEEEEvE6invokeERNS1_15function_bufferE
+0x20)[0x4a2fde]
PassengerHelperAgent(_ZNK5boost9function0IvEclEv+0x61)[0x4d358b]
PassengerHelperAgent(_ZN3oxt20dynamic_thread_group11thread_mainERN5boost8functionIFvvEEEPNS0_13thread_handleE
+0x2f)[0x4d3679]
PassengerHelperAgent(_ZNK5boost4_mfi3mf2IvN3oxt20dynamic_thread_groupERNS_8functionIFvvEEEPNS3_13thread_handleEEclEPS3_S7_S9_
+0x72)[0x4a22da]
PassengerHelperAgent(_ZN5boost3_bi5list3INS0_5valueIPN3oxt20dynamic_thread_groupEEENS2_INS_8functionIFvvEEEEENS2_IPNS4_13thread_handleEEEEclINS_4_mfi3mf2IvS4_RS9_SC_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i
+0x73)[0x4a2db5]
PassengerHelperAgent(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf2IvN3oxt20dynamic_thread_groupERNS_8functionIFvvEEEPNS5_13thread_handleEEENS0_5list3INS0_5valueIPS5_EENSE_IS8_EENSE_ISB_EEEEEclEv
+0x36)[0x4a2df6]
PassengerHelperAgent(_ZN5boost6detail8function26void_function_obj_invoker0INS_3_bi6bind_tIvNS_4_mfi3mf2IvN3oxt20dynamic_thread_groupERNS_8functionIFvvEEEPNS8_13thread_handleEEENS3_5list3INS3_5valueIPS8_EENSH_ISB_EENSH_ISE_EEEEEEvE6invokeERNS1_15function_bufferE
+0x20)[0x4a2e18]
PassengerHelperAgent(_ZNK5boost9function0IvEclEv+0x61)[0x4d358b]
PassengerHelperAgent(_ZN3oxt6thread11thread_mainEN5boost8functionIFvvEEENS1_10shared_ptrINS0_11thread_dataEEE
+0x50)[0x4ef5c4]
PassengerHelperAgent(_ZN5boost3_bi5list2INS0_5valueINS_8functionIFvvEEEEENS2_INS_10shared_ptrIN3oxt6thread11thread_dataEEEEEEclIPFvS5_SB_ENS0_5list0EEEvNS0_4typeIvEERT_RT0_i
+0x6e)[0x4aa116]
PassengerHelperAgent(_ZN5boost3_bi6bind_tIvPFvNS_8functionIFvvEEENS_10shared_ptrIN3oxt6thread11thread_dataEEEENS0_5list2INS0_5valueIS4_EENSD_IS9_EEEEEclEv
+0x36)[0x4aa19e]
PassengerHelperAgent(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvPFvNS_8functionIFvvEEENS_10shared_ptrIN3oxt6thread11thread_dataEEEENS2_5list2INS2_5valueIS6_EENSF_ISB_EEEEEEE3runEv
+0x1c)[0x4aa1bc]
PassengerHelperAgent(thread_proxy+0x59)[0x519013]
/lib64/libpthread.so.0[0x2ad34eaa373d]
/lib64/libc.so.6(clone+0x6d)[0x2ad34f51e4bd]


Then it starts again.

Any clue about this?

[chposter]

I sent this the other day ... but

Hi,

we have an app working on centos 5.6 x86_64, with ruby Ruby Enterprise

Edition 1.8 7 2011-02-18 patchlevel 334 passsenger 3.0.7 and apache
2.2.17, defined as rack app, and rails 3.0.7.

The app works fine but unexpectedly the respawning fails:

First we start seeing errors like this (a lot):

[ pid=20143 thr=47969805936176 file=ext/apache2/Hooks.cpp:789
time=2011-05-13 08:52:15.11 ]: Either the vistor clicked on the 'Stop'
button in the web browser, or the visitor's connection has stalled and
couldn't receive the data that Apache is sending to it. As a result,
you will probably see a 'Broken Pipe' error in this log file. Please
ignore it, this is normal. You might also want to increase Apache's
TimeOut configuration option if you experience this problem often.

TimeOut is 300. Second the rewspaning fails (a lot):

[ pid=21681 thr=194380080 file=utils.rb:176 time=2011-05-13
09:01:54.429 ]: *** Exception LoadError in
PhusionPassenger::ClassicRails::ApplicationSpawner (No such file to
load -- application) (process 21681, thread #<Thread:0x172c0260>):

this app works fine normally . This is a high load app, so it tries
respawning a lot, freezing the server ruby response.

and then :

[Hongli Lai]

Does Passenger Standalone work for you?

> --
> You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.
>
>

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[testes]

HI,

Unfortunately we cant use it, we are downgrading to 2.2.15 to see how
it performs. I will post if it works.
The previous version of this app with rails 2.x and passenger 2.2.x runs fine.

2011/5/18 Hongli Lai <hon...@phusion.nl>:

[Steffen]

I have the same problem!

First the client reported that he gets 503 Service Unavailable from
time to time, maybe 5-10 times a day.

I had the app setup to 3 MinInstances. In Unionstation app I saw
yesterday, that it always just has 3 processes running, even though
during the day more should be needed.
Therefore I changed MinInstances to 10, hoping to get rid of the
problem and eliminate the need of respawning, but it still seems to
try to respawn.

In the apache error log, I see the following errors as well:
[ pid=578 thr=1125484880 file=ext/apache2/Hooks.cpp:789
time=2011-06-01 12:51:59.380 ]: Either the vistor clicked on the

'Stop' button in the web browser, or the visitor's connection has
stalled and couldn't receive the data that Apache is sending to it. As
a result, you will probably see a 'Broken Pipe' error in this log
file. Please ignore it, this is normal. You might also want to
increase Apache's TimeOut configuration option if you experience this
problem often.

My apache passenger config:
LoadModule passenger_module /opt/ree/lib/ruby/gems/1.8/gems/
passenger-3.0.7/ext/apache2/mod_passenger.so
PassengerRoot /opt/ree/lib/ruby/gems/1.8/gems/passenger-3.0.7
PassengerRuby /opt/ree/bin/ruby

PassengerDefaultUser root

PassengerMaxPoolSize 50

(/opt/ree is symlink to /opt/ruby-enterprise-1.8.7-2011.01)

In my vhost file I have:
PassengerMinInstances 10
and
PassengerPreStart url (after the virtualhost specification)



After graceful restart of apache, I'm seeing the following in the
error log:
[Tue May 31 14:29:07 2011] [notice] Apache/2.2.9 (Debian)
Phusion_Passenger/3.0.7 configured -- resuming normal operations
/usr/bin/env: ruby: No such file or directory
/usr/bin/env: ruby: No such file or directory
/usr/bin/env: ruby: No such file or directory

I'm on Debian 5.0.6, 2.6.26-2-amd64.

Before yesterday I had passenger 3.0.4 installed, which had the same
problem, now using 3.0.7.

I'm also running rvm for ruby 1.9 on the machine, a standalone
passenger is handling my 1.9 app.

Thanks for any advice!

[Steffen]

Regarding the following error message which I get after restarting
apache
[Wed Jun 01 16:23:22 2011] [notice] Apache/2.2.9 (Debian)

Phusion_Passenger/3.0.7 configured -- resuming normal operations
/usr/bin/env: ruby: No such file or directory
/usr/bin/env: ruby: No such file or directory
/usr/bin/env: ruby: No such file or directory

I have 3 vhosts/rails apps running, that's why it outputs 3 times "/
usr/bin/env: ruby: No such file or directory".
Wonder what it causes and if this is the reason for the spawning
problem.
All apps are Rails 2.1 apps (same app basically).

[Steffen]

Anybody any idea what I can do here?

[testes]

Hi,

At the end we downgraded to 2.2.15. We were unable to fix the problem .
I dont know if it is a bug or no in 3.0.7 but it works fine in 2.2.15

2011/6/6 Steffen <steffen...@gmail.com>:

> Anybody any idea what I can do here?
>

[Steffen]

Hmmm, Phusion guys, any idea?
What can I do to help you localize the problem?

Thanks!!

But downgrading is a good idea for the meanwhile I guess.

On Jun 7, 4:16 am, testes <elchicopes...@gmail.com> wrote:
> Hi,
>
> At the end we downgraded to 2.2.15. We were unable to fix the problem .
> I dont know if it is a bug or no in 3.0.7 but it works fine in 2.2.15
>

> 2011/6/6 Steffen <steffen.hil...@gmail.com>:

[Steffen]

So I didn't have a ruby in /usr/bin/ruby.
Symlinked my REE ruby to /usr/bin/ruby and now spawning seems to work.

[testes]

Atm i can't reproduce it, but if it comes to be useful to fix the (not)bug ,
we were using a custom ruby location.

2011/6/20, Steffen <steffen...@gmail.com>:

> So I didn't have a ruby in /usr/bin/ruby.
> Symlinked my REE ruby to /usr/bin/ruby and now spawning seems to work.
>

[cmyk]

We have the same issue. The app (http://www.peakhunter.com) just
freezes.
We will downgrade to 2.2.15.
I don't think 3.0.8 fixes this problem, or should it?

[Steffen Hiller]

Not sure about 3.0.8.
We switched first to 2.2.15, which solved it.
But we still continued to get occasional 503 errors,
which made us switch from apache to nginx, which solved that.

[Abel Martin]

We have the same issue on our server. Is there another option besides
dumping Apache? I really don't want to change my web server in order
to get around this issue.

We're using...
Cent OS 5.2
Passenger 3.0.2
Ruby 1.8.7-p22 (yes, I know we're working on it)
Rails 2.3.2 (yes, again, I know. We're working on that too)
Apache 2.0.63 (this may or may not change depending on if it needs
to.)

...and our errors look like a ton of these in a row:

[ pid=2309 thr=139648351237904 file=ext/apache2/Hooks.cpp:863
time=2011-10-04 17:52:28.99 ]: Unexpected error in mod_passenger:
Cannot connect to Unix socket '/tmp/passenger.1.0.2742/generation-7/

socket': No such file or directory (2)
Backtrace:
in 'Passenger::ApplicationPool::Client*
Passenger::ApplicationPool::Client::connect(const std::string&, const
std::string&, const Passenger::StaticString&)' (Client.h:446)
in 'Passenger::ApplicationPool::Client*
Hooks::getApplicationPool()' (Hooks.cpp:271)
in 'Passenger::SessionPtr Hooks::getSession(const
Passenger::PoolOptions&)' (Hooks.cpp:297)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)

Any help would be great.
Thanks!
-a

PS: Sorry about the repost, but I wanted to add the operating system &
an example of our error message.

[Abel Martin]

If it helps, here's our Passenger conf include:

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/
passenger-3.0.2/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.2
PassengerRuby /usr/bin/ruby

PassengerMaxPoolSize 20
PassengerUseGlobalQueue on

RailsAppSpawnerIdleTime 0
RailsFrameworkSpawnerIdleTime 0

Thanks,
-a

[Abel Martin]

I found the culprit for the huge stack trace error: Apache restarts
scheduled by cPanel that we were unaware of.

We're using cPanel 11.30.x.x. Since cPanel 11.25.x.x, they
implemented a mandatory Apache restart every 2 hours or so (I found
the article on how to fix it here:
http://techiezdesk.wordpress.com/2011/01/08/apache-graceful-restart-requested-every-two-hours/).
We didn't realize ...

(A) that the restarts were happening due to all of the "Unexpected
error in mod_passenger: Cannot connect to Unix socket" errors we saw
constantly pouring into the log. I assumed that this merely happened
due to all of the Passenger errors. I finally saw the pattern last
night and started troubleshooting Apache/cPanel.
(B) the effect this restart has Passenger.

If it helps any of you, tripple check to see if Apache is getting
restarted. We're still getting the...
Cannot connect to Unix socket '/tmp/passenger.1.0.3207/generation-0/

socket': No such file or directory (2)

...errors, but at least we stopped the huge stack trace dump.

On Oct 6, 7:03 pm, Abel Martin <abel.mar...@gmail.com> wrote:
> If it helps, here's our Passenger conf include:
>
>   LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/
> passenger-3.0.2/ext/apache2/mod_passenger.so
>   PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.2
>   PassengerRuby /usr/bin/ruby
>
>   PassengerMaxPoolSize 20
>   PassengerUseGlobalQueue on
>
>   RailsAppSpawnerIdleTime 0
>   RailsFrameworkSpawnerIdleTime 0
>
> Thanks,
> -a
>

[testes]

Hi,

Yes, you are right. As far as i know cpanel does graceful restarts
every two hours, no hard restart ( nothing weird ). Imo a graceful
restart shouldnt cause this problems anyway. I forgot to say it in my
first post but it was a cpanel and, yes, the problem appears when you
do graceful restarts. The only way to troobleshoot it was a hard
restart.

I have been seeing this problem with other apps (much less then the
one that caused the original problem) and i am starting to think maybe
it is related with timeout and spawn time. But i still could not test
it

2011/10/19, Abel Martin <abel....@gmail.com>: