passenger hangs forever, no logs, how to debug?

Showing 1-19 of 19 messages
passenger hangs forever, no logs, how to debug? Viktor Trón 4/28/11 11:15 AM

Apache/2.2.3 with passenger 3.0.7 (on a prehistoric SLES11 linux)

testing with simpleton rack app, with config.ru

app = proc do |env|
[200, { "Content-Type" => "text/html" }, ["hello, world"]]
end
run app

then browsers (or curl) hang forever on all requests

vhost conf:

<VirtualHost *:80>
DocumentRoot /srv/www/rails_app_test/public
<Directory "/srv/www/rails_app_test/public">
Options FollowSymLinks -MultiViews
Order allow,deny
Allow from All
</Directory>
</VirtualHost>

everthing under /srv/www has correct permissions on the file system

if instead of this hello world app, i test with a rails 3 app skeleton, it serves up the index page, but all other requests hang 
leaving no trace in either apache access/error logs or rails app log

does this ring a bell to anyone? all help or hints to debug appreciated

Vik
Re: [phusion-passenger] passenger hangs forever, no logs, how to debug? Hongli Lai 4/28/11 12:44 PM
Try passenger-status.

> --
> You received this message because you are subscribed to the Google Groups
> "Phusion Passenger Discussions" group.
> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to
> phusion-passen...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/phusion-passenger?hl=en.
>

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

Re: [phusion-passenger] passenger hangs forever, no logs, how to debug? Viktor Trón 5/3/11 4:36 AM

On Thursday, 28 April 2011 20:44:25 UTC+1, Hongli Lai wrote:
Try passenger-status
 
if I restart apache, it shows this rather uninteresting snippet below, but if i
put a request in that is passed on to passenger, it hangs
and passenger-status hangs as well. 

What is more, even if I abort the request (either in browser or curl), passenger-status still hangs
so it looks like a request is paralysing it.

does this ring a bell to anyone?
i have absolutely no clue how to go on, no logs, no debug, no error, nothing

thanks

$ rvmsudo passenger-status -v
----------- General information -----------
max = 6
count = 0
active = 0
inactive = 0
Waiting on global queue: 0

Re: passenger hangs forever, no logs, how to debug? foreverman 5/3/11 7:33 AM
Try to kill the frozen rails process by issuing 'kill -6
rails_process_id', then look at log.
See "7.3. Debugging frozen applications" in this document:
http://www.modrails.com/documentation/Users%20guide%20Apache.html.
Re: [phusion-passenger] passenger hangs forever, no logs, how to debug? Hongli Lai 5/3/11 4:15 PM
On Tue, May 3, 2011 at 1:36 PM, Viktor Trón <vikto...@gmail.com> wrote:
>
> On Thursday, 28 April 2011 20:44:25 UTC+1, Hongli Lai wrote:
>>
>> Try passenger-status
>
>
> if I restart apache, it shows this rather uninteresting snippet below, but
> if i
> put a request in that is passed on to passenger, it hangs
> and passenger-status hangs as well.

Try passenger-status --show=backtraces
Also try setting PassengerLogLevel to 2.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

Re: passenger hangs forever, no logs, how to debug? vsagarv 5/5/11 7:26 AM
I've been seeing similar issues as well. I've recently moved my Rails
app from ruby1.8.7+Rails2.3.4+Passenger2.x to the env mentioned below.
Frequently, after a fresh deploy, I see that my app fails to respond.

> Try passenger-status --show=backtraces

"rvmsudo bundle exec passenger-status' hangs (backtraces follow),
while passenger-memory-stats runs fine. There are no Rails processes
running as seen on 'ps ax | fgrep Rails'. The passenger max pool size
is at 6. (Underlying mysql pool size is far greater at 20; so can't be
a DB hang).

===
Env: {rvm + ruby1.9.2 + Rails2.3.11} / {nginx 0.8.54 + passenger
3.0.3} / {Linux 2.6.31-302-ec2 #7-Ubuntu SMP Tue Oct 13 19:06:04 UTC
2009 i686 unknown unknown GNU/Linux}

===
$ nginx -V
nginx version: nginx/0.8.54
built by gcc 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx --with-http_ssl_module --with-
pcre=/tmp/root-passenger-1271/pcre-8.10 --add-module=/home/ubuntu/.rvm/
gems/ruby-1.9.2-p180@rails/gems/passenger-3.0.3/ext/nginx

===
$ rvmsudo bundle exec passenger-status --show=backtraces
Thread 'Main thread':
     in 'void Server::mainLoop()' (HelperAgent.cpp:915)
     in 'int main(int, char**)' (HelperAgent.cpp:976)

Thread 'Client thread 1':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'MessageServer thread':
     in 'void Passenger::MessageServer::mainLoop()' (MessageServer.h:
543)

Thread 'Client thread 2':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 3':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 4':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 5':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 6':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 7':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 8':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 9':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 10':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 11':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 12':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 13':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 14':
     in 'Passenger::ProcessPtr
Passenger::SpawnManager::sendSpawnCommand(const
Passenger::PoolOptions&)' (SpawnManager.h:347)
     in 'virtual Passenger::ProcessPtr
Passenger::SpawnManager::spawn(const
Passenger::PoolOptions&)' (SpawnManager.h:573)
     in
'std::pair<boost::shared_ptr<Passenger::ApplicationPool::Pool::ProcessInfo>,
boost::shared_ptr<Passenger::ApplicationPool::Pool::Group> >
Passenger::ApplicationPool::Pool::checkoutWithoutLock(boost::unique_lock<boost::timed_mutex>&,
const Passenger::PoolOptions&)' (Pool.h:963)
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 15':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 16':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 18':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 17':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 20':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 19':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 23':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 21':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 22':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'Client thread 24':
     in 'virtual Passenger::SessionPtr
Passenger::ApplicationPool::Pool::get(const
Passenger::PoolOptions&)' (Pool.h:1136)
     in 'void
Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:
472)
     in 'void Client::threadMain()' (HelperAgent.cpp:595)

Thread 'ApplicationPool analytics collector':
     in 'void
Passenger::ApplicationPool::Pool::analyticsCollectionThreadMainLoop()' (Pool.h:
790)

Thread 'ApplicationPool cleaner':
     (empty)
Thread 'MessageServer client thread 35':
     in 'void
Passenger::ApplicationPool::Server::processInspect(Passenger::MessageServer::CommonClientContext&,
Passenger::ApplicationPool::Server::SpecificContext*, const
std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >&)' (Server.h:400)
     in 'void
Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:
470)

Thread 'MessageServer client thread 37':
     in 'void
Passenger::ApplicationPool::Server::processInspect(Passenger::MessageServer::CommonClientContext&,
Passenger::ApplicationPool::Server::SpecificContext*, const
std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >&)' (Server.h:400)
     in 'void
Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:
470)

Thread 'MessageServer client thread 38':
     in 'void
Passenger::ApplicationPool::Server::processInspect(Passenger::MessageServer::CommonClientContext&,
Passenger::ApplicationPool::Server::SpecificContext*, const
std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >&)' (Server.h:400)
     in 'void
Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:
470)

Thread 'MessageServer client thread 39':
     in 'void
Passenger::ApplicationPool::Server::processInspect(Passenger::MessageServer::CommonClientContext&,
Passenger::ApplicationPool::Server::SpecificContext*, const
std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >&)' (Server.h:400)
     in 'void
Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:
470)

Thread 'MessageServer client thread 40':
     in 'virtual bool
Passenger::BacktracesServer::processMessage(Passenger::MessageServer::CommonClientContext&,
boost::shared_ptr<Passenger::MessageServer::ClientContext>&, const
std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > >
>&)' (BacktracesServer.h:47)
     in 'void
Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:
470)
===
Aborting with ctrl-c gives this familiar trace:

$ rvmsudo bundle exec passenger-status
^C/home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/passenger-3.0.3/
lib/phusion_passenger/message_channel.rb:221:in `read': Interrupt
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/lib/phusion_passenger/message_channel.rb:221:in
`read_scalar'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/lib/phusion_passenger/message_client.rb:137:in
`read_scalar'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/lib/phusion_passenger/message_client.rb:93:in `status'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/lib/phusion_passenger/admin_tools/server_instance.rb:
246:in `status'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/bin/passenger-status:46:in `block in show_status'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/lib/phusion_passenger/admin_tools/server_instance.rb:
214:in `connect'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/bin/passenger-status:42:in `show_status'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/bin/passenger-status:192:in `start'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/gems/
passenger-3.0.3/bin/passenger-status:211:in `<top (required)>'
        from /home/ubuntu/app_home/gshared/bundle/ruby/1.9.1/bin/
passenger-status:19:in `load'
===

$ rvmsudo bundle exec passenger-memory-stats
------------- Apache processes -------------
*** WARNING: The Apache executable cannot be found.
Please set the APXS2 environment variable to your 'apxs2' executable's
filename, or set the HTTPD environment variable to your 'httpd' or
'apache2' executable's filename.


-------- Nginx processes ---------
PID   PPID  VMSize  Private  Name
----------------------------------
7791  1     6.0 MB  0.2 MB   nginx: master process /opt/nginx/sbin/
nginx
7792  7791  7.9 MB  2.0 MB   nginx: worker process
7793  7791  7.3 MB  1.4 MB   nginx: worker process
### Processes: 3
### Total private dirty RSS: 3.56 MB


---- Passenger processes ----
PID   VMSize   Private  Name
-----------------------------
4549  10.3 MB  0.1 MB   Passenger spawn server
7752  4.1 MB   0.2 MB   PassengerWatchdog
7755  18.6 MB  1.7 MB   PassengerHelperAgent
7757  10.3 MB  0.1 MB   Passenger spawn server
7786  9.2 MB   0.5 MB   PassengerLoggingAgent
### Processes: 5
### Total private dirty RSS: 2.63 MB
===


On May 4, 4:15 am, Hongli Lai <hon...@phusion.nl> wrote:
> On Tue, May 3, 2011 at 1:36 PM, Viktor Trón <viktor.t...@gmail.com> wrote:
>
> > On Thursday, 28 April 2011 20:44:25 UTC+1, Hongli Lai wrote:
>
> >> Try passenger-status
>
> > if I restart apache, it shows this rather uninteresting snippet below, but
> > if i
> > put a request in that is passed on to passenger, it hangs
> > and passenger-status hangs as well.
>
> Try passenger-status --show=backtraces
> Also try setting PassengerLogLevel to 2.
>
> --
> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Hongli Lai 5/5/11 8:41 AM
Your server is locked up because an application process froze while
it's being spawned. You need to take a look at that.

> --
> You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.
>
>

--

Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl


Chamber of commerce no: 08173483 (The Netherlands)

Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Viktor Trón 5/5/11 9:23 AM

Hello all, thanks for your replies, here's the thing:

I installed crash-watch and set PassengerLogLevel 3 in the vhost conf

after sudo /etc/init.d/apache2 start
> ps auxwww|grep Passen|grep -v grep
root 14189 0.0 0.0 10456 1964 ? Ssl 14:23 0:00 PassengerWatchdog
root 14192 0.0 0.0 19200 2404 ? Sl 14:23 0:00 PassengerHelperAgent
root 14194 0.5 0.0 26964 7800 ? Sl 14:23 0:00 Passenger spawn server
nobody 14197 0.0 0.0 19100 3168 ? Sl 14:23 0:00 PassengerLoggingAgent

this looks normal as does apache error log 

now I put request in for http://VHOST/
i see the rails app index page.

and the request shows up in the access log

now i remove that page, the same request stalls forever

> ps auxwww|grep Passen|grep -v grep
root 14189 0.0 0.0 10456 1964 ? Ssl 14:23 0:00 PassengerWatchdog
root 14192 0.0 0.0 19268 2540 ? Sl 14:23 0:00 PassengerHelperAgent
root 14194 0.1 0.0 27628 8496 ? Sl 14:23 0:00 Passenger spawn server
nobody 14197 0.0 0.0 19100 3168 ? Sl 14:23 0:00 PassengerLoggingAgent
root 14262 0.0 0.0 28696 7672 ? Sl 14:25 0:00 Passenger ApplicationSpawner: /srv/www/rails_app_test.rails

error log shows:

[ pid=16028 thr=1082411328 file=ext/common/ApplicationPool/Pool.h:939 time=2011-05-05 14:59:03.11 ]: Spawning a process for
/srv/www/rails_app_test.rails because there are none for this app group
[ pid=16028 thr=1082411328 file=ext/common/ApplicationPool/../SpawnManager.h:291 time=2011-05-05 14:59:03.11 ]: Spawning a
new application process for /srv/www/rails_app_test.rails...


apache access_log shows nothing
rails app production.log is empty
now I do
kill -6 14194
(nothing happens, the request still stalled, and passenger (quite correctly launches another spawner)

after i shut down apache the two spawner processes get stuck and need to be killed

when the request is hanging, passenger-status hangs too, interrupt signal gives this trace:

 
> rvmsudo bundle exec passenger-status 
/usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/message_channel.rb:221:in `read': Interrupt
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/message_channel.rb:221:in `read_scalar'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/message_client.rb:137:in `read_scalar'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/message_client.rb:93:in `status'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/admin_tools/server_instance.rb:246:in `status'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passenger-status:46:in `block in show_status'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion_passenger/admin_tools/server_instance.rb:214:in `connect'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passenger-status:42:in `show_status'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passenger-status:192:in `start'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passenger-status:211:in `<top (required)>'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/bin/passenger-status:19:in `load'
from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/bin/passenger-status:19:in `<main>'


Hongli, i found a very similar thread here:
your reply in both cases is that:

Your server is locked up because an application process froze while
it's being spawned. You need to take a look at that.

however my (trivial test) application does launch with webrick (tried in production mode), so if freezing is app-internal problem it is still passenger specific, which I doubt since it is only a scaffold created with rails, plus
i also tried with a hello world rack app and the same happens.

I am simply unable to move on, any hints appreciated

thanks a lot
Vik
Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Hongli Lai 5/5/11 10:54 AM
On Thu, May 5, 2011 at 6:23 PM, Viktor Trón <vikto...@gmail.com> wrote:
> Hongli, i found a very similar thread here:
> https://groups.google.com/d/topic/phusion-passenger/tZT5FdGFslk/discussion
> your reply in both cases is that:
> Your server is locked up because an application process froze while
> it's being spawned. You need to take a look at that.
> however my (trivial test) application does launch with webrick (tried in
> production mode), so if freezing is app-internal problem it is still
> passenger specific, which I doubt since it is only a scaffold created with
> rails, plus
> i also tried with a hello world rack app and the same happens.
> I am simply unable to move on, any hints appreciated

Try putting

STDERR.puts "starting"

at the beginning of your hello world Rack app's config.ru. Do you ever
see this message appearing in your web server error log?

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

Re: passenger hangs forever, no logs, how to debug? vsagarv 5/5/11 12:48 PM
Hongli,

> Your server is locked up because an application process froze while it's being spawned. You need to take a look at that.

Okay. Strange. Does "Your server" mean nginx? Or the spawn server? If
that is so, am unclear why things come back to normalcy after I force
stop-start nginx.

> Do you ever see this message appearing in your web server error log?

I've been seeing app logs coming into nginx logs. Is this a related
problem?

@Viktor: Our logs are very similar. My cap deploys used to be quite
fine with my earlier env (ruby1.8.7+Rails2.3.4+Passenger2.2.10).

Thanks
--
sAgar

On May 5, 10:54 pm, Hongli Lai <hon...@phusion.nl> wrote:
> On Thu, May 5, 2011 at 6:23 PM, Viktor Trón <viktor.t...@gmail.com> wrote:
> > Hongli, i found a very similar thread here:
> >https://groups.google.com/d/topic/phusion-passenger/tZT5FdGFslk/discu...
> > your reply in both cases is that:
> > Your server is locked up because an application process froze while
> > it's being spawned. You need to take a look at that.
> > however my (trivial test) application does launch with webrick (tried in
> > production mode), so if freezing is app-internal problem it is still
> > passenger specific, which I doubt since it is only a scaffold created with
> > rails, plus
> > i also tried with a hello world rack app and the same happens.
> > I am simply unable to move on, any hints appreciated
>
> Try putting
>
> STDERR.puts "starting"
>
> at the beginning of your hello world Rack app's config.ru. Do you ever
> see this message appearing in your web server error log?
>
> --
> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Hongli Lai 5/5/11 12:54 PM
On Thu, May 5, 2011 at 9:48 PM, vsagarv <vijay...@gmail.com> wrote:
> Okay. Strange. Does "Your server" mean nginx? Or the spawn server? If
> that is so, am unclear why things come back to normalcy after I force
> stop-start nginx.

Phusion Passenger currently assumes that apps behave well during
startup. If an app freezes during startup then the lock on the Phusion
Passeger application pool will never be released, resulting in a stall
like the one you're experiencing.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl


Chamber of commerce no: 08173483 (The Netherlands)

Re: passenger hangs forever, no logs, how to debug? vsagarv 5/5/11 12:57 PM
> now i remove that page, the same request stalls forever

Interesting. As part of cap deploys, we do some symlinking of files
(all cap releases including 'current' share a common 'shared'
directory for things such as logs & config. During a deploy, some
symlinks are deleted and new ones are created. And that might be a
reason for the hang. As if passenger/app is holding an open fd to an
underlying file, which froze it all up.

On May 5, 9:23 pm, Viktor Trón <viktor.t...@gmail.com> wrote:
> Hello all, thanks for your replies, here's the thing:
>
> I installed crash-watch and set PassengerLogLevel 3 in the vhost conf
>
> after sudo /etc/init.d/apache2 start> ps auxwww|grep Passen|grep -v grep
>
> root 14189 0.0 0.0 10456 1964 ? Ssl 14:23 0:00 PassengerWatchdog
> root 14192 0.0 0.0 19200 2404 ? Sl 14:23 0:00 PassengerHelperAgent
> root 14194 0.5 0.0 26964 7800 ? Sl 14:23 0:00 Passenger spawn server
> nobody 14197 0.0 0.0 19100 3168 ? Sl 14:23 0:00 PassengerLoggingAgent
>
> this looks normal as does apache error log
>
> now I put request in forhttp://VHOST/
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/message_channel.rb:221:in
> `read': Interrupt
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/message_channel.rb:221:in
> `read_scalar'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/message_client.rb:137:in
> `read_scalar'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/message_client.rb:93:in
> `status'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/admin_tools/server_instance.rb:246:in
> `status'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passeng er-status:46:in
> `block in show_status'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/lib/phusion _passenger/admin_tools/server_instance.rb:214:in
> `connect'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passeng er-status:42:in
> `show_status'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passeng er-status:192:in
> `start'
> from
> /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/gems/passenger-3.0.7/bin/passeng er-status:211:in
> `<top (required)>'
> from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/bin/passenger-status:19:in
> `load'
> from /usr/local/rvm/gems/ruby-1.9.2-p180@Repowt/bin/passenger-status:19:in
> `<main>'
>
> Hongli, i found a very similar thread here:https://groups.google.com/d/topic/phusion-passenger/tZT5FdGFslk/discu...
Re: passenger hangs forever, no logs, how to debug? vsagarv 5/5/11 1:02 PM
> Phusion Passenger currently assumes that apps behave well during startup.
Fair enough expectation, Hongli. Your replies are much appreciated.
Will keep digging around.

And how about the app logs getting mixed up with nginx logs? Is that a
side-effect of this freeze up?

On May 6, 12:54 am, Hongli Lai <hon...@phusion.nl> wrote:
> On Thu, May 5, 2011 at 9:48 PM, vsagarv <vijayasa...@gmail.com> wrote:
> > Okay. Strange. Does "Your server" mean nginx? Or the spawn server? If
> > that is so, am unclear why things come back to normalcy after I force
> > stop-start nginx.
>
> Phusion Passenger currently assumes that apps behave well during
> startup. If an app freezes during startup then the lock on the Phusion
> Passeger application pool will never be released, resulting in a stall
> like the one you're experiencing.
>
> --
> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Hongli Lai 5/5/11 1:19 PM
On Thu, May 5, 2011 at 10:02 PM, vsagarv <vijay...@gmail.com> wrote:
> Fair enough expectation, Hongli. Your replies are much appreciated.
> Will keep digging around.
>
> And how about the app logs getting mixed up with nginx logs? Is that a
> side-effect of this freeze up?

No. That typically means that your log/production.log is writable by
the Rails process, so Rails resorts to sending all logs to STDERR
which is redirected to the web server error log.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl


Chamber of commerce no: 08173483 (The Netherlands)

Re: passenger hangs forever, no logs, how to debug? vsagarv 5/5/11 1:35 PM
> No. That typically means that your log/production.log is writable by the Rails process,
Hmmm I read it as 'is *not* writable'. And then debug it from there
(we do some symlink shifts as part of the cap deploy; it is possible
there's an undetected catch-22 somewhere). Thanks.
--
sAgar

On May 6, 1:19 am, Hongli Lai <hon...@phusion.nl> wrote:
> On Thu, May 5, 2011 at 10:02 PM, vsagarv <vijayasa...@gmail.com> wrote:
> > Fair enough expectation, Hongli. Your replies are much appreciated.
> > Will keep digging around.
>
> > And how about the app logs getting mixed up with nginx logs? Is that a
> > side-effect of this freeze up?
>
> No. That typically means that your log/production.log is writable by
> the Rails process, so Rails resorts to sending all logs to STDERR
> which is redirected to the web server error log.
>
> --
> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Hongli Lai 5/5/11 10:19 PM
On Thu, May 5, 2011 at 10:35 PM, vsagarv <vijay...@gmail.com> wrote:
>> No. That typically means that your log/production.log is writable by the Rails process,
> Hmmm I read it as 'is *not* writable'.

Yes sorry, typo.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail
: in...@phusion.nl


Chamber of commerce no: 08173483 (The Netherlands)

Re: [phusion-passenger] Re: passenger hangs forever, no logs, how to debug? Viktor Trón 5/11/11 5:03 AM

> Try putting
> STDERR.puts "starting"

Hi Hongli, 
I put it in, it does NOT appear, so the spawner does not make it until startup. 
surely not a problem with app, am still in the dark.

thanks
Vik
Re: passenger hangs forever, no logs, how to debug? Viktor Trón 5/13/11 3:49 AM
sorry for bumping this post.
since it is still unresolved and in fact a mystery, i opened this up on http://stackoverflow.com/questions/5976384/phusion-passenger-mod-rails-on-apache-hanging
hoping to hear from you 
Vik
Re: passenger hangs forever, no logs, how to debug? Joe Dollard 8/27/12 7:54 AM
Sorry for the very late addition to this thread, but I ran into the same behavior and thought I'd post details here incase someone else runs into this.

Passenger would hang when I'd make a request to any application where passenger was enabled; even with the trivial 4 line rack example app on the passenger site.  I traced through the passenger code and noticed it would hang when it called the switch_user method in passenger_native_support.c (the call is from PhusionPassenger::Utils).  I tried replacing the call into this method with the equivalent calls from Process and Process::Sys (i.e. initgroups, getgid and setuid) and observed the call to setuid and setgid would both hang in my environment.

I found I could work around the issue by setting 'PassengerUserSwitching' to off.

Regards,
Joe