nginx Passenger helper fails to spawn Rails instances some mornings!

[Christophe Porteneuve]

Hi all,

Over the past week I've had a couple mornings where, at 6:30 or 6:40,
the PassengerHelperAgent would just break. I get a Watchdog line
saying "Some Phusion Passenger agent processes did not exit in time,
forcefully shutting down all", then no instances will ever be spawned.
And the instances loaded at that time do get killed. So the website
is down for a few hours until I get to the office, kill the passenger
helper processes, and restart nginx.

The cronjobs I have running at that time don't tweak/restart/anything
nginx and Passenger. They do run the app for batch processing, but
through a script/runner CLI (and this processing works fine and
happens every 15 mins). The actual daily processing at this time
doesn't touch the app, it's backup stuff.

I use nginx 0.8.54 with Passenger 3.0.4 over REE 1.8.7 (2010.02) on
Ubuntu Server 10.04 LTS.

I'll be happy to post any config info or extra details you may need.
This is a major hassle, as it means the site can break on any given
morning and needs some crutch monitoring to basically kill everything
and force a restart. What gives?

Best,

[Hongli Lai]

On Mon, Mar 14, 2011 at 12:12 PM, Christophe Porteneuve
<chris...@ciblo.net> wrote:
> Hi all,
>
> Over the past week I've had a couple mornings where, at 6:30 or 6:40,
> the PassengerHelperAgent would just break. I get a Watchdog line
> saying "Some Phusion Passenger agent processes did not exit in time,
> forcefully shutting down all", then no instances will ever be spawned.
> And the instances loaded at that time do get killed.  So the website
> is down for a few hours until I get to the office, kill the passenger
> helper processes, and restart nginx.

This message implies that the web server itself shut down, gracefully
even, so the watchdog follows. When your problem occurs is your web
server still alive and can it serve static assets?

For now you can try switching to a reverse proxy setup with Phusion
Passenger Standalone. Maybe that's more stable for you.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)