Graceful rolling restart with Passenger and Capistrano?

[casey]

I recently switched from a cluster of servers running Thin to a
cluster running Apache+Passenger. I'm using haproxy to load-balance
across my machines.

I used to be able to redeploy/restart my application while it was
"hot" without affecting any of the users. It worked like so:

* capistrano deploy, the running Thin instances would continue to use
the old version of the app
* gracefully shut down Thins and then restart them one group at a
time. They would go down after completing current requests and come
back up with the new version of the app.

I'm having trouble doing this with Passenger because deploying new
code (which changes symlinks to myapp/current) seems to trigger
Passenger to restart. Is this normal behavior?

I'd be happy just restarting httpd to reload as long as I knew that
stopping Apache would gracefully shutdown Passengers.

[Dan Cheail]

Casey:

My experience with passenger is that any currently running processes
will continue to run even after a re-start has been triggered; my
users have never noticed downtime, even when doing several repeated
deploys. I have noticed occaions where i've got live passengers of
different versions, but all processes went to the most recent
versions, and the old passenger instances eventually died out anyway.


--
Dan Cheail
Chief Geek, Codeape.
phone: 0402 114 697
web: http://codeape.net

[amos]

we had to switch to rolling restarts in capistrano in order to avoid
downtime. word is the passenger guys are working on a fix.

[casey]

So you deploy to one server at a time instead of letting capistrano
deploy to all at once?

I just tried another deploy (under load) and had an outage again - it
seems that Passenger does *something* when "current" symlink is moved
and doesn't recover properly when under load.

[amos]

yes, in my experience passenger can't handle replacing the symlink
under load which is why we use 'httpd restart'.

we wrote a special capistrano hook to restart a server, wait 20
seconds and go onto the next. unfortunately, now it takes forever to
deploy new versions of our code.

[casey]

Thanks very much amos,

It was good to know that I'd have to solve the problem myself. I
ended up doing something similar. For each server in my cluster,
Capistrano...

* Deletes a file that my load balancer (HAProxy) is looking for from /
public which causes it to be removed from rotation
* Waits 5 seconds for any requests to finish
* Regular capistrano update
* Put the /public/file back so that the server goes back into rotation
* Wait 20 seconds for Passenger to warm up, then move on

[casey]

oops - forgot the important part - an Apache restart after the Cap
update. I touch Passenger's restart.txt but there are no requests
coming in and these Apaches are dedicated to passenger, might as well
start fresh.