I recently switched from a cluster of servers running Thin to a
cluster running Apache+Passenger. I'm using haproxy to load-balance
across my machines.
I used to be able to redeploy/restart my application while it was
"hot" without affecting any of the users. It worked like so:
* capistrano deploy, the running Thin instances would continue to use
the old version of the app
* gracefully shut down Thins and then restart them one group at a
time. They would go down after completing current requests and come
back up with the new version of the app.
I'm having trouble doing this with Passenger because deploying new
code (which changes symlinks to myapp/current) seems to trigger
Passenger to restart. Is this normal behavior?
I'd be happy just restarting httpd to reload as long as I knew that
stopping Apache would gracefully shutdown Passengers.
My experience with passenger is that any currently running processes
will continue to run even after a re-start has been triggered; my
users have never noticed downtime, even when doing several repeated
deploys. I have noticed occaions where i've got live passengers of
different versions, but all processes went to the most recent
versions, and the old passenger instances eventually died out anyway.
> I recently switched from a cluster of servers running Thin to a
> cluster running Apache+Passenger. I'm using haproxy to load-balance
> across my machines.
> I used to be able to redeploy/restart my application while it was
> "hot" without affecting any of the users. It worked like so:
> * capistrano deploy, the running Thin instances would continue to use
> the old version of the app
> * gracefully shut down Thins and then restart them one group at a
> time. They would go down after completing current requests and come
> back up with the new version of the app.
> I'm having trouble doing this with Passenger because deploying new
> code (which changes symlinks to myapp/current) seems to trigger
> Passenger to restart. Is this normal behavior?
> I'd be happy just restarting httpd to reload as long as I knew that
> stopping Apache would gracefully shutdown Passengers.
> My experience with passenger is that any currently running processes
> will continue to run even after a re-start has been triggered; my
> users have never noticed downtime, even when doing several repeated
> deploys. I have noticed occaions where i've got live passengers of
> different versions, but all processes went to the most recent
> versions, and the old passenger instances eventually died out anyway.
> > I recently switched from a cluster of servers running Thin to a
> > cluster running Apache+Passenger. I'm using haproxy to load-balance
> > across my machines.
> > I used to be able to redeploy/restart my application while it was
> > "hot" without affecting any of the users. It worked like so:
> > * capistrano deploy, the running Thin instances would continue to use
> > the old version of the app
> > * gracefully shut down Thins and then restart them one group at a
> > time. They would go down after completing current requests and come
> > back up with the new version of the app.
> > I'm having trouble doing this with Passenger because deploying new
> > code (which changes symlinks to myapp/current) seems to trigger
> > Passenger to restart. Is this normal behavior?
> > I'd be happy just restarting httpd to reload as long as I knew that
> > stopping Apache would gracefully shutdown Passengers.
So you deploy to one server at a time instead of letting capistrano
deploy to all at once?
I just tried another deploy (under load) and had an outage again - it
seems that Passenger does *something* when "current" symlink is moved
and doesn't recover properly when under load.
On Dec 17, 2:31 pm, amos <famosea...@gmail.com> wrote:
yes, in my experience passenger can't handle replacing the symlink
under load which is why we use 'httpd restart'.
we wrote a special capistrano hook to restart a server, wait 20
seconds and go onto the next. unfortunately, now it takes forever to
deploy new versions of our code.
On Dec 17, 9:12 pm, casey <cassidyfor...@gmail.com> wrote:
> So you deploy to one server at a time instead of letting capistrano
> deploy to all at once?
> I just tried another deploy (under load) and had an outage again - it
> seems that Passenger does *something* when "current" symlink is moved
> and doesn't recover properly when under load.
> On Dec 17, 2:31 pm, amos <famosea...@gmail.com> wrote:
> > we had to switch to rolling restarts in capistrano in order to avoid
> > downtime. word is the passenger guys are working on a fix.
It was good to know that I'd have to solve the problem myself. I
ended up doing something similar. For each server in my cluster,
Capistrano...
* Deletes a file that my load balancer (HAProxy) is looking for from /
public which causes it to be removed from rotation
* Waits 5 seconds for any requests to finish
* Regular capistrano update
* Put the /public/file back so that the server goes back into rotation
* Wait 20 seconds for Passenger to warm up, then move on
On Dec 18, 4:39 pm, amos <famosea...@gmail.com> wrote:
> yes, in my experience passenger can't handle replacing the symlink
> under load which is why we use 'httpd restart'.
> we wrote a special capistrano hook to restart a server, wait 20
> seconds and go onto the next. unfortunately, now it takes forever to
> deploy new versions of our code.
> On Dec 17, 9:12 pm, casey <cassidyfor...@gmail.com> wrote:
> > So you deploy to one server at a time instead of letting capistrano
> > deploy to all at once?
> > I just tried another deploy (under load) and had an outage again - it
> > seems that Passenger does *something* when "current" symlink is moved
> > and doesn't recover properly when under load.
> > On Dec 17, 2:31 pm, amos <famosea...@gmail.com> wrote:
> > > we had to switch to rolling restarts in capistrano in order to avoid
> > > downtime. word is the passenger guys are working on a fix.
oops - forgot the important part - an Apache restart after the Cap
update. I touch Passenger's restart.txt but there are no requests
coming in and these Apaches are dedicated to passenger, might as well
start fresh.
On Dec 20, 12:21 am, casey <cassidyfor...@gmail.com> wrote:
> It was good to know that I'd have to solve the problem myself. I
> ended up doing something similar. For each server in my cluster,
> Capistrano...
> * Deletes a file that my load balancer (HAProxy) is looking for from /
> public which causes it to be removed from rotation
> * Waits 5 seconds for any requests to finish
> * Regular capistrano update
> * Put the /public/file back so that the server goes back into rotation
> * Wait 20 seconds for Passenger to warm up, then move on
> On Dec 18, 4:39 pm, amos <famosea...@gmail.com> wrote:
> > yes, in my experience passenger can't handle replacing the symlink
> > under load which is why we use 'httpd restart'.
> > we wrote a special capistrano hook to restart a server, wait 20
> > seconds and go onto the next. unfortunately, now it takes forever to
> > deploy new versions of our code.
> > On Dec 17, 9:12 pm, casey <cassidyfor...@gmail.com> wrote:
> > > So you deploy to one server at a time instead of letting capistrano
> > > deploy to all at once?
> > > I just tried another deploy (under load) and had an outage again - it
> > > seems that Passenger does *something* when "current" symlink is moved
> > > and doesn't recover properly when under load.
> > > On Dec 17, 2:31 pm, amos <famosea...@gmail.com> wrote:
> > > > we had to switch to rolling restarts in capistrano in order to avoid
> > > > downtime. word is the passenger guys are working on a fix.
|
