Webistrano stuck on update_code only when trying to deploy to multiple hosts

[burg]

I'm trying to set up Webistrano + Capistrano to deploy to multiple
servers:
When trying to deploy to each server separately, everything works
great. However when I select multiple hosts to deploy it just gets
stuck in the update_code stage (if I run 'restart' for example on
multiple hosts - it works just fine).
How can I try and debug this further? there is no error code and error
log is empty

* executing `deploy:update_code'
updating the cached checkout on all servers
* executing "if [ -d /usr/local/shared/cached-copy ]; then cd /usr/
local/shared/cached-copy && git fetch -q origin && git reset -q --hard
ebfe2bc7525a09fcdc2dcb1d98db980a92f2ea84; else git clone -q --depth 1
ssh://ro...@snort1-1.isonet/var/repos/metacafe /usr/local/shared/cached-
copy && cd /usr/local/shared/cached-copy && git checkout -q -b deploy
ebfe2bc7525a09fcdc2dcb1d98db980a92f2ea84; fi"
servers: ["web7.isonet", "web34.isonet"]

[[STUCK... no more output... ]]

production.log is empty

[Lee Hambley]

Burg, Looks like the checkout is simply taking a long time; how large is your repository? 

-- Lee Hambley

Twitter: @leehambley | @capistranorb

[burg]

When deploying to one server only, it takes few seconds...when
selecting 2+ hosts, it's stuck indefinitely

[Lee Hambley]

As per the instructions on the google group homepage, please post all the debugging info we'll need please!

[burg]

Missing info:
$ ruby --version
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]
Capistrano 2.50
Webistrano 1.4
Effective configuration:
application Metacafe Portal
copy_exclude %w(.git)
current_dir metacafe_capistrano
deploy_to /usr/local
deploy_via :remote_cache
git_shallow_clone 1
password ************
repository /var/repos/metacafe
scm git
scm_password ************
scm_user root
user root

Some clue: “Deploying 36 hosts from CentOS RackSpace workstation to
CentOS Apache Web servers. Application is PHP only.”

[burg]

It is pretty clear to me that Webistrano / Capistrano is stuck in the
'git fetch' stage probably because of some permissions issues. However
it is not outputing it and I cannot seem to find any proper debug
method for what's going on behind the NetSSH

[Lee Hambley]

Thanks for the info, nothing suspicious there apart from deploying to /usr/local which is odd.

But according to your last email you can deploy these hosts individually?

"when deploying to one server only, it takes few seconds...when selecting 2+ hosts, it's stuck indefinitely"

[burg]

Here's an example of a succesful one server deployment:
** loading stage recipe 'finalize_update'
** loading stage recipe 'before_symlink'
** loading stage recipe 'prod_before_deploy'
** loading stage recipe 'symlink'
* executing `deploy'
* executing `deploy:update'
triggering before callbacks for `deploy:update'
* executing `prod_before_deploy'
** transaction: start
* pushing latest code to vsync
* done
** transaction: commit
** transaction: start

* executing `deploy:update_code'
updating the cached checkout on all servers
* executing "if [ -d /usr/local/shared/cached-copy ]; then cd /usr/
local/shared/cached-copy && git fetch -q origin && git reset -q --hard

8def518cd3709a102ccbb508b79ad3d000a91eb8; else git clone -q --depth 1

ssh://ro...@snort1-1.isonet/var/repos/metacafe /usr/local/shared/cached-
copy && cd /usr/local/shared/cached-copy && git checkout -q -b deploy

8def518cd3709a102ccbb508b79ad3d000a91eb8; fi"
servers: ["web7.isonet"]
[web7.isonet] executing command
command finished
copying the cached version to /usr/local/releases/20091115182706
* executing "rsync -rp --exclude=\"%w(.git)\" /usr/local/shared/
cached-copy/* /usr/local/releases/20091115182706 && (echo
8def518cd3709a102ccbb508b79ad3d000a91eb8 > /usr/local/releases/
20091115182706/REVISION)"
servers: ["web7.isonet"]
[web7.isonet] executing command
command finished
* executing `deploy:finalize_update'
* executing `deploy:symlink'
triggering before callbacks for `deploy:symlink'
** transaction: commit
* executing `deploy:restart'

[Lee Hambley]

Sure, but does that run, successfully - individually on both servers, invidually? Might I ask the reason for a deploy to /usr/local

[burg]

yep... each server runs successfully.

I just looked at the 'TOP' of the webistrano server and found this -
if it might help:
top - 14:02:41 up 7 days, 9:59, 4 users, load average: 2.42, 2.17,
1.69
Tasks: 132 total, 3 running, 129 sleeping, 0 stopped, 0 zombie
Cpu(s): 48.0%us, 2.4%sy, 0.0%ni, 49.6%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 2051948k total, 1152812k used, 899136k free, 227292k
buffers
Swap: 6004728k total, 0k used, 6004728k free, 351192k
cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
31392 www-data 20 0 139m 56m 3712 R 100 2.8 0:24.30 ruby
script/runner -e production deployment = Deployment.find(273);
deployment.prompt_config =
29375 www-data 20 0 137m 54m 3712 R 100 2.7 144:40.03 ruby
script/runner -e production deployment = Deployment.find(254);
deployment.prompt_config =
31205 www-data 20 0 144m 61m 2584 S 1 3.1 0:02.63 Rails: /
var/rails/
webistrano
31399 root 20 0 19132 1340 980 R 0 0.1 0:00.10
top



Seems like the Ruby is trashing the CPU for some reason....

[burg]

strace of those processes shows repetitive:
select(8, [6], [], [], {0, 0}) = 1 (in [6], left {0, 0})
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

[Lee Hambley]

Burg, what happens if you don't use Webistrano (this isn't supported here, they have their own list) -- and its important to rule it out, I don't use it - but I know I had problems last time I tried.

Were the ruby & capistrano versions you listed, from the Webistrano server (not your workstation)

[Lee Hambley]

Burg, what happens if you don't use Webistrano (this isn't supported here, they have their own list) -- and its important to rule it out, I don't use it - but I 

-- Lee Hambley

Twitter: @leehambley | @capistranorb

Blog: http://lee.hambley.name/
Working with Rails: http://is.gd/1s5W1

2009/11/15 burg <burg...@gmail.com>

--
* You received this message because you are subscribed to the Google Groups "Capistrano" group.
* To post to this group, send email to capis...@googlegroups.com
* To unsubscribe from this group, send email to capistrano+...@googlegroups.com For more options, visit this group at http://groups.google.com/group/capistrano?hl=en

[Lee Hambley]

Damn, Burg - update your Capistrano 2.5.10 released last week; (mostly this will upgrade your Net::SSH client, there was a bug with threading that they fixed at 2.0.14, but AFAIK it hasn't manifested with so few hosts before.

[Jonathan Weiss]

On Sun, Nov 15, 2009 at 8:08 PM, Lee Hambley <lee.h...@gmail.com> wrote:
> will upgrade your Net::SSH client, there was a bug with threading that they
> fixed at 2.0.14, but AFAIK it hasn't manifested with so few hosts before.

Yes, please try Webistrano from master, it includes the Net::SSH fix
that will hopefully fix your problem.

Jonathan

--
Jonathan Weiss
http://blog.innerewut.de
http://twitter.com/jweiss

[burg]

Hi Guys,
I've upgraded to the GitHub version of Webistrano + probably the
NetSSH required upgrade, and this seemed to solve the problem.

Thanks a bunch

On Nov 15, 10:33 pm, Jonathan Weiss <j...@innerewut.de> wrote: