Limit concurrency in capastrano v3

[Jochen Billen]

Hi

I'm looking into Capistrano (v3) to deploy an PHP application to 100+ servers and I'm am concerned that running the 'git update' and 'composer install' task will put to much strain on the infrastructure (git server, SAN environment and network)

Is there an easy way to limit the number of hosts to run tasks on in parallel? Or should I override particular tasks which could be to stressful using sshkit's 'on(in: :groups, limit: 2, wait: 5) { ... }'.

Best Regards,

Jochen Billen

Versions:

• Ruby: 1.9.2
• Capistrano: 3.2.1
• Rake: 10.3.1

Platform:

• Working on: Debian
• Deploying to: Debian

[Lee Hambley]

Hi Jochen,

Unfortunately not, however you might try forking the gem, adding a Gemfile to your project (to grab your own version of Capistrano) and modifying the git.rake/git.rb tasks to sun serially, or at least in rolling groups.

You could achieve this by (roughly speaking) the following:

$ gem install bundler

$ cd ./my/project/path

$ bundle init

(this will create a Gemfile)

Add the line:

gem "capistrano", github: 'jochen/capistrano'

and, then run:

$ bundle

$ bundle exec cap production deploy

You are now working on your own Capistrano version.

You can also clone your own Capistrano fork to your machine, and tweak it, without having to commit, and push back to Github, with something like:

gem "capistrano", path: '../capistrano'

If you are successful, maybe we could change the main Gem to prefer groups of 5/10 servers in a rolling configuration as the default, since this is larger than *most* people's starting setups.

Alternatively, you can look at http://blog.jayfields.com/2008/02/rake-task-overwriting.html which you might use (in your deploy.rb, or Capfile) to temporarily monkey patch your own installation, however I'd prefer the "own gem" solution if I were you, as it leads naturally to a bit more control over your environment, and the chances that we change something, and nerf your monkey patch are mitigated.

Lee Hambley

--

http://lee.hambley.name/

+49 (0) 170 298 5667

--
You received this message because you are subscribed to the Google Groups "Capistrano" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capistrano+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/capistrano/1f0ff59b-6ea1-460a-bd8c-f1acb6b62557%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Richardson]

Jochen Billen <joc...@smartbit.be> wrote:
> I'm looking into Capistrano (v3) to deploy an PHP application to 100+
> servers and I'm am concerned that running the 'git update' and
> 'composer install' task will put to much strain on the infrastructure
> (git server, SAN environment and network)

You could also add a command to the process which basically just does
sleep $(RANDOM)

One thing that I have also done to reduce stress (particularly against
github.com), is to take one of the checked out trees, move it, merge the
.git/objects directories together, and then add a --reference to that.
Not only does it speed things up, but it can significantly reduce the disk
space requirement.

--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | network architect [
] m...@sandelman.ca http://www.sandelman.ca/ | ruby on rails [

[Jochen Billen]

Hi Lee

Thank you for the feedback.

I'm going for the fork-approach (https://github.com/jbillen/capistrano). I am however not familiar with ruby so it might take a while before I come up with something useful.

Best Regards

Jochen Billen

Op woensdag 21 mei 2014 11:22:21 UTC+2 schreef Lee Hambley:

[Jochen Billen]

Hi Michael

Thanks for the feedback.

I'm afraid that putting in a random sleep will not give the level of control I'm looking for. So I'm going to try the fork-approach as suggested by Lee in the above post.

Best Regards

Jochen Billen

Op woensdag 21 mei 2014 14:39:07 UTC+2 schreef Michael Richardson:

[slowjack2k]

Hi Jochen,

I would try to override the git wrapper script and use trickle to limit bandwith.

namspace :my

desc 'Upload the custom git wrapper script'

  task :wrapper do

    on release_roles :all do

      upload! StringIO.new("#!/bin/sh -e\nexec  /usr/bin/trickle -u 100 /usr/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no \"$@\"\n"), "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"

      execute :chmod, "+x", "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"

    end

  end

end

after 'git:wrapper', 'my:wrapper'

I think this can work.

regards
dieter

[slowjack2k]

PS: for git downloads the trickle switch would be -d not -u. Best you use booth.

[Jochen Billen]

Hi

I have create a fork of the capistrano project and made some minor changes to support rolling groups. It turns out to work well for our code deployments. 

You can find the fork at https://github.com/jbillen/capistrano

Throttling the deployment can be achieved by putting following lines in config/deploy.rb:

set :rolling_group_size, 5

set :rolling_group_wait, 2

Best Regards

Jochen

Op woensdag 21 mei 2014 10:58:06 UTC+2 schreef Jochen Billen:

[Lee Hambley]

Might be worth a PR Jochen?

Lee Hambley

--

http://lee.hambley.name/

+49 (0) 170 298 5667

--

You received this message because you are subscribed to the Google Groups "Capistrano" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capistrano+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/capistrano/da51027b-a6e6-4ca8-91a9-f559da5cb4d4%40googlegroups.com.

[slowjack2k]

Maybe a more general abroach is possible.

set :default_exec_options, {in: groups, wait: 2 ...}

and change https://github.com/capistrano/sshkit/blob/master/lib/sshkit/coordinator.rb
default options accordingly or merge the options into
https://github.com/capistrano/sshkit/blob/master/lib/sshkit/dsl.rb on option.

regards
dieter