Apply parallelism (concurrency) to multiple tasks

33 views
Skip to first unread message

Eric Rutherford

unread,
Mar 17, 2016, 2:43:15 AM3/17/16
to Capistrano
Versions:
  • Ruby 2.0.0
  • Capistrano 3.4.0
I'm trying to work out a solution to limit the number of servers that a group of tasks (remove from load balancer, push code, restart service, add back to load balancer for example) can run on at a time.  I've been digging through past conversations and documentation, but I've failed to find a solution with capistrano 3.x so far.  I've found some examples from v2.x, but most of the work in that example relied on features that are no longer in existence.  I've been working through creating a custom deploy task that specifies "in: :sequence" and calls outside tasks, but I'm not sure if I'm on the right track and figured that I might ask for some pointers (or a push in the right direction) before I head too far in the wrong direction.

Lee Hambley

unread,
Mar 17, 2016, 2:46:00 AM3/17/16
to Capistrano
HI Eric,

Certainly if you're looking for restricting, or choosing your preferred kind of parallelism/concurrency for your own tasks you are in the right place with the `on(in: ....)` functions (https://github.com/capistrano/sshkit/blob/master/README.md#parallel)

For Capistrano and it's plugins you are at the mercy of the maintainers (yours truly) or the plugin authors to have exposed the concurrency settings.

If there's any more I can help with (long question, short answer!) I'm more than happy, just drop me a note, you shouldn't have to wait 20h in the moderation queue whilst I ignore your email next time ;-)


--
You received this message because you are subscribed to the Google Groups "Capistrano" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capistrano+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/capistrano/cf49591c-0be0-44d6-b1b3-c44bf2027a91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Eric Rutherford

unread,
Mar 17, 2016, 11:48:44 PM3/17/16
to Capistrano
Hello Lee
No problem at all on the length of time for the response (I actually considered it to be pretty quick) and I appreciate your time on the matter. 

I think the problem may be that the tasks I'm trying to group together are ones that are executed on the host and others that are executed by the capistrano run.  For example I've built a capistrano plugin for de-registering and registering our nodes with our AWS ELBs.  I'd like to call that deregistration task on the same node that I'm restarting the service for, then call the registration task after the restart before de-registration/restart task on the next node.

I thought I could invoke a task from the on() block and have it execute a task/function, but I had mixed results when I did that.  I also wasn't sure of that was really how that was intended to be used.

Any hints or a kick in the right direction would be greatly appreciated.

Eric

Lee Hambley

unread,
Mar 18, 2016, 10:04:34 AM3/18/16
to Capistrano
I thought I could invoke a task from the on() block and have it execute a task/function, but I had mixed results when I did that.  I also wasn't sure of that was really how that was intended to be used.

For whatever it's worth you can write your code in classes, and then just pass the SSHKit context into them:


This would allow you to do some testing, however....

The on() block is the unit of synchronisation, one on() block will wait for all servers to complete before it returns.
 -- SSHKit README.md

If you can send me some pseudo code of what you wanted to work, I'd be more than glad to try and go a bit deeper, it's also worth pointing out that there's a pretty decent split between the task structure (`namespace`/`task` from Rake, which makes up the Capistrano DSL) and the underlying do something on a server code (SSHKit) ... the latter should be threadsafe as the parallel() helpers in SSHKit make liberal use of threads...

What I'm trying to say is:
  • You can:
    • Write your own classes (and, test them!)
    • Avoid our synchronisation primitives and use threads if ours don't work
I hope that helps, but I must admit that a handful of lines of pseudo code from your end would raise my confidence factor in making suggestions 10x.

Cheers, Lee

Eric Rutherford

unread,
Mar 18, 2016, 5:06:55 PM3/18/16
to Capistrano
My "mixed results" was completely a misunderstanding on my part.  I didn't realize you needed to re-enable a task that's invoked in a loop, otherwise it's just called once.  Thanks for your help, I've figured out what I was doing wrong.

Eric

Lee Hambley

unread,
Mar 19, 2016, 12:15:20 PM3/19/16
to Capistrano
http://blog.simplificator.com/2014/12/30/rake-execute-vs-invoke/ 

Might help, although honestly if you get as far as a Cap/Rake task like "rolling restart of servers", it makes sense to just pass the context object into class and manage that yourself.

Honestly, I think invoke vs. execute catches people out frequently enough that I ought to alias one to the other. Rake as a build tool, invoke is more useful for avoiding unnecessary work, for tools like Cap, I barely see the case for having a "once" option.


To view this discussion on the web, visit https://groups.google.com/d/msgid/capistrano/a2d82095-1cc8-4aff-9069-ab2156d22042%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages