how to run tasks in parallel ?

[Valerio Schiavoni]

Hello

consider the following user-defined task:

desc "Run everything (upload_app,run_introducer,run_peers)"
task :run_all do
upload_app
run_introducer
run_peers
end

the run_all task executes 3 other tasks sequentially.

the problem is that the run_introducer, and the run_peers should be
run in parallel.
In particular, the run_introducer invoke sme commands on a server
machine, and the run_peers execute some other commands on (many) peers
machines.

how do you solve this ?

It would have been nice to have something lile:
task :run_all do
upload_app
run_introducer || run_peers
end

[Gregory Seidman]

On Wed, Feb 27, 2008 at 01:44:27AM -0800, Valerio Schiavoni wrote:
> consider the following user-defined task:
>
> desc "Run everything (upload_app,run_introducer,run_peers)"
> task :run_all do
> upload_app
> run_introducer
> run_peers
> end
>
> the run_all task executes 3 other tasks sequentially.
>
> the problem is that the run_introducer, and the run_peers should be
> run in parallel.
> In particular, the run_introducer invoke sme commands on a server
> machine, and the run_peers execute some other commands on (many) peers
> machines.

First, consider why you want to do this. Parallel processing is an
optimization technique. Do you really need to optimize your deployment
process? Also, remember that parallelism introduces non-deterministic
behavior. Do you really want to complicate debugging your deployment
process?

> how do you solve this ?

Assuming that, yes, you really want to take on parallelizing your
deployment process, you need to look at threading. Capistrano is just Ruby,
which means you can use Ruby's threading support. This may or may not give
an advantage, given that Ruby uses green threads, but it's that or hacking
together a forking or pthreads C extension to run the commands, and I'm
assuming you don't want to do that. Of course, I don't know what global
state Capistrano keeps, so it may not be thread-safe.

> It would have been nice to have something lile:
> task :run_all do
> upload_app
> run_introducer || run_peers
> end

That is syntactically impossible. Remember that Capistrano is still Ruby.
The || operator means something specific, and cannot be redefined. You
could, however, do this (if Capistrano is thread-safe):

task :run_all do
upload_app
intro = Thread.new { run_introducer }
peers = Thread.new { run_peers }
intro.join
peers.join
end

I strongly recommend against trying to parallelize your deployment,
however. Simplicity and determinism are the cornerstones of a repeatable
deployment process.

--Greg

[Jamis Buck]

Note that using threads in this way is not recommended, although it
_might_ work. The safest way to parallelize those tasks is to write a
shell script that executes the commands you want in parallel, and then
execute that script via capistrano:

task :run_all do
upload_app
run "/path/to/run_introducer_and_peers.sh"
end

Capistrano will then invoke the shell script on all servers, and your
shell script can then fire off and manage those processes however you
want.

- Jamis

> --~--~---------~--~----~------------~-------~--~----~
> To unsubscribe from this group, send email to capistrano-...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/capistrano
> -~----------~----~----~----~------~----~------~--~---
>

[Valerio Schiavoni]

I simply want to mimic the same as doing:

cap run_introducer
cap run_peers

in 2 different shells.

Using the Thread.new approach works for me.
The processes launched are really independant, from the capistrano
point-of-view.
And I like it the most, as it permits me to keep the deployment script
in 1-single-file.

On Feb 27, 4:04 pm, Jamis Buck <ja...@37signals.com> wrote:
>The safest way to parallelize those tasks is to write a  

safest in regard of what aspect ?

> Capistrano will then invoke the shell script on all servers, and your  
> shell script

i want to stay AWAY from shell scripts..or I would have not moved to
capistrano :-)
Our whole deployment management was based on bash scripting..and 1) i
dont know bash but i do know (little) ruby, 2) it's un-mantainable, 3)
it's a mess...

[Jamis Buck]

The problem is that Capistrano::Command was not built with the idea of
multiple instances running in a thread, in mind. Like I said, it
*might* work. It might also blow up spectacularly, or hang
unspectacularly.

Note, though, that Capistrano was _never_ intended as a replacement
for shell scripting. Underneath, everything cap does is shell
scripting. You give it a shell command and it executes it. Do not fear
shell scripting, embrace it! If you are avoiding shell scripts, you're
only taking advantage of about half of what cap can do for you.

- Jamis