Parallel job execution

[Joshua Braegger]

I've seen this mentioned on the list before, but haven't come across any type of solution or workaround.  Does anyone know how I can execute multiple jobs at the same time?  Particularly in groups.  Something like in this case:

jobset1:

    job1:

        command1

        command2

    job2:

        command3

        command4

jobset2:

    job3:

        command5

job1 and job2 are executed at the same time, and job3 wait until job1 and job2 are finished, then executed.  If someone has run across something like this and could provide feedback, it would be appreciated.

Thanks,

-Josh

[Luiz Casey]

Not sure it can be done from within Rundeck alone. We do this via Jenkins to manage the jobs/commands. Each Jenkins job contains the rundeck jobs/commands we want to run in parallel.

[Greg Schueler]

Hi Kelly,

Rundeck doesn't (yet) have a facility for parallelizing Steps within a workflow. It does allow the steps to operate on multiple Nodes in parallel, but for now workflow Steps always happen sequentially on a particular Node.  Jobs themselves can be run in parallel to each other, but Rundeck has no special facility to orchestrate that aspect. Also, the mechanism such as Job3 waiting until Jobs 1 and 2 are done could be manually done using the Rundeck API to monitor the jobs and trigger the third job, but again there's no built-in way to do it.

We've been discussing these types of features, and are still in the ideas stage, but we are very interested in the kinds of use cases that they are needed for. The gist of our thinking is that Jobs and Workflows could use a more event-based system. That way, jobs could be triggered by an event such as a certain Job completing successfully.  As well, workflows wouldn't need a strict sequential ordering, but steps could be triggered similarly by an appropriate event. 

On Sun, Apr 14, 2013 at 9:45 PM, Kelly Shkuratoff <kshku...@gmail.com> wrote:

Did you (or anyone else) figure out a way to do this? 

Kelly

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Levent Tutar]

Hi Greg,

Are there any new developments about parallelizing? My use case is:
Job 1:
- Create one apache instance in Amazon
- Wait until ssh connection to the instance is working (This takes 2 minutes)

Job 2:
- Create one tomcat instance in Amazon
- Wait until ssh connection to the instance is working (This takes 4 minutes)

Job 3:
- Create one tomcat instance in Amazon
- Wait until ssh connection to the instance is working (This takes 4 minutes)

I now wait +/- 11 minutes. If I can run these jobs in parallel, I have to wait 4 minutes.

Kind regards,

Levent

[John Matheson]

+1 on this.
I too have a user case that parallel running jobs would be helpful.

Job 1 - shutdown app1 on node list1
Job 2 - shutdown app2 on node list2
Job 3 - shutdown app3 on node list3
Wait till complete
Job 4 - migrate storage
Wait till complete
Job 5 - start app1 on node list4
Job 6 - start app2 on node list5
Job 7 - start app3 on node list6


The startup and shutdown jobs dont need to be waiting on each other thus just consuming time.

Thanks,
John

[John Matheson]

Sorry, I realised now that this feature has been added in 2.4.0

[Dhanasekaran Anbalagan]

Hi Guys,

I am really interested use this feature, currently I can't find any example and docs on this.

Please guide me any example commands so that I can understand.

-Dhanasekaran.

[John Matheson]

Dhanasekaran

1. You need to turn the feature on in rundeck-config.properties:
  feature.incubator.parallelWorkflowStrategy=true 

2. You will see the Parallel strategy option 

Others might have a more efficient way to do this BUT I have separate jobs running parallel steps and one main job as an orchestration wrapper that triggers them in a sequence to managed error states.

Hope that helps.

John

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/hdzxZb9v6WA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/55a31d59-9f9f-476a-a9ff-1232f56a74ab%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fred Ellenberger]

Tried this feature out on RunDeck 2.6.2 on RHEL 7, and am seeing some quirky behavior using a job with 4 steps executed across 2 nodes.

It appears that the execution order of steps is not being honored when the parallel option is enabled.  Is anyone else seeing this behavior? 

Specifically, Step 4 (copy file action) ends up being executed seconds before Step 2 (inline script).  Furthermore, seeing the following error with the inline script step (no error when parallel is not utilized):

removeScriptPluginCache: /var/lib/rundeck/libext/cache/rd-winrm-plugin-1.3.2 for: /var/lib/rundeck/libext/rd-winrm-plugin-1.3.2.zip


It is also worth noting that the rd-winrm plugin is being utilized for this job, and both nodes are Win2k12 R2 servers.

[Greg Schueler]

The Parallel strategy executes the steps in parallel: meaning due to timing and threading, any step may execute before any other.

If you want a strict ordering you should not use Parallel

--

You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/92b76f94-be41-4b87-8e08-78ee763494ca%40googlegroups.com.