Prevent two+ different jobs from running at the same time

1,199 views
Skip to first unread message

Mark Kegel

unread,
Dec 2, 2015, 6:00:24 PM12/2/15
to rundeck-discuss
We have a use-case where we've set up multiple jobs for the same set of nodes. The first job, let's call it 'deploy', deploys code to the node and the second job, call it 'import', runs a data import process on the node. We would like to make it so that our 'deploy' job will not run at the same time as our 'import' job and vice-versa, since if they did a node would end up in an inconsistent state.

Is there any way for us in Rundeck to configure a job A so that it either won't run while another different job B is running, or will wait until said job is done? I can code the given scripts so that they check if the other is running, but then Rundeck won't have any notion of why the job failed/paused. Having the functionality as a first-class option in Rundeck would be ideal.

Alex Honor

unread,
Dec 2, 2015, 6:11:20 PM12/2/15
to rundeck...@googlegroups.com
Hi Mark,

You can check on another job execution using the 'Job State Conditional' workflow step. I added a screenshot of the step. You can see that execution state and how to fail or halt the job can be set. Note also the Status field where you can define your own string. You can then use an errorhandler to catch the thrown fail action and decide how to proceed.

Inline image 1

On Wed, Dec 2, 2015 at 2:40 PM, Mark Kegel <mark....@gmail.com> wrote:
We have a use-case where we've set up multiple jobs for the same set of nodes. The first job, let's call it 'deploy', deploys code to the node and the second job, call it 'import', runs a data import process on the node. We would like to make it so that our 'deploy' job will not run at the same time as our 'import' job and vice-versa, since if they did a node would end up in an inconsistent state.

Is there any way for us in Rundeck to configure a job A so that it either won't run while another different job B is running, or will wait until said job is done? I can code the given scripts so that they check if the other is running, but then Rundeck won't have any notion of why the job failed/paused. Having the functionality as a first-class option in Rundeck would be ideal.

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/d0f3cb35-339b-4a18-a4fd-e8302c1fbb8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Alex Honor

[SimplifyOps, Inc | a...@simplifyops.com ]

Be sure to comment and vote on Rundeck Feature Development!

Mark Kegel

unread,
Dec 3, 2015, 11:24:12 AM12/3/15
to rundeck-discuss
Awesome I'll give this a try. Thank you.

Daniel Valente

unread,
Jan 25, 2017, 12:45:40 PM1/25/17
to rundeck-discuss
Hi, 

  In Rundeck, how can we assure that n jobs(more than 2) don't run at same time. Do we have to put "Job State Conditional" in all n jobs, with all n-1 conditions?
  Example: We have 4 jobs that run x time per day with different duration, but can't run at same time.
                 If Job1 starts executing, all the others should only start when job1 finishes (ideally with callback, not pooling).
                 Do I have to put a 3 "Job State Conditional" for all the jobs:
                  Job1 : Start only if Job2 is not executing.
                            Start only if Job3 is not executing.
                            Start only if Job4 is not executing.

                  Job2 : Start only if Job1 is not executing.
                            Start only if Job3 is not executing.
                            Start only if Job4 is not executing.
                 .....

Best Regards

Jason McIntosh

unread,
Jan 25, 2017, 5:13:05 PM1/25/17
to rundeck...@googlegroups.com
The plugin I'm working on:
Is also designed for this use case.  This sounds like a semaphore situation.  My plugin requires Consul, where you'd create a "limit" and "used" key/values, and the plugin would use those to determine how many can run concurrently, how long to wait for an available resource before failing, etc.

Jason


To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/483c702c-ef2d-440d-83b0-462bf83b9082%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Daniel Valente

unread,
Jan 27, 2017, 7:20:19 AM1/27/17
to rundeck-discuss
Thanks Jason, I will test it.

Daniel Valente

unread,
Feb 3, 2017, 10:54:31 AM2/3/17
to rundeck-discuss
Hi Jason,

  How do you suggest to manage many resources(key/values), on hundreds of jobs. The plugin is great for few jobs, and a small number of resources.
  It would be nice to have the resource management and jobs dependency on rundeck job definition .
   

Best Regards

Jason McIntosh

unread,
Feb 3, 2017, 11:24:31 AM2/3/17
to rundeck...@googlegroups.com
I suppose it wouldn't be hard to auto setup a few things, e.g. update the plugin to "auto create" a resource pool as needed.  Plugin has JUST started.  There might be other options as well.  Not looked into it that far yet...  and I'm completely open to ideas on how to manage this ;)  I'm fairly new to RunDeck and am just playing with it right now myself.
Jason

To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/d7e9bb58-cd47-495a-83b7-487b7a15635a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Daniel Valente

unread,
Feb 3, 2017, 12:11:29 PM2/3/17
to rundeck-discuss
As a suggestion, and I'm testing it, the first thing the "acquire_resource.sh" could do, is test if the resource exists, if not, create it with a default limit (eg: 1). 
It would be perfect to manage the limits of each resource on rundeck, for example in manage plugin page (eg: search resource and change limit).

I think that you had a great idea on using consul with rundeck, maybe it can be useful for other rundeck configuration in order to maintain them centralized when having many servers.

Jason McIntosh

unread,
Feb 3, 2017, 12:24:56 PM2/3/17
to rundeck...@googlegroups.com
Should be pretty easy to add that functionality :)  I just setup an issue, will fiddle with this when I can get some time.

To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/2e080a07-6b29-4bb7-a2ab-c397c5412735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Daniel Valente

unread,
Feb 3, 2017, 2:50:51 PM2/3/17
to rundeck...@googlegroups.com
Yup. I've tested and it works fine.

To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/2e080a07-6b29-4bb7-a2ab-c397c5412735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/3wWjCO6SRO0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/CAD6m6fEiYea3Fg7No-u2LWik-D2P2hpVEYn4ajOiSMwBJmFadw%40mail.gmail.com.

Daniel Valente

unread,
Feb 6, 2017, 5:33:23 AM2/6/17
to rundeck...@googlegroups.com
Hello Jason,

  Another issue came up.
  Unfortunately, Rundeck only allow one "error handler" per step, this can be a problem when you have multiple actions, for example, on error, we have to release the consul resource, otherwise it will never be released.
  Example:
       * Aquire resource
       * step 1
              on error ( Job 5)
       * step 2
              on error (Job 6)
       * Release resource
  In these case, How do you suggest to "Release Resource" if an error occur?

Best Regards

Jason McIntosh

unread,
Feb 6, 2017, 11:04:03 AM2/6/17
to rundeck...@googlegroups.com
Well there's always go in and manually change the count ;)  I'd say most jobs with auto try and other error handling shouldn't fail.  BUT I'm going to show my rundeck newness here and say I'm not fully sure what the answer is.  You COULD I'd guess create a separate workflow job to "recover" from failures that would be a more complicated error handler.  That second job could have a workflow of "retry, on failure release resource and fail" kinda thing.


Best Regards


To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/2e080a07-6b29-4bb7-a2ab-c397c5412735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/3wWjCO6SRO0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discuss+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/CABBVLemR82EvcYwbbuZY6OZiDEiGx4WCeqOHRG4d2DCUyJtaJw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages