Job parallel execution limit & queuing

[Blair Paine]

Hi all,

I have a requirement where I need to limit the number of jobs being executed at the same time.   This is to help avoid load on shared infrastructure (i.e. SAN).   I'll be automating across 1500+ Windows nodes over the space of a few hours.

Does Rundeck have such a feature in the current version?   Are jobs queued when the limit is reached?

If not I would envisage this should work similar to how Node dispatching and filtering works but for jobs.  It would also be fantastic if I could also rank the order of the jobs.   Similar to how we can rank nodes.   I guess this would require jobs to have tags and attributes.

Example:

I have a scheduled master job which will then call 400+ custom jobs.  Each of those 400 custom jobs call a set of common jobs.  The custom jobs define the nodes, and the common jobs accept the nodes as parameters to the underlying scripts.   When the master job runs the 400+ jobs, I only want n jobs running at that level at a time.

My current thinking is I may have to use Jenkins and integrate with Rundeck to do this today.   I would much prefer to be able to do it all within Rundeck.

Best regards,

Blair.

[Scott Chapman]

Yeah, I'd like that too.

I ended up building a shell script process to parse the output of "ps" to see how many of a particular thing was running, then go to sleep for a bit if there's >= the limit running. Repeat until there's < than the limit. I stick a call to that script as the first item in the job, looking for whatever the next step is actually running. So the job effectively waits until it's time to release the job. I also had to increase the number of concurrent tasks in the quartz scheduler in rundeck though because I may have dozens of jobs waiting. 

It's kind of a hack. But it mostly works.

I didn't know how much I would miss JES and JCL when working on a non-mainframe platform. It's funny how these sort of problems were solved decades ago there.