Limit number of jobs per project

[Sergey Andreev]

Hi,

We run a Rundeck instance with several projects. Each project has 5+ jobs that run on schedule. We would like to limit the number of parallel jobs per each project.

As of now, it looks like that the only way to support this use case is to move each project to a separate Rundeck instance and set the org.quartz.threadPool.threadCount to the desired number of parallel jobs per instance. 

I would like to ask if there is a better way to do that or if it is something that might be provided as a new feature in the upcoming releases?

Kind Regards,

Sergey

[Alex Honor]

Hi Sergey,

Setting the threadPool size per project is not a current capability. Curious about the scenario though. Should jobs queue once the limit is reached or should they be skipped?

Thanks

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/7d605103-a1a1-4e47-8048-0fd7b3b6fe82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex Honor

[SimplifyOps, Inc | a...@simplifyops.com ]

Be sure to comment and vote on Rundeck Feature Development!

[Sergey Andreev]

Hi Alex,

Let me elaborate on the use case.

We have one project that has around 20-30 scheduled jobs. We would like to limit the number of parallel jobs to 3 at the same time (jobs should queue once the limit is reached).

However, we have other projects (deploy code, run custom scripts on different nodes) that we run from time to time and we don't want them to affect the project with scheduled jobs. 

Kind Regards,

Sergey

Alex Honor

[SimplifyOps, Inc | a....@simplifyops.com ]

[Scott Chapman]

I wish it had that capability too.

I created a script that checks for (among other things, such as the current server load) the number of currently executing processes of a particular name. (ps | grep "name" | wc -l or something like that) If the count is above my limit, the script goes to sleep for x seconds before waking up and checking again. I make that script the first step in the jobs that I want to limit, so that while multiple might be submitted, only approximately my limit will really be running. Approximately because there's timing issues--if the active count goes below the limit and there are multiple jobs waiting, two jobs might notice it at the same time and release, so that there are now limit+1 jobs running. Also, the jobs don't necessarily run in order that they were submitted. And you can't tell from rundeck (without going into the job) that it's really waiting in the script because from rundeck's perspective it's "running". 

Also, if you submit lots beware of the default limit of 20 concurrent jobs in rundeck. I bumped that to 99--there are times where I have dozens of jobs waiting to execute. 

If you're submitting onto a different server this would also be problematic. 

So... definitely not a perfect solution, but it's something to consider.

[Sergey Andreev]

Thanks, Scott, looks like a good workaround though unfortunately it doesn't fit our use case right now.

I was wondering if providing job limit per project is a feature on the roadmap or something that could be contributed to the project?

Kind Regards,

Sergey

[dagu...@simplifyops.com]

Hi, the upcoming changes are listed here https://trello.com/b/sn3g9nOr/rundeck-development

If you want to add a new feature you can contact the rundeck team here http://simplifyops.com/development.html or open an issue here https://github.com/rundeck/rundeck/issues

Best!