Modify the number of jobs that can run in parallel for a project

[Mark Sinclair]

I have several projects, each with many associated jobs.  I need to control the number of jobs that run in parallel for each project.  Currently, I'm using slaves to accomplish this.  Each slave runs jobs for a specific project and I set the number of executors manually for each slave.  Note - each job kicks off a variable number of tests managed outside of Jenkins. 

Now I need to change the number of jobs that run in parallel in a more automated way.  The easy thing would be to write a groovy script that changed the # of executors, but unfortunately it appears that method is private for slaves.

My jobs are freestyle and use the build periodically build trigger.  I want to increase the number of executors, but not use all the executors all the time.

I was thinking of using ScriptTrigger plugin to evaluate a script and decide to launch, but I want the job to stay in the queue (at the head) if the script says it needs to wait.  It appears these triggers will skip the job.

Looking for ideas - thanks!

[Victor Martinez]

You can set the number of concurrent builds of a project running per node or globally with the below plugin:

- https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin

Cheers

[Mark Sinclair]

Thanks - I had looked into throttleconcurrent as well.  I think the method to change the number of concurrent per project is private though, so it appears the only way to modify is manual.  If there's a way to do it with a groovy script that plugin would be perfect.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/XoC9AF4ZQMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/daabcd88-dd9f-4557-93eb-011802702a73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Victor Martinez]

As far as I see that method is public as you can see in the javadoc:

- http://javadoc.jenkins-ci.org/hudson/model/Slave.html#setNumExecutors(int)

You might need to execute save afterwards:

- http://javadoc.jenkins-ci.org/hudson/model/Node.html#save()

Cheers

[Mark Sinclair]

I must have been mistaken - Thanks for the info!!

> --
> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/XoC9AF4ZQMc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8465cc6b-7c75-4e85-942b-ec42d23d36ca%40googlegroups.com.

[Mark Sinclair]

I'm still not able to use setNumExecutors.  I was able to use setNodeLabel though with the following code in script console:

for (slave in jenkins.model.Jenkins.instance.slaves) {
    oldLabelName = slave.getLabelString()
    println("Label= $oldLabelName")

    if (oldLabelName.contains('synth')) {
        println("Found $oldLabelName")
        newLabelName = oldLabelName + " " + "EXTRALABEL"
        slave.setLabelString(newLabelName)
        numexecutors = slave.getNumExecutors()
        println("executors=$numexecutors")
        //slave.setNumExecutors(1)
    }
}

This proved that I can access a slave and modify parts of it (the label), but when I uncomment the setNumExecutors line, I get the following error:

groovy.lang.MissingMethodException: No signature of method: hudson.slaves.DumbSlave.setNumExecutors() is applicable for argument types: (java.lang.Integer) values: [1]
Possible solutions: getNumExecutors(), getNumExecutors()
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
	at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
...

A bit more research and I found this, though I'm not clear on the final response from Jesse Glick.
https://issues.jenkins-ci.org/browse/JENKINS-23534

It appears he's saying that modifying the number of executors is possible, but you need to replace the whole slave?  Not sure how to do this - it would be a lot easier if I could just call setNumExecutors.

[Mark Sinclair]

I posted a comment on the JIRA and read through Jenkins source code for a while, but did not find a way to do it.

I would have preferred a simple job that ran periodically and just adjusted the number of executors on each slave, but I came up with an alternate approach.

• Increased the number of executors on slaves to the max that will be needed
  
• Created a new job 'hold_executor'
• Scheduled to launch periodically
• High priority via priority sorter plugin
• Limited concurrent threads with throttle concurrent plugin, so a minimum number of executors are always available for real work
  
• Job runs a script to check for conditions
• If executor should be disabled, the job sleeps while holding the executor slot for a time period then checks again
• If executor should be freed for real work, the job exits

Not as clean, but it meets my needs for now.