Throttle concurrent builds plugin

[John Vacz]

Hi all,

before I read the "deprecating plugins" thread, I was not aware of the
throttle concurrent builds plugin (TCB). It sounds very interesting for
me, because I am still searching an alternative of locks and latches
plugin (LaL). My situation is the following:

We have several jobs doing embedded jboss testing, they cannot run
concurrently on one slave due to port conflict. We've tried locks and
latches, let the test-jobs sharing the same lock. It looks like that LaL
blocks the global build queue (for the test-jobs), and although we have
several slaves idle, we can not utilize them. Finally we fall back to
another sub-optimal solution: every slave has only one executor, that
solved the concurrent problem, but it narrowd the whole "bandwidth".
From now and then, jobs piled up in queue waiting for these very few
executors to be available again.

Now my question: is TCB aware of slaves? That means, can I configure a
resource lock (or something similar) that works for every single slave,
but not globally? So that I can activate multiple executors for each
slave, the test jobs are distributed to the slaves, yet on every one
particular slave, only one running test job is allowed?

greetings
-- vf

PS.: Now I am not quite sure whether LaL is actually capable of this, I
am glad to be proven wrong.

[Eyal Edri]

> -----Original Message-----
> From: jenkins...@googlegroups.com [mailto:jenkinsci-
> us...@googlegroups.com] On Behalf Of John Vacz
> Sent: Wednesday, August 10, 2011 8:37 AM
> To: Jenkins Users
> Subject: Throttle concurrent builds plugin
>
> Hi all,
>
> before I read the "deprecating plugins" thread, I was not aware of the
> throttle concurrent builds plugin (TCB). It sounds very interesting for
me,
> because I am still searching an alternative of locks and latches plugin
(LaL).
> My situation is the following:
>
> We have several jobs doing embedded jboss testing, they cannot run
> concurrently on one slave due to port conflict.

[EE] use this plugin to dynamicly assign ports:
http://wiki.hudson-ci.org/display/HUDSON/Port+Allocator+Plugin

We've tried locks and latches,
> let the test-jobs sharing the same lock. It looks like that LaL blocks
the global
> build queue (for the test-jobs), and although we have several slaves
idle, we
> can not utilize them. Finally we fall back to another sub-optimal
solution:
> every slave has only one executor, that solved the concurrent problem,
but it
> narrowd the whole "bandwidth".
> From now and then, jobs piled up in queue waiting for these very few
> executors to be available again.
>

> Now my question: is TCB aware of slaves? That means, can I configure a
> resource lock (or something similar) that works for every single slave,
but not
> globally? So that I can activate multiple executors for each slave, the
test jobs
> are distributed to the slaves, yet on every one particular slave, only
one
> running test job is allowed?

[EE] [EE] I had the same problem.
Very easy solution: use the '> throttle concurrent builds plugin' and tick
the 'Execute concurrent builds if necessary (beta)' option.
This will allow you running each job on all slaves, but you can confine it
to 1 instance per node (slave).

[John Vacz]

Am 10.08.2011 09:02, schrieb Eyal Edri:
>
>> -----Original Message-----
>> From: jenkins...@googlegroups.com [mailto:jenkinsci-
>> us...@googlegroups.com] On Behalf Of John Vacz
>> Sent: Wednesday, August 10, 2011 8:37 AM
>> To: Jenkins Users
>> Subject: Throttle concurrent builds plugin
>>
>> Hi all,
>>
>> before I read the "deprecating plugins" thread, I was not aware of the
>> throttle concurrent builds plugin (TCB). It sounds very interesting for
> me,
>> because I am still searching an alternative of locks and latches plugin
> (LaL).
>> My situation is the following:
>>
>> We have several jobs doing embedded jboss testing, they cannot run
>> concurrently on one slave due to port conflict.
> [EE] use this plugin to dynamicly assign ports:
> http://wiki.hudson-ci.org/display/HUDSON/Port+Allocator+Plugin

I will try this.

I will dig into this and see how these two plugins playing together. Thanks!

[Andrew Bayer]

Yes, that's exactly the use case I wrote it for - I still need to improve the usability of the configuration, but the first option, if non-0, limits the total number of active builds of a project (or set of projects belonging to a category, if the project value is 0 and the category value, set in the main Jenkins config, is non-0) to that number. So 1 would be the equivalent of Locks and Latches, where any job in the category can only build if nothing else from the category is already building, across all slaves.

The second option behaves similarly, with a non-0 option for a single project or a 0 value for a single project that's part of a category with a non-0 value configured, but on a per-slave basis. So if you have multiple jobs in a category with 0/1 as its values, or a single job with concurrent builds enabled and 0/1 as its project values, so long as you have multiple slaves capable of building the job(s), each slave will run a build, but no more than one build belonging to that job/category at a time.

A.

[John Vacz]

Fantastic! Although I must admit I do not quite understand the configure :D

Does 0 imply unlimited?

Take my use case as example, say I have some slaves each has several executors running, I have 3 jobs J1, J2, J3, cannot run concurrently on one slave. So I create a category C, including J1, J2 and J3; set the second option (per node) as 0/1 or 3/1? What should I do with the first option (maximum total concurrent builds)? Leave them blank (ignore them) or give something like 0/0?

[John Vacz]

I think i figured it out.

In main Jenkins config:
* Maximum Total Concurrent Builds:   number of nodes
* Maximum Concurrent Builds Per Node: 1

In job config:
* Maximum Total Concurrent Builds: 0  
* Maximum Concurrent Builds Per Node: 1

This seems to work.

greetings
-- vf

[ann.ca...@shawinc.com]

On a tangential note...

I've been wondering lately if there's a way to throttle concurrent builds of specific unrelated projects...

I've got several jobs who only purpose is to put code over to Sonar for analysis. I made the mistake yesterday of running two of them simultaneously. Builds that normally take <5 min took nearly half an hour because the box was thrashing.

Is there a plugin out there that will let me configure "don't build this while these are building: ____________" or do I need to code that myself?

Ann Campbell
Engineer-Systems-IS Prod Sys-Shop Floor Sys
Shaw Industries Inc.
201 South Hamilton Street
Dalton, GA 30720
Email: ann.ca...@shawinc.com  Office: 706.275.3857

Please consider the environment before printing.

**********************************************************
Privileged and/or confidential information may be contained in this message. If you are not the addressee indicated in this message (or are not responsible for delivery of this message to that person) , you may not copy or deliver this message to anyone. In such case, you should destroy this message and notify the sender by reply e-mail.
If you or your employer do not consent to Internet e-mail for messages of this kind, please advise the sender.
Shaw Industries does not provide or endorse any opinions, conclusions or other information in this message that do not relate to the official business of the company  or its subsidiaries.
**********************************************************

[Andrew Bayer]

Yup, the Throttle Concurrent Builds plugin's Categories support is exactly what you're looking for - create a category in the main Jenkins config, with 1 for the "Maximum Total Concurrent Builds" and 0 (or any value - doesn't really matter) for "Maximum Concurrent Builds per Node". Then go to each job you want to add to the category, check the Throttle Concurrent Builds checkbox, put in 0 for the total values there, and then check the category you just created. Now only one build of any job belonging to that category can run at the same time.

And John, the 0 means unlimited, and in the job config, it means unlimited unless something else is specified in a category the job belongs to. I think I'm going to make the job config be an either/or - either you've got per-job limits, or you're using categories for that job, not both, since that's led to a lot of confusion. =)

A. 

[ann.ca...@shawinc.com]

Hot diggity! Thanks!

Ann Campbell
Engineer-Systems-IS Prod Sys-Shop Floor Sys
Shaw Industries Inc.
201 South Hamilton Street
Dalton, GA 30720
Email: ann.ca...@shawinc.com  Office: 706.275.3857

Please consider the environment before printing.

[John Vacz]

Thanks for clarifying. Nice work, Andrew!

[Martin Ba]

Hi Andrew,

I've recently added a bit of extra documentation to he plugin's wiki page.
I think the way it is at the moment -- 1st check single job, then check cats. -- is really ok and shouldn't be changed unless the change could allow for some more complex use cases.
Personally I think the few extra sentences on the wiki go a long way of preventing confusion :-)

thanks for the great work, cheers,
Martin

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Andrew Bayer <andrew...@gmail.com> schrieb:

[kalpsat]

I ahve a situation where matrix job sub job(one per slave) needs to be
throttled with free style job, but the free style job can run 4-5 concurrent
builds. Looks like throttle does not work for matrix build. or am i wrong.



--
View this message in context: http://jenkins-ci.361315.n4.nabble.com/Throttle-concurrent-builds-plugin-tp3732090p4675876.html
Sent from the Jenkins users mailing list archive at Nabble.com.