Load Balancing

[Lukas Rytz]

Hi,

Is there a way to change the way Jenkins does load balancing? I started 6 new jobs, and 5 of

them ended up  on the same slave.

I'd prefer the load be distributed equally across all slaves..

Thanks.

[Michaël Pailloncy]

Hi,

In each of your slave, you can change the default strategy of how Jenkins use its slaves.
http://your-jenkins:8080/computer/yourslave/computer
Property 'Usage'

Michaël

2012/7/26 Lukas Rytz <lukas...@epfl.ch>

[Lukas Rytz]

Yes, I'm aware of that.

All of the slaves in question have that field set to "Utilize this slave as much as possible".

On Thursday, July 26, 2012 2:58:46 PM UTC+2, mpapo - Michaël Pailloncy wrote:

Hi,

In each of your slave, you can change the default strategy of how Jenkins use its slaves.
http://your-jenkins:8080/computer/yourslave/computer
Property 'Usage'

Michaël

2012/7/26 Lukas Rytz

Hi,

[Michaël Pailloncy]

Did you launch your 6 differents jobs at the same time ?
It's curious that Jenkins use the same slave for 5/6 builds.
How many slaves do you have ?

2012/7/26 Lukas Rytz <lukas...@epfl.ch>

[Lukas Rytz]

There was a delay of ~10 seconds between starting each build.

You can see some details in the screenshot of my original post: We have 5 slaves here,

one of them was already running one job, all of the others were idle.

It would be nice if the scheduling was configurable. It seems that the "Throttle Concurrent

Builds" plugin [1] works in that direction: it can make sure to never run more than X builds

on one machine. What I'm looking for is not a limited, but a balanced scheduling.

Lukas

[1] https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin

[Michaël Pailloncy]

Ok, sorry ! I had not seen your screenshot.
Jenkins behaviour does not surprise me. It seems to set the priority to slave where a build has already be done.
Otherwise, why do you need this feature ? I'm sure that if you create 10 jobs and launch 10 builds on each, Jenkins will use all available slots without restrict builds to the prefered slave.

2012/7/26 Lukas Rytz <lukas...@epfl.ch>

[Ferenc Kovacs]

On Thu, Jul 26, 2012 at 11:49 AM, Lukas Rytz <lukas...@epfl.ch> wrote:

Hi,

Is there a way to change the way Jenkins does load balancing? I started 6 new jobs, and 5 of

them ended up  on the same slave.

I'd prefer the load be distributed equally across all slaves..

somebody will correct me, if I'm wrong, but Jenkins uses the following algorithm:

- it only uses nodes with free executors
- it will prefer to run the build on the same node where it was last executed

- it will prefer nodes where there a workspace already exists.

- if nothing else, it will chose a node in ~random.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

[Lukas Rytz]

On Thursday, July 26, 2012 4:43:38 PM UTC+2, mpapo - Michaël Pailloncy wrote:

Ok, sorry ! I had not seen your screenshot.
Jenkins behaviour does not surprise me. It seems to set the priority to slave where a build has already be done.
Otherwise, why do you need this feature ?

Because the builds will complete faster.

 

I'm sure that if you create 10 jobs and launch 10 builds on each, Jenkins will use all available slots without restrict builds to the prefered slave.

OK, but I obviously don't want to copy the same job configuration 10 times...

[Les Mikesell]

On Thu, Jul 26, 2012 at 9:06 AM, Lukas Rytz <lukas...@epfl.ch> wrote:
> There was a delay of ~10 seconds between starting each build.
>
> You can see some details in the screenshot of my original post: We have 5
> slaves here,
> one of them was already running one job, all of the others were idle.
>
> It would be nice if the scheduling was configurable. It seems that the
> "Throttle Concurrent
> Builds" plugin [1] works in that direction: it can make sure to never run
> more than X builds
> on one machine. What I'm looking for is not a limited, but a balanced
> scheduling.

The simple-minded strategy is that if you don't want 6 jobs running on
a single machine, don't start 6 executors there - and if running 6
concurrent jobs is OK, just let it go. When you have a larger set of
jobs, randomness will eventually take over. But, I agree that it
would be nice if jenkins would favor the least-loaded machine when
starting a new job that had not been run before and thus does not have
an established location preference. You could disconnect the machine
that has assumed too much of the load and run a few jobs to force them
to migrate.

--
Les Mikesell
lesmi...@gmail.com

[Lukas Rytz]

> It would be nice if the scheduling was configurable. It seems that the
> "Throttle Concurrent
> Builds" plugin [1] works in that direction: it can make sure to never run
> more than X builds
> on one machine. What I'm looking for is not a limited, but a balanced
> scheduling.

The simple-minded strategy is that if you don't want 6 jobs running on
a single machine, don't start 6 executors there - and if running 6
concurrent jobs is OK, just let it go.

Well, I want 6 jobs there, but only if needed - if there is less work to do,

I'd like to distribute them across machines, which gives (much) faster

results..

[Michaël Pailloncy]

Ok, I'm not sure that 6 builds of the same job (for example) will be faster if you run them on 6 slaves or in one slave with 6 slots.
Taking in account that the 6 slaves have the same capacity and are properly sized to accommodate 6 simultaneous builds.
Maybe it could be in interesting experience to lead :)

2012/7/26 Lukas Rytz <lukas...@epfl.ch>

[Adam PAPAI]

Anyway:

Why don't we have a "round-robin" load balance option?

I hate that sometimes jenkins uses 3 full node and the other 10 is unused...

How hard is to implement it?

Micha�l Pailloncy wrote:
> Ok, I'm not sure that 6 builds of the same job (for example) will be
> faster if you run them on 6 slaves or in one slave with 6 slots.
> Taking in account that the 6 slaves have the same capacity and are
> properly sized to accommodate 6 simultaneous builds.
> Maybe it could be in interesting experience to lead :)
>
>

> 2012/7/26 Lukas Rytz <lukas...@epfl.ch <mailto:lukas...@epfl.ch>>
>
>
>
> On Thursday, July 26, 2012 4:43:38 PM UTC+2, mpapo - Micha�l

> Pailloncy wrote:
>
> Ok, sorry ! I had not seen your screenshot.
> Jenkins behaviour does not surprise me. It seems to set the
> priority to slave where a build has already be done.
> Otherwise, why do you need this feature ?
>
>
> Because the builds will complete faster.
>
> I'm sure that if you create 10 jobs and launch 10 builds on
> each, Jenkins will use all available slots without restrict
> builds to the prefered slave.
>
>
> OK, but I obviously don't want to copy the same job configuration 10
> times...
>
>

--
Adam PAPAI
E-mail: wo...@wooh.hu
Phone: +3630 3355735
Web: http://www.wooh.hu

[Lukas Rytz]

That of course depends on the task at hand. In our case, the run times vary between

1h30 and 3h30 depending on the load. If the server is not loaded, our test suite, which

runs tests in parallel, can make full use of all 16 cores.

[Lukas Rytz]

On Thursday, July 26, 2012 5:05:06 PM UTC+2, Adam PAPAI wrote:

Anyway:

Why don't we have a "round-robin" load balance option?

I hate that sometimes jenkins uses 3 full node and the other 10 is unused...

Yes, exactly my point! :)

[Les Mikesell]

If the 6 jobs compete for the same disk spindle, it might be faster to
reduce the number and let jenkins queue some of them - you'd have to
measure the actual time difference but if it is enough to notice you
probably have too many executors.

Also, this will sort-of take care of itself if you have a lot of
different jobs. If your system is only going to ever build the one
job it might be worth looking at the plugin(s) that control
concurrency. Or maybe you can make some different variations of the
job.

--
Les Mikesell
lesmi...@gmail.com

[Sami Tikka]

Why? That's simple: because no-one has implemented it yet. Jenkins is free and open source software. If there is feature that is missing or is important to you, your best bet is to implement it yourself. Just fork the code on github, implement the feature and send it using a pull request.

Other option is to hire someone to do it for you. I believe Cloudbees offers this kind of service. Maybe others too. (I have no affiliation with Cloudbees)

Currently Jenkins seems to strongly favor the slave that already has the project workspace checked out. This is a very valid strategy if the project is large and time-consuming to check out. But clearly it does not play well if your projects are small and you want to execute concurrent builds.

Currently the best you can do is to limit the number of executors on a node.

-- Sami

Adam PAPAI <wo...@wooh.hu> kirjoitti 26.7.2012 kello 18.05:

> Anyway:
>
> Why don't we have a "round-robin" load balance option?
>
> I hate that sometimes jenkins uses 3 full node and the other 10 is unused...
>
> How hard is to implement it?
>

> Michaël Pailloncy wrote:
>> Ok, I'm not sure that 6 builds of the same job (for example) will be
>> faster if you run them on 6 slaves or in one slave with 6 slots.
>> Taking in account that the 6 slaves have the same capacity and are
>> properly sized to accommodate 6 simultaneous builds.
>> Maybe it could be in interesting experience to lead :)
>>
>>
>> 2012/7/26 Lukas Rytz <lukas...@epfl.ch <mailto:lukas...@epfl.ch>>
>>
>>
>>

>> On Thursday, July 26, 2012 4:43:38 PM UTC+2, mpapo - Michaël

[Michaël Pailloncy]

In my opinion, if you have a slave with 6 slots, It should be able to do 6 builds of the same job in the same time that if those 6 builds were allocated on 6 slaves.
For me, it's certainly a problem of capacity calibration of slaves.
If you have some builds that take many resources, be in round-robin does not solve the problem.

2012/7/26 Lukas Rytz <lukas...@epfl.ch>

[Lukas Rytz]

On Thursday, July 26, 2012 5:29:16 PM UTC+2, sti wrote:

Why? That's simple: because no-one has implemented it yet.

Thanks, that's a valuable piece of information!

[domi]

Actually, this plugin already exists at cloudbees: http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-features-even-load-strategy-plugin.cb

Unfortunately it is not OSS

/Domi

[Kamal Ahmed]

I have 2 instances of Jenkins running and without ANY Custom workspace defined.

On 1.471, its /var/lib/jenkins/workspace/<job name>

and on 1.474 its /var/lib/jenkins/jobs/<job name>/workspace

did the location changed, or is this something wrong with my configuration.

Thanks,

-Kamal.

[Jesse Farinacci]

Greetings,

On Sat, Jul 28, 2012 at 7:31 AM, Kamal Ahmed <kamal22...@yahoo.com> wrote:
> I have 2 instances of Jenkins running and without ANY Custom workspace
> defined.

Jenkins doesn't do well with two instances running on the same workspace.

> On 1.471, its /var/lib/jenkins/workspace/<job name>
> and on 1.474 its /var/lib/jenkins/jobs/<job name>/workspace
> did the location changed, or is this something wrong with my configuration.

I sincerely doubt that it was ever /var/lib/jenkins/workspace/<job
name>, on 1.471 or any other release, without you manually tweaking
the job itself to have a custom workspace. This is generally a very
bad idea.

https://wiki.jenkins-ci.org/display/JENKINS/Administering+Jenkins

Anyhow, virtually any problems you are running into are invalid
because of the two Jenkins instances thing.

-Jesse

--
There are 10 types of people in this world, those
that can read binary and those that can not.

[Kamal Ahmed]

The 2 jenkins instances are on two different physical hosts (both Fedora)

Sorry , should have mentioned it earlier

---

From: Jesse Farinacci <jie...@gmail.com>
To: jenkins...@googlegroups.com
Sent: Saturday, July 28, 2012 9:04 AM
Subject: Re: Default location of workspace changed ?

[Brendan Nolan]

I've created a new plugin - https://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin to do just that.

Any feedback appreciated.