Matrix Resource Addition

[Tim Brown]

Hi,

*Sorry if you've al;ready been emailed this, but I got delivery failure notifications when emailing from work*

We have been using Jenkins for a while now (nearly two years) and have come up against a problem. We need to run unit tests on Multiple OSs and against multiple resources. We also need to lock each of those resources while they are in use and not let other jobs use them. Our jobs currently look as Follows:

Job1: RHEL6
Gen1
Gen2
Gen3

Job2: RHEL5
Gen1
Gen2
Gen3
Job3: Windows
Gen1
Gen2
Gen3

These jobs currently run sequentially (Job1 -> Job2 -> Job3) and take 9 hours to complete an hour each job Generation in each job. If we could combine them into one Matrix job and setup the following:
|                    |       Operating Systems      |
|                    |    X     |    X    |    X    |
|    Generations     |    X     |    X    |    X    |
|                    |    X     |    X    |    X    |

And could also configure the Jenkins to lock a resource (Gen 1, Gen 2 and Gen3) then we should be able to cut down the job time significantly (in theory to 3 hours).

As such we have looked into the following plugins that we thought could help us:
Throttle Concurrent Jobs:
We found that this worked at the child level, there is a fix for this: https://issues.jenkins-ci.org/browse/JENKINS-12240 but even then this does not fix our problem (as we add one Category Name per matrix job and this applied to all child jobs).

External Resource Dispatcher:
                We found this had the same problem a above (the Resource is assigned to all the children and there is no way to define which child uses which resource).

Seeing as none of these seem to fit what we are trying to do we were thinking of add the concept of a that can be managed in a similar way to nodes (Manage Jenkins -> Manage Resource -> add/Configure Resource). This would allow people to set labels, names and Number of Executors. The we could add an Axis to Matrix Jobs (e.g. ‘Resources’) to allow people to select a group of slaves and a groups of resources in their jobs.

We think this would give us the functionality we want in an easy to use fashion.

Please let us know what you think. Any advice on best practices or things to avoid would be greatly appreciated.

Cheers,
Tim

P.S. I am hoping to draw up a diagram of the structure we are going for when I get more familiar with Jenkins :)

[Chris Withers]

Hi Tim,

On 17/05/2013 13:42, Tim Brown wrote:
> We have been using Jenkins for a while now (nearly two years) and have
> come up against a problem. We need to run unit tests on Multiple OSs and
> against multiple resources. We also need to lock each of those resources
> while they are in use and not let other jobs use them. Our jobs
> currently look as Follows:

Yep, same here, we have many environments in a matrix, but each point in
the matrix requires a database server (postgres version is one of our
axes) and host to run on (linux distro version is another axis) and only
one point in the matrix can run on a host at one time.

> As such we have looked into the following plugins that we thought could
> help us:
> Throttle Concurrent Jobs:

Well, Matrix jobs have the feel of a horrible hack (check out the fact
that the parent job executes all the pre-build steps, even though
they're never used...) but they're too useful not to use...

Throttle Concurrent Jobs tries to work around some of these hacks, but
it's buggy and the conflicting usecases of some users haven't been
properly coded for in the configuration options...

> We found that this worked at the child level,

I think that's what you want though... Each point in the matrix is a
child, and it's each child that you want to lock resources, right?
You wan the children to run in parallel to speed things up, but not
trample on each other, right?

> there is a fix for this:
> https://issues.jenkins-ci.org/browse/JENKINS-12240 but even then this
> does not fix our problem (as we add one Category Name per matrix job and
> this applied to all child jobs).

I've given up on the Throttle Concurrent Jobs plugin, it's buggy, and
doesn't work. We've fallen back to having a bunch of slaves with
database servers on them that only have one executor. They're tagged up
with db server version and os version and we use those as axes in the
matrix. It works fine, but it means that other builds (C++ compiles,
python and JS static code analysis, etc) can't run on those hosts, which
is a shame as they're quite beefy hosts with plenty of spare cycles...

> Seeing as none of these seem to fit what we are trying to do we were
> thinking of add the concept of a that can be managed in a similar way to
> nodes (Manage Jenkins -> Manage Resource -> add/Configure Resource).
> This would allow people to set labels, names and Number of Executors.

> The we could add an Axis to Matrix Jobs (e.g. �Resources�) to allow

> people to select a group of slaves and a groups of resources in their jobs.

I think the config is all there already with labels on nodes.

I think two things are needed:

- a plugin that actually lets you say how many jobs using a particular
label can run on a machine at one time (ie: 1 in both your and my usecases).

- fix the crazy in matrix parent jobs that mean they execute all their
pre-build steps (which can easily trample child jobs on the same node!)

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[Daniel Beck]

Hi Chris,

On 17.05.2013, at 19:29, Chris Withers <ch...@simplistix.co.uk> wrote:

> I've given up on the Throttle Concurrent Jobs plugin, it's buggy, and doesn't work. We've fallen back to having a bunch of slaves with database servers on them that only have one executor. They're tagged up with db server version and os version and we use those as axes in the matrix. It works fine, but it means that other builds (C++ compiles, python and JS static code analysis, etc) can't run on those hosts, which is a shame as they're quite beefy hosts with plenty of spare cycles...

You can run several slaves on the same host. I've been using this approach to prevent parallel execution of specific jobs for about two years. One slave node is labeled with the required resource (e.g. 'database') and has one executor, the other has no relevant label and has four executors. Run the jobs that don't need exclusive access to the resource on the second node.

Both nodes have their own home directory, and might run as the same or a different user. I've used this approach on Windows (some SCM tool didn't allow parallel checkout/update) and Linux (homebrew build scripts with hard-coded paths for temporary files).

Regards
Daniel

[Tim Brown]

Hey Chris,
I agree that the Matrix jobs could do with some work, for instance I find I keep getting confused between the Master and Child elvels when looking for execution output (as they look the same, but you need to go through on to get to the other).
Also they could make it more clear what sections will be executed on the Child and which on the master job (and make this more configurable - as there are some things you cannot do with matrix jobs that you can do with normal jobs).

However I'm not sure I described our issues properly. Whilst the below is true:

> We found that this worked at the child level,

I think that's what you want though... Each point in the matrix is a
child, and it's each child that you want to lock resources, right?
You wan the children to run in parallel to speed things up, but not
trample on each other, right?

 

The Throttle Concurrent job plug in only allows you to specify labels which will affect all the child jobs.
For example if we specified we wanted to throttle Gen 1, Gen 2 and Gen 3 to one Executor each. Then if we told a Matrix job to use the Throttle concurrent categories Gen1, Gen2 and Gen3, each child would say it's using Gen 1 Gen 2 and Gen 3 - which would in affect make them run serially.
What in fact we want is one child job working on Gen 1, one on Gen2 and one on Gen 3 (so they could all run in parallel - provided the Slave has 3 executors). This is why we thought of adding a new axis as this seems to be the only way to give Child jobs of a Matrix job different arguments.


Regards,
Tim