Building the same branch in different repositories/jobs

[Thomas Ferris Nicolaisen]

Hi,

We have a product which is split into a number of different Git repositories. They are built and released separately, but have dependencies on each other that we track by using the same branch-names and tags in each repository.

To illustrate, here are three repositories with three branches in each:

* repo "foo-library"

    master

    2.17.x

    2.16.x

* repo "foo-framework" (depends on foo-library)

    master

    2.17.x

    2.16.x

* repo "foo-application" (depends on foo-framework)

    master

    2.17.x

    2.16.x

Each of these repositories have one job in Jenkins to build and deploy them into our Maven repository. The jobs trigger each other respectively.

This works really well, as long as we stick to using one branch (master). But once we start doing work in the other branches we are running into trouble:

When "foo-library" builds the 2.16.x branch, it should trigger builds downstream and also order them to build the 2.16.x branches in "foo-framework" and "foo-application". 

However, normal behavior is to build the branch with the latest changes (which may, or may not be 2.16.x).

*** Our attempts at a solution ***

We've tried parameterizing the downstream jobs with a ${GIT_BRANCH}, which is passed properly from the upstream job, and using it as branch-specifier in the downstream jobs: origin/${GIT_BRANCH}

However, when a downstream job like foo-application is triggered from an SCM change, the GIT_BRANCH parameter is empty, and the build fails with no branch found.

The knee-jerk reaction to that is to provide a default GIT_BRANCH value in case none is provided, like "origin/master". This limits the SCM-polling to only detect and run builds in the master branch. If we try specifying a default like "origin/*", the wildcard is not evaluated by the Jenkins Git plugin (Could not checkout origin/**).

If we keep the default GIT_BRANCH as "origin/master", and add more branch specifiers (so we have one "origin/${GIT_BRANCH}" and one "origin/*", we come back to the problem that the latest changed branch is built, also when triggered from upstream.

*** The sub-optimal solutions ***

1) Create SCM-trigger-sister jobs, whose only purpose is to scan for changes in SCM, and notify the real jobs with proper GIT_BRANCH parameters.

2) Stick to default GIT_BRANCH=origin/master, and teach our developers to manually trigger jobs in other branches when needed (having them punching in GIT_BRANCH=2.17.x when they want to run that branch).

Problem with the first one is that we actually have around 40 jobs that do various things with our repositories, and about 30 of these would need new trigger-sister jobs. That's a lot of new Jenkins jobs to maintain, for something which sounds easy in principle..

Problem with the second one is as with all manual work, it will be forgotten, old build artifacts will be assumed to be freshly built, confusion ensues, etc.

I've scanned through the Jenkins plugins looking for something like a "conditional branch selector", but haven't found anything. 

If we express some logic in the Git branch specifier that says "if GIT_BRANCH parameter is null, just use the ** specifier", we'd be fine. 

Anyone have any ideas how we can pull this off?

[Sami Tikka]

A pretty pickle you've gotten yourself into.

These are the options that come to mind (in no particular order):

a) Instead of having the 3 jobs with git branch as parameter, make several sets of 3 jobs. Each set has a fixed branch they build from. If creating the jobs is too much manual work, you can automate the creation of the jobs, either with a script or with a Jenkins job.

b) Try to use Parameterized Trigger plugin to trigger downstream builds. Pass a "predefined parameter" in the form of BRANCH_TO_BUILD=$GIT_BRANCH. Apparently the git plugin sets GIT_BRANCH, I just do not know if that is available in the Parameterized Trigger, but I'm fairly sure it should be.

c) Make a new plugin that supports picking up the git branch and passing it on.

d) Install Groovy post build plugin and use a groovy script to dig up the branch and pass it on.

-- Sami

[Thomas Ferris Nicolaisen]

Thanks for the suggestions, Sami. I'll reply in-line:

On Friday, July 6, 2012 7:10:36 PM UTC+2, sti wrote:

a) Instead of having the 3 jobs with git branch as parameter, make several sets of 3 jobs. Each set has a fixed branch they build from. If creating the jobs is too much manual work, you can automate the creation of the jobs, either with a script or with a Jenkins job.

This would explode the number of jobs. We still all use the "All jobs" view when looking at Jenkins, and I would hate getting the team into using Views as default. We're a small team of 5 people with about 90 jobs, which is small enough to everyone keeping an eye on them in one view.

b) Try to use Parameterized Trigger plugin to trigger downstream builds. Pass a "predefined parameter" in the form of BRANCH_TO_BUILD=$GIT_BRANCH. Apparently the git plugin sets GIT_BRANCH, I just do not know if that is available in the Parameterized Trigger, but I'm fairly sure it should be.

Yeah, this is already working. The problem is configuring the downstream jobs' SCM triggering without messing anything up.

 

c) Make a new plugin that supports picking up the git branch and passing it on.

As above, passing the branch along is not a problem. 

 

d) Install Groovy post build plugin and use a groovy script to dig up the branch and pass it on.

Again, passing the branch value from the upstream job works fine.

It's making sure that the downstream build both

* builds the branch $BRANCH_TO_BUILD when passed from upstream, and

* monitors all branches for SCM changes

If there was something like a "Groovy pre build plugin" where I could decide the above logic before building, that would be perfect.

Otherwise, I think I'm left with extending the Jenkins Git Plugin with the option of overriding configured Branch Specifier with some environment variable (OVERRIDE_GIT_BRANCH_SPECIFIER) that I can pass from upstream.

[Thomas Ferris Nicolaisen]

Otherwise, I think I'm left with extending the Jenkins Git Plugin with the option of overriding configured Branch Specifier with some environment variable (OVERRIDE_GIT_BRANCH_SPECIFIER) that I can pass from upstream.

Closely related issue btw: https://issues.jenkins-ci.org/browse/JENKINS-14276