Pipeline: Multiproject repository

[Sverre Moe]

I need some help brainstorming a solution for multiproject repositories.

Pipeline building a repository that contains several delivery projects.

The root of the repository contains several delivery projects.

- delivery-project

  - delivery1

    - CMakeList.txt

    - delivery-project-delivery1.spec

    - src

  - delivery2

    - CMakeList.txt

    - delivery-project-delivery2.spec

  - delivery3

    - CMakeList.txt

    - delivery-project-delivery3.spec

    - src

I could keep one Jenkinsfile on the root. Then iterate through all directories and build each delivery.

However when there is an SCM change in only one of the deliveries I only want that one to be built.

This could be solved if I could determine which directory has been changed. Something like this (ignore closure which doesn't work in Pipeline):

changedDirectories.each { directory ->
  dir(directory) {
      doBuild()
  }
}

Though I'm not sure how to get the SCM changes to determine this.

Considering this delivery git repository will only have a master branch, a Multibranch Pipeline project is not necessary, and a standard Pipeline job would suffice.

I could also create a Folder job for delivery-project, and one Pipeline job for each delivery. Each subproject could contain its own Jenkinsfile which would be specified in the configuration in Script Path as delivery1/Jenkinsfile. However when a new delivery project is added to the repository it would require me to manually add a new Pipeline job for it.

[jer...@bodycad.com]

I try to keep 1 Jenkins file per folder into the delevery system and it was a pain in the end. I simplify things by moving the Jenkinsfile into a sub repos (we are using Mercurial). So I keep my Jenkinsfile revision outside the main branch and our repos only point to the proper Jenkinsfile sub repos. So a single change to Jenkinsfile can be spread to all project by just updating sub repos version. So we can branch the Jenkins file for each project into the solution individually. It may not apply to your case but may help to consider something like it. 

Take care, you will need a up to date Jenkins version to do so, I ran into bugs until really recently to make this work properly.

Hope it help,

Jerome

[Thomas Keller]

Multibranch pipelines are really only suitable for single-project multi-branch repos. The farest I got with a multi-project setup was using the Jenkins Repo Plugin (for the Google Repo Tool), but this would reverse the use case you're looking for: It pings all subprojects / subrepos and kicks off a build if any of those changed; you'd then still have to figure out manually what exactly would need to be build once your build has been kicked off.

I however suspect you could achieve this by remembering the last revision a specific subtree / repo was changed yourself (git log -1 path/to/subtree).

[Jinna Balu]

Hi i have the same issue in jenkins pipeline, I have multiple projects in one repo. Is this issue got solved, if solved please post the sample link so that it will be helpful. Thanks in advance. 

[Sverre Moe]

It is still an issue for us anyway.

Maybe the main Jenkinsfile on the repository root could determine which subproject has changes and build them.

Another solution is to create a git submodule for each subprojects.

Would Jenkins notifyCommit work when changes are pushed to a git submodule. It would probably then only check for changes in the main repository, not the submodules and a build will not start since it doesn't have any changes.

I need to do some experimenting here with git submodules.

Probably the best solution would be to create a git repository for each subprojects, thus building them as normal Multibranch projects.

[Pierre Lupien]

This is an old thread I have a solution that worked for me so I thought I'd share.  I am no Jenkins expert so I don't know if this is some kind of horrible hack, but basically the idea is to ignore the scm object provided by the multi-branch pipeline and setup your own.

I use this for a multi-project Java repository and so far it is working well.  I created one Jenkins multi-branch pipeline for each of my Java sub-project and they all point to the same Jenkinsfile.

This solution has 2 restrictions:

  - The Git URL must be in the Jenkinsfile.

  - The name of the Jenkins multi-branch pipeline project must be the same as the name of the sub-project directory.

Example Jenkinsfile:

// Git URL
def projectGitURL = 'https://<myhost>/<mymultiproject-repo>'
def projectGitCredsName = ''
 
//applicationName is derived from Jenkins project name (which is second to last in the full name)
def jobPathElements = currentBuild.fullProjectName.split('/')
def applicationName = jobPathElements[jobPathElements.length >= 2? jobPathElements.length-2: jobPathElements.length-1]
 
// Project's POM file
def projectPom = applicationName + '/pom.xml'
 
// Git Branch/Paths to check for changes
def projectGitWatchedPathRegex = applicationName + '/.*'
def projectGitWatchedBranches = [[name: '*/' + env.BRANCH_NAME]]
def projectReleaseBranchRegex = 'master.*'  

...
pipeline {
     
    agent any;

...
    options {
        buildDiscarder logRotator(artifactNumToKeepStr: '5', numToKeepStr: '5')
        disableConcurrentBuilds()
        timestamps()
        skipDefaultCheckout() //skip SCM pull before first stage, we'll do our own
    }

...
    stages {
        stage('Checkout from SCM') {
            steps {
                //We won't be using the simple "checkout(scm)" here because we want
                //to setup our own path restriction and that is not supported
                //by the default multibranch git provider.
                //checkout(scm)
                 
                checkout([$class: 'GitSCM',  
                    userRemoteConfigs: [[credentialsId: projectGitCredsName,  
                                         url: projectGitURL]],
                    branches: projectGitWatchedBranches,  
                    extensions: [[$class: 'PathRestriction', //SCM poll filter by path
                                    excludedRegions: '',  
                                    includedRegions: projectGitWatchedPathRegex],
                                 [$class: 'LocalBranch',  
                                    localBranch: '**']]])
            }
        }

...
        stage('Build and Deploy') {
            steps {
                withMaven(maven: 'maven') {
                    sh(script: "mvn --batch-mode --errors --update-snapshots -Dbuild_number=${BUILD_NUMBER} -f ${applicationName} clean deploy")
                }
            }
        }
...
    }
}