Multi-Pipeline job idea

[Steven Foster]

(I posted this in the SIG Roadmap ideas topic, but don't want to clutter that thread with discussion about this idea.)

I have been mulling over an improvement to Pipeline jobs for a while based on my experience with fitting my team's build into Pipeline. I'll describe our pipeline as background for the idea, I do understand that the SIG is not necessarily a place for "help me with my pipeline" topics so I'm not trying to gather that kind of feedback :). I'll keep the proposal short and at a medium-high level. I think I know just about enough of the inner workings of pipeline/multibranch jobs to figure out how this might be implemented, but don't want to bog things down with that detail.

Background:

We are building some C++ libraries for many platforms (~15). Each platform is given its own multibranch pipeline job in Jenkins. This is for a few reasons. Each platform is significant enough to warrant its own build history and the ability to trigger a build independently. Each platform needs to have parallel stages to cover toolset versions/configurations. I am using Declarative syntax, so the level of parallelization (with acceptable visualization at least) is limited to 1.

To create a single insertion point for Jenkins, there is an extra "parent" multibranch pipeline job which is simply a parallel stage per platform that triggers the respective platform build on the same branch. This is the only job that responds to github webhooks. This allows for navigation between platform builds for a particular commit. It's a hacky implementation of some kind of parallel-pipeline construct.

This architecture is a bit flimsy and has a few drawbacks. There is a ton of wasted effort by the Jenkins master to index all of those multibranch jobs. Each platform job is a separate Jenkinsfile, and all the Jenkinsfiles look very similar to each which can be a pain to maintain. Ultimately, the platform jobs are essentially disconnected and unrelated as far as Jenkins is concerned. If a user manually triggers a platform build directly, which is sometimes necessary, it can get lost and makes things confusing.

Proposal:

Design a new job type which would be a factory of pipeline jobs by some arbitrary function, similar to how multibranch pipeline is a factory of pipeline jobs by SCM heads.

One possible function could be a list of platform names, as an example relevant to my case. If I configured the job with the list "PlatformA PlatformB PlatformC", it would result in a folder containing Jobs PlatformA, PlatformB and Platform C, all pointing at the same Jenkinsfile if desired. This could nest into multibranch pipeline job. Instead of the SCM head (branch, PR, tags) being a job, it would be an additional folder containing the 3 Platform jobs. Ideally this folder could be runnable to run all of the child jobs. It would come with all of the normal features of folders such as configurable health thresholds.

Obviously this factory function would be an extension point to allow for plugins to cover many more use cases.

Remarks:

Sometimes it seems like the philosophy of the Jenkins Pipeline is an "all-in-one" tool for expressing the entire software build & delivery pipeline from start to finish. It certainly works very well for many projects. I've tried to reconcile the issues I described above in order to port everything into one pipeline, but it just doesn't seem suitable. Having multiple separate pipelines related to the same project must be a valid option. Other platforms, such as Jenkins-X, promote smaller disconnected pipelines that run in parallel as appropriate. The GitHub Checks API seems to assume that each check is independently re-runnable, which is problematic for the traditional Jenkins Pipeline where a certain check might be a stage somewhere in the middle of the pipeline.

I'd love to hear people's thoughts about this proposal. I think there's a lot of potential beyond just my selfish use case. I spoke to a lot of people at Jenkins World who were struggling with a similar parallel scale issue and were seeking better solutions than the weird workaround I described.

[Oleg Nenashev]

Hi Steven,

Sorry for the late follow-up, and thanks for your proposal! It would be definitely interesting for many practical use-cases, not only in the area of multi-platform support. The use-case can be partially addressed by Pipeline Templating engine and seed jobs in JobDSL, but indeed there is no easy way to implement it out-of-the-box in Jenkins.

For example, there is a long-standing request to have a Promotion support for Pipeline - JENKINS-36089. It is actually one of the most voted issues in Jenkins Jira. The problem there is that there is no legitimate way in Jenkins to have a "sub-job" or a "sub-Pipeline" for job definitions in Jenkins. Old Promoted Builds plugin works it around by somewhat abusing Jenkins core APIs, and it leads to numerous issues there. If we had a native support for "sub-jobs API" in Jenkins, it would de-finitely unlock many use-cases.

Best regards,

Oleg

[Steven Terrana]

I'm obligated to mention the Jenkins Templating Engine (Templating Engine Plugin in the update center). 

The idea behind JTE is that it allows you to create tool-agnostic, templated pipelines that can be shared across multiple projects. 

There has been a Jenkins Meetup and there is some extensive documentation.

Curious how much of your ask this can meet?