Jenkins Roadmap: Pipeline items

[Oleg Nenashev]

Dear all,

As you may have already seen, we are working on a new public roadmap for the Jenkins project: https://jenkins.io/project/roadmap/ . The goal there is to accumulate and present major initiatives in the Jenkins project, and to facilitate contributions there. Jenkins Pipeline is a critical part of the Jenkins project, and it is one of the most user-facing parts of Jenkins. It would be great to list more ongoing/planned initiatives there, and I invite all SIG members to share their ideas!

Currently we have 3 initiatives listed, based on feedback we have got in the developer mailing list:

• GitHub Branch Source: Support authentication as a GitHub App: https://issues.jenkins-ci.org/browse/JENKINS-57351
• GitHub Checks API integrations
• Pipeline as YAML

Here are some ideas I have in mind and to which I would be interested to contribute:

• Porting of Blue Ocean Pipeline browsing parts into the Jenkins core and Pipeline plugins. https://issues.jenkins-ci.org/browse/JENKINS-33185 

• Promotion support for Pipeline jobs (being able to manually/automatically promote artifacts in sub-jobs). https://issues.jenkins-ci.org/browse/JENKINS-36089
• Container steps: Simple execution steps in Docker or in Kubernetes sidecar containers, without installing Java and Agents on them. No tracking issue at the moment
• Make the Pipeline engine pluggable, support invoking other engines (like Tekton or Jenkins X Pipelines) inside Jenkins while sharing surrounding features like Multi-Branch Pipeline, SCM sources, etc.
• Better filtering of Pipeline Multi-Branch when using GitHub: Support ignoring pull requests by label, draft status, WiP message, etc. Ultimately it should help Pipeline authors to optimize resource usage on their instances

Moe ideas are appreciated, especially ones you are actually planning to work on (or working on them now)! If you are interested in user requests, see this Jira query. It could help to determine some feature areas which could help users to write and diagnose Pipelines.

Best regards,

Oleg

[Marky Jackson]

I personally would like to see better GDSL support be on the roadmap. 

I also am working on a roadmap that I would like to roll up in to the larger roadmap. It is an action item of mine.

--
You received this message because you are subscribed to the Google Groups "Jenkins Pipeline Authoring SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-pipeline-auth...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-pipeline-authoring-sig/c973de29-23d1-4d52-aa05-9f6c45be40d1%40googlegroups.com.

[steven...@gmail.com]

I have been mulling over an improvement to Pipeline jobs for a while based on my experience with fitting my team's build into Pipeline. I'll describe our pipeline as background for the idea, I do understand that the SIG is not necessarily a place for "help me with my pipeline" topics so I'm not trying to gather that kind of feedback :). I'll keep the proposal short and at a medium-high level. I think I know just about enough of the inner workings of pipeline/multibranch jobs to figure out how this might be implemented, but don't want to bog things down with that detail.

Background:

We are building some C++ libraries for many platforms (~15). Each platform is given its own multibranch pipeline job in Jenkins. This is for a few reasons. Each platform is significant enough to warrant its own build history and the ability to trigger a build independently. Each platform needs to have parallel stages to cover toolset versions/configurations. I am using Declarative syntax, so the level of parallelization (with acceptable visualization at least) is limited to 1.

To create a single insertion point for Jenkins, there is an extra "parent" multibranch pipeline job which is simply a parallel stage per platform that triggers the respective platform build on the same branch. This is the only job that responds to github webhooks. This allows for navigation between platform builds for a particular commit. It's a hacky implementation of some kind of parallel-pipeline construct.

This architecture is a bit flimsy and has a few drawbacks. There is a ton of wasted effort by the Jenkins master to index all of those multibranch jobs. Each platform job is a separate Jenkinsfile, and all the Jenkinsfiles look very similar to each which can be a pain to maintain. Ultimately, the platform jobs are essentially disconnected and unrelated as far as Jenkins is concerned. If a user manually triggers a platform build directly, which is sometimes necessary, it can get lost and makes things confusing.

Proposal:

Design a new job type which would be a factory of pipeline jobs by some arbitrary function, similar to how multibranch pipeline is a factory of pipeline jobs by SCM heads.

One possible function could be a list of platform names, as an example relevant to my case. If I configured the job with the list "PlatformA PlatformB PlatformC", it would result in a folder containing Jobs PlatformA, PlatformB and Platform C, all pointing at the same Jenkinsfile if desired. This could nest into multibranch pipeline job. Instead of the SCM head (branch, PR, tags) being a job, it would be an additional folder containing the 3 Platform jobs. Ideally this folder could be runnable to run all of the child jobs. It would come with all of the normal features of folders such as configurable health thresholds.

Obviously this factory function would be an extension point to allow for plugins to cover many more use cases.

Remarks:

Sometimes it seems like the philosophy of the Jenkins Pipeline is an "all-in-one" tool for expressing the entire software build & delivery pipeline from start to finish. It certainly works very well for many projects. I've tried to reconcile the issues I described above in order to port everything into one pipeline, but it just doesn't seem suitable. Having multiple separate pipelines related to the same project must be a valid option. Other platforms, such as Jenkins-X, promote smaller disconnected pipelines that run in parallel as appropriate. The GitHub Checks API seems to assume that each check is independently re-runnable, which is problematic for the traditional Jenkins Pipeline where a certain check might be a stage somewhere in the middle of the pipeline.

I'd love to hear people's thoughts about this proposal. I think there's a lot of potential beyond just my selfish use case. I spoke to a lot of people at Jenkins World who were struggling with a similar parallel scale issue and were seeking better solutions than the weird workaround I described.