Pipelines, github fork models and the Jenkinfile

[Arvind Jayaprakash]

I seem to have developed a love/hate relationship with Jenkins 2.x pipelines and some of the concepts that it imposes. Hence, I'd like to hear how best operate in the new world. The one thing I'd like to make clear before I get into the details is that I am suggesting that this the only right way to do things; all I am looking to hear is Jenkins 2.x would be able to support this as one of the many models.

Firstly, let me describe the development model that I'd like to work with. The setup I have is very similar to how a lot of github bases open source projects operate i.e. the main repo very rarely has any branches. All development is done by forking the code, making any number of branches within forks and finally submitting a pull request to the main/parent git repository. There is a fairly elaborate acceptance criteria for a pull request to be accepted into the mainline and this is the workflow that I'd like to trigger and run as a pipeline. The way I have been doing this until now is by having a github pull request builder triggered job that sets of a series of steps within a freestyle job and then having downstream jobs associated with it. It is not clear as to how one would setup something using the pipeline construct.

Secondly, I do not wish to provide the flexibility of defining the pipeline (at least in it's current form) to individual developers. The design choice of needing a Jenkinsfile in the each git repo (and branch) seems equivalent to allowing developers to define their own freestyle jobs in Jenkins 1.x. The model that I have been following is to have a predefined set of templates (setup using the Job DSL) from which the developer can choose along with a minimal set of parameters such as the path of the git repo, the email addresses for notification, choice of JRE, etc. etc. to instantiate specific jobs for their code bases. I'd like to retain this degree of control/standardization as we move into Jenkins 2.x.

In short, I am looking for a way by which I can confine myself to the fork model of git development, pull request based triggers, and job definitions as pipelines, with the pipeline definition residing on jenkins (and not the developer's source tree).

[Mark Waite]

I don't think what you want is available.

You can construct a job which defines the pipeline script for execution in that job.

You can construct a job which automatically defines jobs for every branch where it finds a pipeline script and it will execute that script.

You can construct a job which automatically defines jobs for every branch and will execute a build step (which seems quite close to what you're seeking).  I'm not aware of a way to make a pipeline script a build step.

Can you explain why you don't want others to be able to improve the pipeline definition as they find issues with it?

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5a534ac7-dedb-476f-8748-a25268419fe5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arvind Jayaprakash]

Comments inline

On Sunday, August 14, 2016 at 7:05:57 AM UTC+5:30, Mark Waite wrote:

I don't think what you want is available.

You can construct a job which defines the pipeline script for execution in that job.

This is straightforward as the existing job dsl plugin permits instantiation of pipelines

 

You can construct a job which automatically defines jobs for every branch where it finds a pipeline script and it will execute that script.

You can construct a job which automatically defines jobs for every branch and will execute a build step (which seems quite close to what you're seeking).  I'm not aware of a way to make a pipeline script a build step.

I don't need anything to happen in response to creation of branches. The only time I want the job (pipeline) to run is in response to pull requests. Not sure how make this happen.

 

Can you explain why you don't want others to be able to improve the pipeline definition as they find issues with it?

That is a loaded question. Mine is an enterprise setup wherein there are a whole bunch of engineering practices that gets enforced as part of allowing changes to the mainline. This is one area where I'd wish to retain control and be able to assure myself that the rules/practices/guidelines are being adhered to. Note that there are a handful of patterns and not just a single one that is being followed. These templates cover more than the proverbial 90% cases. This 90% is on a base of around 200-300 git repos. Deviations from the existing set of templates fall into two categories: (a) improvements to what exists (b) something new (experimental) being tried out. Both are encouraged. Have a look at this blog post on the official jenkins blog to get a sense of the organizational dynamics that I am dealing: Jenkins Pipeline Scalability in the Enterprise. A world wherein the following can happen is god enough

• "jenkins team" is able to offer pipelines with restricted set of configuration points
  
• "development team" exercises these configs through a file in the source code (seed.properties) but doesn't get to tinker with the pipeline script
• a way for the "jenkins team" to ensure that the latest pipeline libraries are being used

[Arvind Jayaprakash]

It appears that having the ghprb trigger be pipeline compatible might solve my problems. Most of the other triggers seem to have been ported to be pipeline compatible.

[Jesse Glick]

On Sunday, August 14, 2016 at 1:28:50 AM UTC-4, Arvind Jayaprakash wrote:

A world wherein the following can happen is god enough

• "jenkins team" is able to offer pipelines with restricted set of configuration points
  
• "development team" exercises these configs through a file in the source code (seed.properties) but doesn't get to tinker with the pipeline script
• a way for the "jenkins team" to ensure that the latest pipeline libraries are being used

There is no open-source feature for this that I know of, though you could probably cobble something together. CloudBees offers a proprietary version:

https://www.cloudbees.com/blog/new-angle-pipeline-code

[Arvind Jayaprakash]

The documentation seems to suggest that is does almost everything I want.

[Arvind Jayaprakash]

This plugin is letting me do everything I wanted.
https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Integration+Plugin

The plugin lets me trigger pipelines based on pull request triggers from
github without imposing any conventions on files that needs to be
present in the source trees.

I then able to simply provide my pipeline contents using the standard
pipeline feature wherein a custom file from an SCM repo different from
the one that contains the actual code to be built can be configured.