Opinions about aggressive reuse strategy for Jenkins pipelines

[David Karr]

I work for a very large organization, on a project with many teams, working on many microservices, all of which were generated from a home-grown application template system.  Most of the Java-based microservices have a very similar build process. In fact, most of the projects have an identical Jenkinsfile, with some small parameter value differences (which are set in a different system, not in the Jenkinsfile).

Over time, we've evolved the template and common features, including modularizing and cleaning up the Jenkinsfile.  A small shared library was developed, to hold some of the core function definitions, even though the basic structure of the build was still "in-line" in the Jenkinsfile.

The latest version of this template has taken reuse just about as far as it can go.  The entire Jenkinsfile for most projects will be a small properties block to override some defaults (that are not set in the other system), along with a single method call, including the shared library reference.  The shared library has now expanded to specify the entire build process, from that one method call.

There are a handful of services in the collection that will still have some custom changes (all in my domain), so I will reference a branch of the shared library in those Jenkinsfiles.  If my custom changes are eventually accepted as standard (many of them have been), then they will be merged to the master branch and I can change our Jenkinsfiles to use the master branch.

I'm describing all of this for background.  What I'm asking the community is, is this a reasonable strategy?  I can't help the feeling that this is taking reusability a bit too far. I can't give any technical arguments against it, so I'm just looking for some perspective.

[Slide]

Sounds like a good system to me. :-)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f959c755-7a1a-499d-bad7-f0df8da44ce6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Website: http://earl-of-code.com

[Aaron Digulla]

Hi David

I'm describing all of this for background.  What I'm asking the community is, is this a reasonable strategy?  I can't help the feeling that this is taking reusability a bit too far. I can't give any technical arguments against it, so I'm just looking for some perspective.

I think this is a sound strategy. Without this, projects will quickly become snowflakes. Every team will start to invent their own solution for similar problems. Many of those will be haphazard because people don't have time and/or knowledge to understand the underlying issue and correctly fix many problems. So the solutions tend to be brittle and there will be a lot of conversations: "Jenkins sucks. It always breaks" "What happened?" "Well ... [long explanation]" "Hey, we had that too. We just [...]" "Oh."

Regards,

Aaron Digulla

[Ivan Fernandez Calvo]

It is a fair approach, however my opinion is that the pipeline definition (stages and steps) should be in the project pipeline (Jenkinsfile), in the shared library we have only steps that make only one thing an they make it well or steps that combine several others, every step has his own unit test and we release a new version of the library every Monday, all the jenkinsfiles load the ’current’ version of the library that it is the stable one. In that way we can go backwards between versions without change anything in projects. The reason to not put the whole pipeline in the library is that at some point you will start to write spaghetti pipeline code to cover the whole options needed to all your projects, when that fails would be a pain to debug such thing

[Jan Monterrubio]

We’ve taken sort of a middle ground approach with this on our end. We define standard flows for a particular build system (like maven)

The 80-90% use cases are allowable with our pipeline, consumers only define eome configuration options in a closure. 

For the extra 10% use case, we provide steps you can use “checkoutProject” “buildSite” etc, so those non standard pipelines can still make use of our functions. 

This has helped us standardize our disjoint development process, on the pipeline everyone deploys master but only runs verify on branches and PRs. If that doesn’t fit your use case, and the configuration option doesn’t work, we either:

1. Have a call for feedback on new option just to see how many of our consumers need the new behavior 

2. Help the consumer team use our functions to make their own pipeline. 

On Wed, Apr 10, 2019 at 13:46 Ivan Fernandez Calvo <kuisat...@gmail.com> wrote:

It is a fair approach, however my opinion is that the pipeline definition (stages and steps) should be in the project pipeline (Jenkinsfile), in the shared library we have only steps that make only one thing an they make it well or steps that combine several others, every step has his own unit test and we release a new version of the library every Monday, all the jenkinsfiles load the ’current’ version of the library that it is the stable one. In that way we can go backwards between versions without change anything in projects. The reason to not put the whole pipeline in the library is that at some point you will start to write spaghetti pipeline code to cover the whole options needed to all your projects, when that fails would be a pain to debug such thing

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e1ac1fd9-f494-4113-8483-f6f1c2bee739%40googlegroups.com.

[David Karr]

I'm finding that simply moving the entire jenkinsfile into a single vars method works well enough, but they've taken this another step further by moving all the separate "def" methods called from the main method into separate vars methods.  I wouldn't think there'd be any technical reason why this would be a problem, but I'm occasionally seeing weird symptoms with these "second-level" vars methods. I find that sometimes it gets to the point where it calls one of those methods, and then just fails the build at that point without any diagnostic at all.  My workaround for those issues has been to simply make an exact copy of that second-level method and define it as a "def" method inside the main file, and that fixes the problem entirely. I didn't even change any of the code of the method, I just copied it into the main file, and that fixed the problem.