Pipeline design question

49 views
Skip to first unread message

Sébastien Hinderer

unread,
Aug 11, 2020, 10:33:33 AM8/11/20
to jenkins...@googlegroups.com
Dear all,

When a pipeline needs to run a sequence of several shell commands, I see
several ways of doing that.

1. Several "sh" invocations.

2. One "sh" invocation that contains all the commands.

3. Having each "sh" invocation in its own step.

4. Putting all the commands in a script and invoking that script through
the sh step.

Would someone be able to explain the pros and cons of these different
approaches and to advice when to use which? Or is there perhaps a
reference I should read?

Thanks,

Sébastien.

Gianluca

unread,
Aug 11, 2020, 11:46:45 AM8/11/20
to Jenkins Users
Hi,
from functional point of view there is no different ... but there are from reporting point of view ... and that's a matter of taste :-)

1. In BlueOcean view, each "sh" invocation will be displayed separately from the other steps and "sh" takes as parameter a name that BlueOcean will use to report the steps.

2. It will appears as a single step in BlueOcean and you will have to go into the details and look at the log output to see the result of each "inner" sh command

3. Assuming you wanted to say in its own "stage" and not step (because that's point 1), then each "sh" command will have its own "circle" on BlueOcean pipeline view that becomes green or red based on the outcome

4. It's exactly as point 2, but probably much easier to maintain

Cheers,
Gianluca.

Jérôme Godbout

unread,
Aug 11, 2020, 12:01:09 PM8/11/20
to jenkins...@googlegroups.com
Hi,
this is my point of view only,but using a single script (that you put into your repos make it easier to perform the build, I put my pipeline script into a separated folder). But you need to make sure your script is verbose enough to see where it has fail if anything goes wrong, sinlent and without output long script will be hard to understand where it has an issue with it.

Using multiple sh make it easier to see where it does fail since Jenkins will display every sh invoke.

You can also put a function that will run some sh command into groovy and load that file and execute a command from it. This leave more flexibility from jenkins (decouple jenkins from the task to do) but you still can invoke multiple sh command into that groovy script. So your repos can can contain a groovy entry point that the pipeline will load and invoke that script can call sh, sh scripts and/or groovy scripts as it please.

pipeline script --> repos groovy script --> calls (sh, groovy, shell scripts...)

That avoid high maintance jenkins pipeline, the repos is more self aware of his needs and can more easily changes between version.

I for one, use 3+ repos.
1- The source code repos
2- The pipeline and build script repos (this can evolve aside form the source, so my build method can change and be applied to older source version, I use branch/tag when backward compatibility is broken or a specific version is needed for a particualr source branch)
3- My common groovy, scripts tooling between my repos
4- (optional) my unit tests are aside and can be run on multiple versions

This work well, I wish the shared library was more flexible and that I could more easily do file manipulation into groovy, but I managed some platform agnostic functions for most file/folder/path operations that I reused between project. This make my pipeline script free of thousand of if(isUnix()) and the like. My pipeline look the same for either MacOS/Windows/Linux.

Hope this can help you decide or plan you build architecture.

Jerome
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20200811143317.GB117214%40om.localdomain.

jeremy mordkoff

unread,
Aug 11, 2020, 12:19:26 PM8/11/20
to Jenkins Users
I try to make the calls in my top level jenkinsfile atomic and complete, i.e. each one performs a single function. By using long, descriptive names I can avoid the need for lots of comments. It also makes building new pipelines easy and encourages reuse across files, stages and steps. If I see several sh calls in a row in a jenkinsfile, I am immediately looking at what they are trying to accomplish and if they will always be executed together, then I move them to a script or function. I do make my scripts slightly wordy. I find that 90% of the time CI problems are transient and by the time you enable debugging, the problem can be gone. A good logging convention is a must so that you can run grep on a log file and get a high level summary.

For each function, I look at whether it is likely to have to change when the product code changes versus when the devops code or the infrastructure changes.  If it is coupled to the product code (e.g. scripts. makefiles, dockerfiles, etc), then I create a shell script and store it in the CI folder with the other product code.  Everything else goes into scripts in the devops folder or groovy functions in the devops library file.  The idea is that developers are responsible for maintaining the CI folder and I own the devops folder. E.g. if the product has a new dependency on a debian package, it is up to dev to add it to the dockerfile, not me. But if I move my debian mirror into artifactory, that is on me.

bottom line .. make your top level jenkinsfile readable and isolated from the nitty-gritty details

my $0.02 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

Sébastien Hinderer

unread,
Aug 14, 2020, 8:18:56 AM8/14/20
to Jenkins Users
Dear Gianluca,

Many thanks for your helpful comments! I do not have a strong opinion at
the moment about what is best, but at least I understand the pros and
cons in a better way.

Best wishes,

Sébastien.

Sébastien Hinderer

unread,
Aug 14, 2020, 11:29:33 AM8/14/20
to jenkins...@googlegroups.com
Hello Jérôme, thanks a lot for your response.

Jérôme Godbout (2020/08/11 16:00 +0000):
> Hi,
> this is my point of view only,but using a single script (that you put
> into your repos make it easier to perform the build, I put my pipeline
> script into a separated folder). But you need to make sure your script
> is verbose enough to see where it has fail if anything goes wrong,
> sinlent and without output long script will be hard to understand
> where it has an issue with it.

Indeed. Generally speaking, we activate the e and x shell options to
have command displayed and scripts stop on the first error.

[...]

> I for one, use 3+ repos.
> 1- The source code repos
> 2- The pipeline and build script repos (this can evolve aside form the source, so my build method can change and be applied to older source version, I use branch/tag when backward compatibility is broken or a specific version is needed for a particualr source branch)
> 3- My common groovy, scripts tooling between my repos
> 4- (optional) my unit tests are aside and can be run on multiple
> versions

That's a very interesting workflow, thanks!

So you add all these repositories to your jobs and then they are run
each time one of those repositories is updated, right?

How do things work on slaves? Is each repos cloned in its own directory
in the workspace directory?

> Hope this can help you decide or plan you build architecture.

It helps a lot! Thanks!

Sébastien.

Sébastien Hinderer

unread,
Aug 14, 2020, 11:34:27 AM8/14/20
to Jenkins Users
Many thanks for your contribution Jeremy. Definitely more than $0.02!
;-)

I am not sure how high-level my Jenkinsfiles are at the moment, but at
least we are departing frm defining the jobs in Jenkins' UI, which feels
good and a step in the right direction on which we can always improve
later.

Best wishes,

Sébastien.

Jérôme Godbout

unread,
Aug 14, 2020, 1:30:49 PM8/14/20
to jenkins...@googlegroups.com
**************
So you add all these repositories to your jobs and then they are run each time one of those repositories is updated, right?

Well, I either have unit tests build every night more into scheduled build into Jenkins pipeline options or manually for the distribution build they are done manually with a tag number injected by the user.
I do this because I don't have enough PC performance to run and build on every commit. But in an ideal world I would put a webhook on my repos to trig the Jenkins build when a push is done into the right branch. The Jenkins common library are always taking the head of the master branch, this should always work and have the most recent functions and bug fix (I do not break backward compatibility and if I do, I update all the Jenkinsfile right away). The setup and deploy is not large enough over here to bother just yet, but you could easily use branch to prevent backward compatibility issues with the common library.

**************
How do things work on slaves? Is each repos cloned in its own directory in the workspace directory?

The master scheckout the pipelines repos to fetch the Jenkinsfile from SCM. That repos only contain that file (or many of them) along with build specific groovy/shell... scripts to help the build process. The first thing it does on the slave node is to checkout the common tools and import the needed one inside a subfolder (Amotus_Jenkins/).

Once the tools are loaded, I do checkout the source and start the build as it normally should. I use ENV var to set the branch or other options that will be used to build that repos. Those env are injected by Jenkins parameters options.

The resulting artifacts are sent to artifactory/appcenter/ftp... and test results are analyze right into Jenkins.

That way, the only thing Jenkins known are credentials to repos and the pipeline repos and parameters ask to user. The rest is done inside the pipeline repos (unit test/build/deploy jenkinsfile). The source repos doesn't even known if any CI exist on him, so If I want to build an old version with a recent CI or change my CI it will still work.

The checkout syntax into Jenkinsfile is a bit painful if you have git submodules, but it will track the changes with the right plugins. My first few stage mostly look like this, it's a shame that nearly all my jenkinsfile file look like this but it straight forward once you see it once:

node("PHP && PHPComposer") {
def amotusModules = [:]; // this dictionary hold the tools modules files I made generic for all projects
def amotusRepos = [
[
name: 'Repos Name'
, url: 'https://bitbucket.org/repos2.git'
, branch: "${params.AMOTUS_BRANCH}"
, path: 'SourceRepos2'
]
];

stage('Checkout Tools') {
dir("Amotus_Jenkins") {
checkout([$class: 'GitSCM'
, branches: [[name: 'master']]
, browser: [$class: 'BitbucketWeb', repoUrl: 'https://bitbucket.org/repos.git']
, doGenerateSubmoduleConfigurations: false
, extensions: [[$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false], [$class: 'CleanCheckout']]
, submoduleCfg: []
, userRemoteConfigs: [[credentialsId: 'BitBucketAmotus', url: 'https://bitbucket.org/repo.git']]
]);
}
}

stage('Load option') {
dir(pwd() + "/Amotus_Jenkins/"){
// Load basic file first, then it will load all others options with their dependencies
load('JenkinsBasic.Groovy').InitModules(amotusModules);
amotusModules['basic'].LoadFiles([
'JenkinsPhp.Groovy'
, 'JenkinsPhpComposer.Groovy'
]);
}
}

stage('Checkout Repos') {
amotusRepos.each { repos ->
dir(repos['path']) {
checkout([$class: 'GitSCM'
, branches: [[name: amotusModules['basic'].ValueOrDefault(repos['branch'], 'master')]]
, browser: [$class: 'BitbucketWeb', repoUrl: repos['url']]
, doGenerateSubmoduleConfigurations: false
, extensions: [[$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false], [$class: 'CleanCheckout']]
, submoduleCfg: []
, userRemoteConfigs: [[credentialsId: 'BitBucketAmotus', url: repos['url']]]
]);
}
}
}

// Perform the build/test stages from here
}

That give a good idea on how things are executed on the slave node. I also use node env var to override default path for tools if they not install into the default path or the OS have a special path (I'm looking at you MacOS).

There is still quiet some room for improvement, but I got so little time allowed for my DevOps...

-----Original Message-----
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> On Behalf Of Sébastien Hinderer
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20200814152915.GA143147%40om.localdomain.

jeremy mordkoff

unread,
Aug 14, 2020, 2:51:56 PM8/14/20
to Jenkins Users
How do you maintain and verify backwards compatibility with older releases if you keep your devops code in a separate repo? I keep my devops code in the same repo as the product code so that I know I can always go back and rebuild an older release and get exactly the same results. 

The only exceptions to this is the code and config files for accessing the infrastructure servers, such as my debian mirrors and docker repo in artifactory, including the URLS, the gpg keys and CA certs needed. For this I generate a base docker image from which all of my other images are derived. This base image has all of these things pre-configured plus the latest security updates for all packages (debian, pypi and npm). I can go a month without rebuilding it and then rebuild it 5 times in a week. I run a build and test for all active branches once a week just to be sure any changes to this base image haven't introduced any new issues. It is very rare that this break an old branch and not master, in fact I only remember this happening once. 

My point is my build has changed dramatically over the years and if I had my devops code in it's own repo, I would have to create branches to match the product branches anyway so why not keep it all together? 

Jérôme Godbout

unread,
Aug 14, 2020, 3:04:06 PM8/14/20
to jenkins...@googlegroups.com

The problem I see, is that the DevOps infracstructure doesn’t stay fix in time. It’s really depends on your need. But building into recent Jenkins an old version seem more like a possibility to me then having to revert to an old Jenkins to build an old version. If you have special command to build, this belong into your source solutions or makefile. The Jenkins should mearly call a single command with given option.

 

If you broke backward compatibility, version you pipeline script and branch it. Each branch belong to source branch/version. 

 

Building an old version where the DevOps scripts can no longer work will happen, keeping them together tie them to stick. Might be what you need if you can always provide the same env and build system (OS and Docker for the service). Cause Jenkins might break backward compatibility one days or you might move away from Jenkins. Having the CI into the same repos it kind of lock you down to reuse the same CI that have the same exact API to perform your build.

 

I started at the pipeline beginning and the API was changing fast at first, this helped me a lot, thing are now more stable, but I still do this so I can deploy/build an old version to new infrastructures easily. But again it depends on your use case and how your software is deployed/used.

 

My devops mostly evolve as I add new things, like more linter, code coverage, etc. This allow me to do those thing on older version as well to compare. The build sequence belong to makefile and solution scripts and should not be part of the pipeline. The pipeline is more a sequencer of high level things to do, how to do it specifically should be a script or make target into your project.

 

This how I depart my code, other use case might vary.

Sébastien Hinderer

unread,
Aug 17, 2020, 11:32:25 AM8/17/20
to jenkins...@googlegroups.com
Deear Jérôme,

I really wanted to thank you for having taken the time to share both
abstract ideas, an overall view of your architecture, and also very
valuable code. To tell you the truth, it didn't help me to become friend
with all this Groovy soup, but this is by no means your fault and I am
on the countrary very grateful that you shared these examples!

Your message really widened my perspectives on what I could do with
Jenkins. I don't knonw whether I'll feel brave enough to learn all the
necessary bits, but it's good to see what is possible and to have, in
addition, hints on how things could work.

So thanks a lot, again, Jérôme, for your time and clear explanations.

Sébastien.
Reply all
Reply to author
Forward
0 new messages