Converting to pipeline questions

[Louis Elston]

07/17/19 – wrote this…

We are currently using Windows \ Jenkins 2.107.1 (no pipeline), and I am researching going to pipeline. We have a nightly build job, that fetches from repositories, and submits and waits on other jobs. I see 9 jobs running on the same Master node (we only have a master), at the same time. I am not clear on if we should have one Jenkinsfile or multiple Jenkinsfiles. It will not be a multibranch pipeline, as we do not create test branches and then merge back to a master. In the repository we have product1.0 branch, product2.0 branch etc., and build only one branch (the latest one). While I do like the Blue Ocean editor, it is only for MultiBranch pipelines.

Looking for directions and\or examples on how to convert existing Jenkins non-pipeline systems, to pipeline.  I did find this…https://wiki.jenkins.io/display/JENKINS/Convert+To+Pipeline+Plugin. It does help a little in that it gives you some converted steps, but cannot convert all the steps, and will give comments in the pipeline script "//Unable to convert a build step referring to...please verify and convert manually if required." There is an option "Recursively convert downstream jobs if any" and if you select that, it appears to add all the downstream jobs to the same pipeline script, and really confuses the job parameters. There is also an option to "Commit JenkinsFile" (if doing declarative). I will play with this some more, but it is not the be all and end all of converting to pipeline, and I still am not sure of whether I should be have one or more scripts.

Added 07/26/19 - Let’s see if I have my research to date correct…

A Declarative pipeline (Pipeline Script from SCM), is stored in a Jenkinsfile in the repository. Every time that this Jenkins job is executed, a fetch from the repository is done (to get the latest version of the Jenkinsfile).

A Pipeline script is stored as part of the config.xml file in the Jenkins\Jobs folder (it is not stored in the repository, or in a separate Jenkinsfile in the jobs folder). There is a fetch from the repository only if you put it in (you do not need to do a repository fetch to get the Pipeline script).

Besides our nightly product build, we also have other jobs. I could create a separate Declarative Jenkinsfile for each of them (JenkinsfileA, JenkinsfileB, etc.) for each of the other jobs and store then in the repository also (in the same branch as the main Jenkinsfile), but that would mean that every one of those additional jobs, to get the particular Jenkinsfile for that job, would also need to do a repository fetch (basically fetching\cloning the repository branch for each job, and have multiple versions of the repository branch unnecessarily downloaded to the workspace of each job).

That does not make sense to me (unless my understanding of things to date is incorrect). Because the main product build does require a fetch every time it is run (to get any possible developer check-ins), I do not see a problem doing Declarative Jenkinsfile for that job. For the other jobs (if we do not leave then for the time being in the classic (non-pipeline) format)), they will be Pipeline scripts.

Is there any way of (or plans for), being able to do Declarative pipeline without having to store in the repository and doing a fetch every time (lessening the need to become a Groovy developer)? The Blue Ocean script editor appears to be an easier tool to use to create pipeline scripts, but it is only for MultiBranch pipelines (which we don’t do).

Serialization (restarting a job), is that only for when a node goes down, or can you restart a pipeline job (Declarative or Scripted), from any point if it fails?

I see that there are places to look to see what Jenkins plugin’s have been ported to pipeline, but is there anything that can be run to look at the classic jobs that you have, to determine up front which jobs are going to have problems being converted to pipeline (non supported plugins)?

[jeremy mordkoff]

My personal advice is to pick declarative or scripted pipelines and stick to that everywhere. Switching back and forth always leads to mistakes.  And so keeping the Jenkinsfiles in source control is huge plus and I would never go back to putting any logic inside my jenkins job config. Why are you averse to the repository fetch? 

I found that there were just enough things that I could not do in declarative so I needed to use scripted. 

Finally, I try to keep the nitty gritty details of my build process in scripts in source control. For instance, I don't call make from Jenkins. I call a bash script which calls make (among other things). I found that the serialization restrictions in Jenkins make writing serious scripts for it pretty difficult and frustrating since what looks like a perfectly valid script will throw unintelligible errors when run inside Jenkins. 

I believe serialization is also important for executing code inside of containers or on other nodes, but I do not know that for a fact. 

[Jordan Sokolic]

Louis,

To answer your last question, if you decide to use declarative pipelines in the end you can use the built-in tools like the declarative directive generator and snippet generator. They will provide much the same UI as a classic freestyle project which you configure according to your needs, spitting out the relevant section which you can then copy/paste into your pipeline script. You can read more about how to use these tools at https://jenkins.io/doc/book/pipeline/getting-started/#built-in-documentation. I've found them to be indispensable when starting out.

To expand on Jeremy's answer, it's possible to embed a scripted section into a declarative pipeline. We do this fairly frequently, for example, to define custom variables that can be used later in a pipeline. This example parses some Git information and saves some variables for later:

    stage('Init') {

      steps {

        buildName "${params.branch} @${params.commit[0..9]}"

        checkout([...])

      

        script {

          dir(repo_subdir) {

            commit_before_merge = params.commit

            author_name  = sh(script: "git log -n 1 ${params.commit} --format='%an'", returnStdout: true).trim()

            author_email = sh(script: "git log -n 1 ${params.commit} --format='%ae'", returnStdout: true).trim()

            commit_message      = sh(script: "git log -n 1 ${params.commit} --format='%B'",  returnStdout: true)

            commit_date         = sh(script: "git log -n 1 ${params.commit} --format='%ci'", returnStdout: true).trim()

            addShortText text: author_email, border: 0, background: 'lightBlue'

          }

        }

      }

    }

 Jordan

[Louis Elston]

Studying and playing with pipelines.  I see that you can use Declarative in the Pipeline Scrip window, but it still stores it in the config.xml file.  And I have played with the combination of both Declarative and non Declarative in the same script.

I am trying to understand the Blue Ocean interface, the word "MultiBranch" is throwing me a little.  We do not create test branches, and them merge them back into the master.  In the repository, we have branches for each release of the product, and we rarely go back to previous branches\versions.  So, if I am working on branchV9 right now, do I also need a Jenkinsfile in the Master branch, or any other of the previous version branches?

I have been playing with Blue Ocean (which only does MultiBranch pipelines).  I am on a Windows system, Jenkins 2.176.2, and have all the latest Blue Ocean plugins as of today (1.18.0).  I am accessing a local Git repository (not GitHub), and am running into the following...

If I try to use use “c:\GitRepos\Pipelines1\.git”, i get "not a valid name"...

Why is this happening?

On Monday, July 29, 2019 at 11:40:56 AM UTC-4, Louis Elston wrote:

[Louis Elston]

I believe that this is a bug.  What do I need to do to either get comments, or action on this?

[Mark Waite]

On Tue, Aug 6, 2019 at 9:36 AM Louis Elston <louels...@gmail.com> wrote:

I believe that this is a bug.  What do I need to do to either get comments, or action on this?

I believe it is not a bug.  Blue Ocean is not designed, tested, or expected to work with a git repository on a local file system.  It is designed, tested, and known to work with remote git servers, including GitHub, Bitbucket, and plain Git.  A pull request is pending to include Perforce support as well.

Why doesn't Blue Ocean support git repositories on a local file system?  Git repositories on a local file system are only available from agents that share the same file system.  Most Jenkins best practices include the recommendation, "Do not run builds on the master, use an agent".  Running builds on the master provides the executing job with full access to the file system of the Jenkins master.

Recommendation: Configure a git server and use that git server as your repository.  A git server could be as simple as a Linux computer with a shell account that hosts the bare repository or could include a web interface with Gitea (my favorite for local installation) or Gitlab or could use a remote repository (like GitHub, Bitbucket, Visual Studio, Assembla, Beanstalk, Gitlab, etc.).

For your multibranch question, you need a Jenkinsfile on every branch that you want to run with a Pipeline from SCM.

Blue Ocean is not limited to multibranch Pipelines.  You can use the Blue Ocean editor to create a Pipeline in a git repository that has no Jenkinsfile on any branch.

The Jenkins community is a community.  Members of the community are motivated by different things to decide whether they will respond or not.

In this case, Jenkins 2.107.1 is 15 months old.  The Jenkins community provides security updates for the current long term support release (2.176.2) and current weekly release (2.187).  LTS releases every 3 months.  Jenkins 2.107.1 was released 16 months ago.  That is 5 LTS releases ago.  Some hesitation to respond may be due to the outdated version you're running.  There have been many improvements to Jenkins Pipeline in the 5 LTS versions since Jenkins 2.107.1.

There are many different ways that you can learn more about Jenkins Pipeline.

• Tutorials -  https://jenkins.io/doc/tutorials/
• User Handbook -  https://jenkins.io/doc/book/pipeline/
• Jenkins Minute videos -  https://jenkins.io/blog/2017/08/08/introducing-jenkins-minute/
• CloudBees' free Pipeline Fundamentals core -  https://standard.cbu.cloudbees.com/cloudbees-university-jenkins-pipeline-fundamentals
• Udemy courses on Jenkins Pipeline -  https://www.udemy.com/courses/search/?src=ukw&q=jenkins%20pipeline

Mark Waite

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/912b3b5f-a204-4b83-ad35-9fd94dcb2aa4%40googlegroups.com.

--

Thanks!

Mark Waite

[Louis Elston]

I guess that either I misread this page, or it is out of date already...https://jenkins.io/doc/book/blueocean/creating-pipelines/ .  It says "You now need to specify a local or a remote repository from which to build your Pipeline project".  I was trying to test converting our existing system to pipeline, and getting permission and funding for another large GitHub repository was not in the cards right now, so, I was attempting to do this work on a test system, with the repository there.  I will try a Git repository on a different\remote system.  We do not have Linux, and I do not have that experience (only Windows). 

It also said "Note: Under the hood, a Pipeline project created through Blue Ocean is actually "multibranch Pipeline". Therefore, Jenkins looks for the presence of at least one Jenkinsfile in any branch of your repository.  So, your comment "Blue Ocean is not limited to multibranch Pipelines," is news to me.

My version of Jenkins is actually 2.176.2 (on my test VM), as I did recently upgrade it.  When I mentioned 2.107, I was probably looking at the actual production system.   I cannot even keep the current classic Jenkins build system up to date (I see all kinds of red warnings when I go into Manage), but it is feared that any change could possible cause a problem that could delay a build or the release of the product.  So thus I try to work on a test system.  We are behind corp firewalls, so they that figure that the production system is safe.  

I had worked my way through the cloudbees Blue Ocean tutorial, but only with help from them.  I cannot recommend it.  I have found over the years, that people who put together online course and write books, have someone who is already familiar with the context, and can fill in the missing pieces, do the reviewing.  No one uses someone new to the subject matter try to work through it, to see if it really can be understood, and completed successfully. Unless you are a real DevOps type, it can be difficult to understand the documentation.  Also a lot the courses\tutorials\books are using containers.  Again something that I do not see them allowing me to play with, so that rules them out.  I had recently purchased the book "Beginning Jenkins Blue Ocean".  No where in the online information did it mention Docker, but that is what the book requires.

I do appreciate you responding, and I will look at the links that you listed.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/912b3b5f-a204-4b83-ad35-9fd94dcb2aa4%40googlegroups.com.

--

Thanks!

Mark Waite

[Louis Elston]

Mark Wrote..."Blue Ocean is not limited to multibranch Pipelines.  You can use the Blue Ocean editor to create a Pipeline in a git repository that has no Jenkinsfile on any branch."

Can someone point me to an example of this?  I have a GitHub repository with a master branch and a branch1 branch.  I used Blue Ocean, selected "new pipeline", selected to store the JenkinsFile in the Master, it created the pipeline.  When I select that particular new job, on the left hand side, it shows "Scan repository now, , ,Delete Multibranch Pipeline".  In other words, this is Multibranch pipeline.  Where is the option to use Blue Ocean to either create, or edit, a Non-MultiBranch pipeline?  What is it that I am missing or not understanding here?   

On Tuesday, August 6, 2019 at 12:07:24 PM UTC-4, Mark Waite wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/912b3b5f-a204-4b83-ad35-9fd94dcb2aa4%40googlegroups.com.

--

Thanks!

Mark Waite

[Mark Waite]

When you used those steps in Blue Ocean, you defined a Pipeline in the branch where the Jenkinsfile was stored by Blue Ocean.  I think that is what you wanted in the SCM repository.  You're correct that Blue Ocean created a multibranch pipeline as part of that editing process.

If you'd like a job which is not a multibranch Pipeline, create that job interactively with the Jenkins "New Item" menu.  Choose "Pipeline" as the item type and use "Pipeline from SCM" with the repository name and the branch name that you want to create.  In that case, the Jenkinsfile must already exist in the repository.  That's a less typical use case, since most users prefer to have Pipelines automatically created and deleted as branches are created and deleted on their git repository.

If that is your preferred working model, then you could use Blue Ocean to create the Pipeline, delete the job which Blue Ocean created, then interactively create a Pipeline job which is not a multibranch Pipeline.

Once the job is created (either through Blue Ocean as a multibranch pipeline or through the interactive "New Item" as a pipeline or through an organization folder), then you can use Blue Ocean to launch jobs and to view the execution progress of the pipeline.

Thanks,

Mark Waite

[Mark Waite]

Thanks also for detecting that issue.  The documentation does say that local repositories are supported.  I was wrong.  However, there are important limitations with local repositories that make them a poor choice for any user and an especially poor choice for first time users.

I've submitted https://github.com/jenkins-infra/jenkins.io/pull/2411 to guide users towards remote repositories.

Thanks!

Mark Waite

[Louis Elston]

This morning, I basically did what you just recommended.  I created a new Pipeline job (not using Blue Ocean), selected "Pipeline Script from SCM', and pointed to the Jenkinsfile that I had created yesterday in the master branch.  Because this job is not a MultiBranch job, even thought you can run it in Blue Ocean, because there are no 'Branches', the pipeline editor pencil will not appear....you cannot edit the script in Blue Ocean.

Are there any plans to modify Blue ocean so that the editor can be used on any declarative script, MultiBranch or not?  If not, what is your recommended alternative (besides the Pipeline Syntax \ Declarative Directive Generator)?

Being that our development process does not involve test branches merged up into the master, and instead is one branch per version of the product where all development work is done, using a Multibranch job does not fit our needs.  At least that is my impression, in that we do not want to possibly have builds being run for any previous versions of the product just because they have a Jenkinsfile.

[Mark Waite]

On Wed, Aug 14, 2019 at 8:36 AM Louis Elston <louels...@gmail.com> wrote:

This morning, I basically did what you just recommended.  I created a new Pipeline job (not using Blue Ocean), selected "Pipeline Script from SCM', and pointed to the Jenkinsfile that I had created yesterday in the master branch.  Because this job is not a MultiBranch job, even thought you can run it in Blue Ocean, because there are no 'Branches', the pipeline editor pencil will not appear....you cannot edit the script in Blue Ocean.

Are there any plans to modify Blue ocean so that the editor can be used on any declarative script, MultiBranch or not?  If not, what is your recommended alternative (besides the Pipeline Syntax \ Declarative Directive Generator)?

There are no plans to modify the Blue Ocean editor to be used on pipelines outside of multibranch.

You might try using a multibranch pipeline and define the multibranch pipeline to build only the branches that you specify.  Multibranch pipelines can be defined to build only a subset of the available branches.  That would allow you to choose which branches are built based on the job definition of the multibranch pipeline and patterns for branch names, rather than creating individual jobs for each branch yourself.

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2f528dcf-ccc2-4031-8ec2-18496f1a4921%40googlegroups.com.

--

Thanks!

Mark Waite

[Louis Elston]

" build only the branches that you specify.".  I am assuming that this is something in the script that does this, as when creating the MultiBranch job, I see no option to allow for the selection of doing executing the job for only one branch.  Not to harp on the documentation, but if this can be done, then document it up front (with an example).  This would make more users understand that Blue Ocean may be more applicable than it currently appears.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2f528dcf-ccc2-4031-8ec2-18496f1a4921%40googlegroups.com.

--

Thanks!

Mark Waite

[Mark Waite]

On Wednesday, August 14, 2019 at 1:34:10 PM UTC-7, Louis Elston wrote:

" build only the branches that you specify.".  I am assuming that this is something in the script that does this, as when creating the MultiBranch job, I see no option to allow for the selection of doing executing the job for only one branch.  Not to harp on the documentation, but if this can be done, then document it up front (with an example).  This would make more users understand that Blue Ocean may be more applicable than it currently appears.

Documentation pull requests are certainly welcomed and encouraged.  See https://github.com/jenkins-infra/jenkins.io .

The multibranch pipeline that is created by Blue Ocean has a "Configure" page that allows you to limit the branch names which can be built..  In the "Behaviors" section of that "Configure" page you select the "Add" button and then select the row labeled "Filter by name (with wildcards)".  If you need something more specific than wildcards will allow, you can use "Filter by name (with regular expressions)".

I have a private GitHub repository that includes several branches.  I confirmed that it was able to filter by branch name with wildcards.

Thanks,

Mark Waite

[Louis Elston]

In Blue Ocean, if you create a new pipeline, and there is a Jenkinsfile in any branch in that repository, when you select “Create Pipeline”, it will execute the Jenkinsfile in each of those branches.  At this point, there is no ability to Configure anything.  Yes, if you then configure that new job you can restrict it to only execute on a particular branch.  This does not solve my problem.  Lets say that in ProductVersion1 branch there was a Jenkinsfile.  We are now Working on Version 2 of the product, and now have ProductVersion2 branch, which also has a JenkinsFile.  If I create a new Blue Ocean pipeline job (which initially cannot be configured to only execute on the new branch), then I am also going to execute a build of the previous branch (which I do not want to do because that version of the product has been released).  How to handle this, is it done in the script, in that I would need to edit the Jenkinsfiles in both branches before creating the new job?

[Mark Waite]

On Thu, Aug 15, 2019 at 12:20 PM Louis Elston <louels...@gmail.com> wrote:

In Blue Ocean, if you create a new pipeline, and there is a Jenkinsfile in any branch in that repository, when you select “Create Pipeline”, it will execute the Jenkinsfile in each of those branches.  At this point, there is no ability to Configure anything.  Yes, if you then configure that new job you can restrict it to only execute on a particular branch.  This does not solve my problem.  Lets say that in ProductVersion1 branch there was a Jenkinsfile.  We are now Working on Version 2 of the product, and now have ProductVersion2 branch, which also has a JenkinsFile.  If I create a new Blue Ocean pipeline job (which initially cannot be configured to only execute on the new branch), then I am also going to execute a build of the previous branch (which I do not want to do because that version of the product has been released).  How to handle this, is it done in the script, in that I would need to edit the Jenkinsfiles in both branches before creating the new job?

Would it work for you if you took the following steps?

1. Use Blue Ocean to create the first Pipeline on the branch that is the current product version.  Blue Ocean will create a multibranch pipeline job that will build all branches that contain a Jenkinsfile.  At this point, there is only one branch, so only one job will be created
2. Edit the multibranch Pipeline job that Blue Ocean created, add the filter to specifically limit that job to only build the precise branch that is the currently active development branch
3. When the time comes in the future that you are ready to add a Jenkinsfile on a new, independent branch of the same repository, add it from Blue Ocean or copy it from one branch to another with Git operations.  Commit to the new branch, knowing that the new branch won't be built because it does not match the pattern for branches being selected for the multibranch pipeline definition
4.  When you're ready to enable the new branch, change the definition of the multibranch pipeline to build both old and new branches.
5. When you're ready to stop building the old branch, change the definition of the multibranch pipeline to only build the new branch

I think that workflow will allow you to use the multibranch pipeline job to precisely define which branches are built and which are not built.  Whether or not a Jenkinsfile exists on a branch, the multibranch pipeline job would define which branches are allowed to build and which are ignored.

Thanks,

Mark Waite

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bd85b6f0-09f1-4c54-a2da-dfbbef9d9690%40googlegroups.com.

--

Thanks!

Mark Waite

[Louis Elston]

Currently (in the classic\non pipeline jobs), when a product version is released, we disable that job (ex productversionv1), and copy the job to productversionv2 and set the job parameters for the correct branch to use.  Occasionally we do need to do a fix in v1, so it is just a matter of enabling the job, running it, and disabling it again. Quick, simple, easy to do.  Both v1 and v2 jobs are always available.

 

#4 “change the definition of the multibranch pipeline”.  That means that for the scenario

 above, that I must modify the job configuration a few times.  Once to run v1, then to set back to run v2, and no option to have them both available to run separately.

 

The reason that I wanted to use Blue Ocean is for the pipeline editor, I did not want to have to get any deeper than I needed to in Groovy (although I believe that it will not be possible to eliminate its use).

 

I am not seeing Blue Ocean MultiBranch job as the way to go (and you can only do MultiBranch in Blue Ocean).  I am even hesitant to store the Jenkinsfile\script in SCM (scripted or declarative).  While currently I am the only person touching the build system, who knows what the future holds.  The next person to maintain the build system may create a MultiBranch job that will then pick up and execute all the Jenkinsfiles in the separate versioned branches, thus creating builds that are unwanted and updating those older version branches.  Thus, I am considering storing the pipeline script in the config.xml file instead of a Jenkinsfile.

 

If anyone has any other suggestions…I am listening.  Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bd85b6f0-09f1-4c54-a2da-dfbbef9d9690%40googlegroups.com.

--

Thanks!

Mark Waite

[Louis Elston]

Installed Eclipse and https://marketplace.eclipse.org/content/jenkins-editor, and can edit Declarative scripts (regular pipeline job, not Blue Ocean MultiBranch job).  Storing the Jenkinsfile in SCM, but naming it different (ex: Jenkinsfile9-2), so this way if anyone does create a MultiBranch pipeline later, it will not be picked up and executed.  Also put code in script to not execute if the branch parameter passed in is not the correct branch...

pipeline {

    agent any

parameters {

string(defaultValue: '9.2', description: 'Branch to build', name: 'BUILD_BRANCH')

}

stages {

      stage('Run if the correct branch was selected') {

    when {

  expression {params.BUILD_BRANCH == "9.2"}

    }

    stages {

      stage('9-2 Build') {

steps {

script {

  if (env.BUILD_BRANCH == '9.2') {

echo 'Branch is 9.2'

  } else {

echo BUILD_BRANCH

echo "Branch is ${env.BUILD_BRANCH}"

  }

}

}

          }   

  stage('Test') {

            steps {

              echo '9-2 Testing..'

            }

  }

  stage('9.2 Deploy') {

            steps {

              echo 'Deploying....'