Jenkins LTS Release Line: Frequency / Process of Upgrading Jenkins on LTS release schedule

[Jason LeMauk]

I am currently working to setup a Jenkins server for continuous integration. One area of Jenkins maintenance I am looking at is Jenkins upgrades.

 

As gaining access to new Jenkins features is less important to our setup than receiving important bug fixes and general system stability, I am considering upgrading our Jenkins instance on the LTS release schedule. It should be noted that we are currently working with Jenkins version 2.46.3 and would start this schedule by upgrading to Jenkins version 2.60.1.

 

In looking at the documentation for [Jenkins LTS Release Line][1], it looks like this would involve upgrading our Jenkins from the previous LTS version to the new LTS version every 6 - 9 weeks:

 

> The cycle starts with picking an LTS baseline at week 0. Then, there

> is a two week period for backporting followed by two weeks for testing

> the release candidate resulting in the release of X.1. Backporting and

> RC testing is repeated twice, producing X.2 and X.3. This concludes

> the cycle for a given baseline and the new one is started immediately.

>

> The baseline release is typically between 2-5 weeks old when it is

> chosen, so X.1 LTS releases are published about 6-9 weeks after their

> baseline.

 

Am I correct in this understanding that if we were to keep up with the LTS release schedule we would be upgrading our Jenkins instance about every 6 - 9 weeks?

 

Also, from what I have seen, if your Jenkins instance is installed via Jenkins WAR file, then the process for upgrading the Jenkins instance to the most recent Jenkins LTS version is:

 

1. Stop Jenkins running as a service.

2. Back up the Jenkins $HOME_DIRECTORY.

3. Download the latest LTS WAR file.

4. Replace the WAR file currently in $JENKINS_HOME directory with the

    newest LTS WAR.

5. Start Jenkins running as a service.

6. Upgrade any plugins if necessary.

7. Restart Jenkins as a service if necessary for plugin installation.

7. Work out any Jenkins job issues caused by upgraded plugins.

 

Is there anything here I'm missing as far as getting our Jenkins instance onto the LTS release schedule? Is this generally best practice? Just wanted to get some input and advice from those with experience maintaining a Jenkins instance on the LTS release schedule.

 

  [1]: https://jenkins.io/download/lts/

[Mark Waite]

That is the technique I've used, though I've preferred to simplify the upgrade process by using either Debian or Ubuntu as the host operating system, then installing Jenkins  from the "apt" package manager.  That simplifies the "upgrade and copy the war" step.  It doesn't really remove any of the other steps.

If you're a Red Hat / CentOS type, then you'll use the rpm based distribution for the same benefit.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/BY2PR12MB0599FFD3DBC9C44D93D0E8C589D60%40BY2PR12MB0599.namprd12.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

[Jason LeMauk]

Thank you for the information! As a matter of fact we are using Ubuntu.

 

As part of the upgrade process, I’ve seen Jenkins administrators do the following:

1.      Stop Jenkins running as a service.

2.      Rename Jenkins current WAR file to jenkins.war.old.

3.      Copy the new WAR file into the $JENKINS_HOME directory.

4.      Start Jenkins running as a service.

5.      Verify Jenkins is working as expected.

6.      Remove the jenkins.war.old file from the $JENKINS_HOME directory.

 

I’d like to keep the old WAR file in the directory until we can verify the new WAR works as expected.

For this reason, I believe it may not be better to wipe out the original WAR file in $JENKINS_HOME with an apt upgrade.

 

Also, if the Jenkins package is installed from ‘apt’, wont a sudo apt get update / upgrade automatically upgrade my Jenkins instance?

 

Thank you again for your advice!

-Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGgvhAbFy52wRS3%3DsaSTeDko5VN%2BjMDVo4Hxj%2BjPh6Zhg%40mail.gmail.com.

[Mark Waite]

Replies inline for clarity.

On Mon, Jul 3, 2017 at 2:46 PM Jason LeMauk <jason....@csquaredsystems.com> wrote:

Thank you for the information! As a matter of fact we are using Ubuntu.

 

As part of the upgrade process, I’ve seen Jenkins administrators do the following:

1.      Stop Jenkins running as a service.

2.      Rename Jenkins current WAR file to jenkins.war.old.

I didn't find a lot of value from a "rename to old" step, since an upgrade failure will likely force me to either press forward with a fix / work around, or that I undo the installation of the upgrade.  I don't remember the last time I had to undo, but https://askubuntu.com/questions/138284/how-to-downgrade-a-package-via-apt-get describes how you can use apt to install a specific version.  That should provide the equivalent of a "downgrade", without bothering with copying the file to safety.

 

3.      Copy the new WAR file into the $JENKINS_HOME directory.

4.      Start Jenkins running as a service.

5.      Verify Jenkins is working as expected.

6.      Remove the jenkins.war.old file from the $JENKINS_HOME directory.

 

I’d like to keep the old WAR file in the directory until we can verify the new WAR works as expected.

For this reason, I believe it may not be better to wipe out the original WAR file in $JENKINS_HOME with an apt upgrade.

 

Also, if the Jenkins package is installed from ‘apt’, wont a sudo apt get update / upgrade automatically upgrade my Jenkins instance?

 

Yes it will upgrade your Jenkins instance unless you pin that Jenkins version.

Mark Waite

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/BY2PR12MB0599DA08EA206C880190B22189D60%40BY2PR12MB0599.namprd12.prod.outlook.com.

[Jason LeMauk]

If a renamed copy of the old WAR file exists in $JENKINS_HOME directory, wouldn’t it just be a matter of changing the previous jenkins war from jenkins.war.old to jenkins.war, and removing the newly copied Jenkins WAR from the $JENKINS_HOME directory? Isn’t that essentially the undo of the upgrade if a problem occurs?

 

And just to be clear, the previous Jenkins WAR file is never copied out of the $JENKINS_HOME directory; it always stays in the same directory and is just renamed. The copy to safety happens in my Jenkins job which copies and tars the $JENKINS_HOME directory to a local directory as well as source control.

 

I’m more trying to use that as a way to quickly rollback the upgrade (a matter of renaming the previous WAR file and deleting the new one), if anything problematic arises.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFyttEXHbT9nemNJVpsKE-ShuSjcbQr1QPsW%3DFPBzd7Aw%40mail.gmail.com.

[Mark Waite]

On Mon, Jul 3, 2017 at 3:05 PM Jason LeMauk <jason....@csquaredsystems.com> wrote:

If a renamed copy of the old WAR file exists in $JENKINS_HOME directory, wouldn’t it just be a matter of changing the previous jenkins war from jenkins.war.old to jenkins.war, and removing the newly copied Jenkins WAR from the $JENKINS_HOME directory? Isn’t that essentially the undo of the upgrade if a problem occurs?

 

And just to be clear, the previous Jenkins WAR file is never copied out of the $JENKINS_HOME directory; it always stays in the same directory and is just renamed. The copy to safety happens in my Jenkins job which copies and tars the $JENKINS_HOME directory to a local directory as well as source control.

 

I’m more trying to use that as a way to quickly rollback the upgrade (a matter of renaming the previous WAR file and deleting the new one), if anything problematic arises.

I think I understand your rationale.  I don't have any solid objection to it, since you seem ok with not using a package manager to track your Jenkins versions.

I'm quite attached to using the package manager to manage software on my Linux machines.  I'm attached enough to the package manager that I'm willing to accept other compromises (like rollback may take longer).

Since you're willing to manage your own war file installation, you could consider taking it one step further, and use a Docker image to host the execution of your Jenkins server.  That might allow you to switch between old and new by passing a different argument for the docker image to use as the base version.

 

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/BY2PR12MB0599074C5316FF42535FECD789D60%40BY2PR12MB0599.namprd12.prod.outlook.com.