Linux installers for Jenkins 2.332.1 LTS

[Mark Waite]

I propose that we use the systemd Linux installers for Jenkins 2.332.1.

The Linux installers for Jenkins 2.335 and Jenkins 2.336 weekly releases use systemd instead of System V init.  They include migration steps that read the existing System V init script settings and convert them to systemd override configuration settings.

The systemd service has been the default and preferred system and service manager on most Linux distributions for many years.  Jenkins use of systemd unifies the service management method across all three of our Linux installers and allows us to describe configuration actions for Debian, Red Hat, and SUSE installers with a single set of instructions.

The systemd implementation resolves the following known issues with the installers:

• JENKINS-41218 -  Switch Linux installers from System V init to systemd
• JENKINS-65809 -  Restart the Jenkins service after plugin updates on Debian 11 (bullseye)
• JENKINS-56219 -  Restart systemd service when the controller exits unexpectedly
• JENKINS-31656 - Return zero from RPM init script on successful stop
• JENKINS-67487 -  Correctly report startup result on Amazon Linux 2 installed with the rpm package
• JENKINS-67404 - Do not fail startup with timeout on systems with slow DNS resolution

We prefer to introduce infrastructure changes like this one on the first release in an LTS series and we like to have some experience with the change in weekly before the LTS release.

If we choose to not use systemd for the 2.332.1 release, then the current configuration of the 2.332.1 Linux installer will use the Linux installer from 2.334.  That installer is a variant of the System V Linux installer but without the requirement for the `daemon` program on Debian or the `daemonize` program on Red Hat and SUSE.  It was only available in one weekly release before it was superseded by the installer based on systemd.

Our choices are:

1. Use systemd installer from 2.335 and 2.336 in 2.332.1
2. Use System V init installer from 2.334 (no dependency on daemon program)
3. Use System V init installer from 2.333 and earlier

We'll need additional documentation for managing services with systemd.  Basil Crow and I are willing to work on that documentation.

We'll need entries in the upgrade guide for the systemd transition.  I'm willing to work on that with Basil's help.

Comments and concerns are welcomed,

Mark Waite

[Mark Waite]

I missed another impact if we choose to use the systemd installer.  There changes in the 2.335 Jenkins weekly to support the systemd notify system.  Those changes would need to be included in the 2.332.1 backports for Jenkins core.  Pull requests are https://github.com/jenkinsci/jenkins/pull/6228 and https://github.com/jenkinsci/jenkins/pull/6237

 

Mark Waite

[Basil Crow]

While the System V init code from 2.333 and earlier is the least
likely to surprise us, it remains full of long-standing bugs
(including JENKINS-65809) and depends on third-party daemonization
packages, one of which is not available at all in Fedora EPEL 9. This
code has caused and continues to cause significant pain for users, and
this pain has increased in recent Debian and Red Hat-based
distributions. In my opinion, this code has outlived its usefulness.

The systemd-based packaging first introduced in 2.335 has been
shipping for two weeks without issue. This code fixes a number of
long-standing bugs and improves integration between Jenkins and the
service manager. It also eliminates a number of fragile shell scripts
in favor of a declarative service configuration file. This should
prove easier to support in the unlikely case that there are issues.
Migration from the old configuration format is fully automated and has
been tested in a variety of configurations.

The System V init code from 2.334 is a compromise between these
choices that, in my opinion, would please nobody. True, it addresses
the biggest deficiencies of the old implementation, including
JENKINS-65809 and the Fedora EPEL 9 issue. Yet, it has been tested the
least (only one week) and retains a number of existing issues. Because
of its reliance on fragile shell scripts, this code is difficult to
support. If there are issues with it, the arduous task of supporting
users would end up being throwaway work, as the entire implementation
has changed in later releases with the introduction of systemd
support.

For these reasons, my vote is to backport the systemd code to 2.332.1
LTS. Although this does not meet the usual criteria for backports, I
believe that the benefits mentioned above outweigh the risks.

[Mark Waite]

I plan to submit a proposed pull request today to backport the systemd changes into the stable-2.332 branch.  2.335, 2.336, and 2.337 have all used the Jenkins Linux installer based on systemd.  Issue reports have been rare and frequently related to misuse of files delivered by the package manager.

I hope the pull request will be a good point for discussion and approval.

I don't feel there will be significantly broader testing by waiting 3 months before including the installer that uses systemd.

Mark Waite

[Mark Waite]

I've submitted https://github.com/jenkinsci/jenkins/pull/6333 as the proposal to use the Linux installers based on `systemd`.  Will submit a matching pull request to the packaging repository.

[Mark Waite]

On Friday, March 4, 2022 at 4:47:43 PM UTC-7 Mark Waite wrote:

I've submitted https://github.com/jenkinsci/jenkins/pull/6333 as the proposal to use the Linux installers based on `systemd`.  Will submit a matching pull request to the packaging repository.

https://github.com/jenkinsci/packaging/pull/291 has been submitted for the packaging repository.

As far as I can tell, the packaging repository branch for stable-2.332 is already configured to deliver the systemd.  I believe that we need https://github.com/jenkinsci/jenkins/pull/6333 merged for core or we need to change the packaging branch of stable-2.332 to switch back to System V init

Mark Waite

[Mark Waite]

On Friday, March 4, 2022 at 5:42:03 PM UTC-7 Mark Waite wrote:

As far as I can tell, the packaging repository branch for stable-2.332 is already configured to deliver the systemd.  I believe that we need https://github.com/jenkinsci/jenkins/pull/6333 merged for core or we need to change the packaging branch of stable-2.332 to switch back to System V init

I was wrong when I said that stable-2.332 is already configured to deliver systemd.  The systemd commit is included in the pull request and is not already on the stable-2.332 branch.

 

[Mark Waite]

On Friday, March 4, 2022 at 5:42:03 PM UTC-7 Mark Waite wrote:

On Friday, March 4, 2022 at 4:47:43 PM UTC-7 Mark Waite wrote:

I've submitted https://github.com/jenkinsci/jenkins/pull/6333 as the proposal to use the Linux installers based on `systemd`.  Will submit a matching pull request to the packaging repository.

https://github.com/jenkinsci/packaging/pull/291 has been submitted for the packaging repository.

Both pull requests have been merged.  I'll update the changelog and upgrade guide pull request tomorrow with the Linux installer information.

Mark Waite