Jenkins running as a service restarts by itself after upgrade from 2.303.2 to 2.319.3

[Chris]

Hi,

I upgraded Jenkins (running as a service) from version 2.235.5 to 2.303.2 (by just replacing .war file and restarting the service), which seemed to work pretty stable. Before doing it, I upgraded Java from OpenJDK 8 to OpenJDK 11, which was the prerequisite for Jenkins upgrade. I tried to upgrade even to 2.319.3 or to 2.332.3 and I was able to log in to GUI with my AD credentials after upgrade, however there are some issues which cause Jenkins to restart by itself from time to time when using these 2 releases, so for now I downgraded it back to 2.303.2.

The question is why is this happening ?

When there was Jenkins 2.319.3 or 2.332.3 running, I could have observed the following messages in Jenkins log:

[…] 2022-06-01 11:10:26.854+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 2022-06-01 11:10:27.484+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2022-06-01 11:10:42.114+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 2022-06-01 11:10:42.160+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 2022-06-01 11:10:42.227+0000 [id=35] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 2022-06-01 11:10:46.047+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded 2022-06-01 11:10:47.060+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted 2022-06-01 11:10:47.060+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 2022-06-01 11:10:47.066+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated 2022-06-01 11:10:47.110+0000 [id=57] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata 2022-06-01 11:10:47.168+0000 [id=57] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 49 ms 2022-06-01 11:10:47.398+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization 2022-06-01 11:10:47.524+0000 [id=23] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running 2022-06-01 11:11:48.936+0000 [id=16] WARNING o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761: no checkouts in startTestEsign #43 2022-06-01 11:11:48.997+0000 [id=16] WARNING o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761: no checkouts in startTestEsign #24 2022-06-01 11:11:53.601+0000 [id=25] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty Running from: /storage/jenkins/jenkins.war webroot: System.getProperty(“JENKINS_HOME”) 2022-06-01 11:11:55.083+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @977ms to org.eclipse.jetty.util.log.JavaUtilLog 2022-06-01 11:11:55.327+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file […]

OS is RH 7.9, I don’t know how exactly this Jenkins was installed - it was done by someone else but ‘rpm -qa | grep -i jenkins’ command doesn’t show anything and ‘yum list | grep -i jenkins’ shows only this:

python2-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

python2-jenkins-job-builder.noarch

python36-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

What may be the issue ? Please help with it.

Thanks in advance! 

Chris

[Fabian Cenedese]

At 08:16 10.06.2022, you wrote:

>Hi,
>
>I upgraded Jenkins (running as a service) from version 2.235.5 to 2.303.2 (by just replacing .war file and restarting the service), which seemed to work pretty stable. Before doing it, I upgraded Java from OpenJDK 8 to OpenJDK 11, which was the prerequisite for Jenkins upgrade. I tried to upgrade even to 2.319.3 or to 2.332.3 and I was able to log in to GUI with my AD credentials after upgrade, however there are some issues which cause Jenkins to restart by itself from time to time when using these 2 releases, so for now I downgraded it back to 2.303.2.
>
>The question is why is this happening ?

I can't offer an explanation or even a solution but we too had that
problem of Jenkins restarting. As consequence I downgraded from
Jenkins 2.332.3. It may have been rather a problem with Java 11
but I don't remember exactly. Currently we're using Jenkins
version 2.319.3 (on Ubuntu) with openJDK 8 and that is stable.
Of course we'd welcome a solution to this problem as well.

Thanks

bye Fabi

[Mark Waite]

On Friday, June 10, 2022 at 7:12:40 AM UTC-6 fcenedese wrote:

At 08:16 10.06.2022, you wrote:

>Hi,
>
>I upgraded Jenkins (running as a service) from version 2.235.5 to 2.303.2 (by just replacing .war file and restarting the service), which seemed to work pretty stable. Before doing it, I upgraded Java from OpenJDK 8 to OpenJDK 11, which was the prerequisite for Jenkins upgrade. I tried to upgrade even to 2.319.3 or to 2.332.3 and I was able to log in to GUI with my AD credentials after upgrade, however there are some issues which cause Jenkins to restart by itself from time to time when using these 2 releases, so for now I downgraded it back to 2.303.2.

One issue might be related to the technique you used to perform the upgrade to Jenkins 2.332.3.  Unfortunately, you haven't provided enough information for others to know if you made that mistake or not.

If the original installation used a Linux package (deb or rpm), then you failed to upgrade correctly when you upgraded by replacing the war file.  Please use the Linux package manager to upgrade if you used the Linux package manager for the initial installation.

Please review the changelog and the upgrade guide for the releases.  Specifically note in the Jenkins 2.332.1 changelog that the Jenkins Linux installers switched from System V init to systemd.  Note in the Jenkins 2.332.1 upgrade guide that the Jenkins Linux installers switched from System V init to systemd.  There is also a blog post that describes the details needed for an issue report that will allow others to help with the issue.

Note that the Linux installer change did not affect Jenkins 2.319.3, so if you were seeing restarts in Jenkins 2.319.3, then it was not related to the change from System V init to systemd.

If the original installation used the Windows installer, then you are making a large enough jump in versions that you should probably install with the latest Windows installer rather than replacing the war file.

Another issue might be related to possible bug in Jenkins that is reported as Java 11 requiring more memory that Java 8 for a similar configuration.  Unfortunately, you haven't provided enough information for others to know if that is the issue or not.  One confirmed instance of that report is the metaspace memory leak report in JENKINS-63766.  If that is the specific issue, then updating plugins to most recent versions should resolve it.

If it is none of those things, then you'll need to provide more details that help isolate the cause of the issue.

 

>The question is why is this happening ?

I can't offer an explanation or even a solution but we too had that
problem of Jenkins restarting. As consequence I downgraded from
Jenkins 2.332.3. It may have been rather a problem with Java 11
but I don't remember exactly. Currently we're using Jenkins
version 2.319.3 (on Ubuntu) with openJDK 8 and that is stable.
Of course we'd welcome a solution to this problem as well.

Since Jenkins weekly will end support of Java 8 in June 2022 and Jenkins LTS will end support of Java 8 in September 2022, it is in your best interest to provide enough information so that others can understand and assist with the issue you're seeing.

Mark Waite

[Chris]

As I've already mentioned (maybe it was not very clear - so just to remind):

1. OS is Red Hat Linux 7.9

2. I can't see any valid jenkins package in the system: ‘rpm -qa | grep -i jenkins’ command doesn’t show anything 

3. Running ‘yum list | grep -i jenkins’ shows only these 3 packages:

python2-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

python2-jenkins-job-builder.noarch

python36-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

What else should I check - do you have any suggestions ?

[Mark Waite]

On Friday, June 10, 2022 at 2:06:48 PM UTC-6 Chris wrote:

As I've already mentioned (maybe it was not very clear - so just to remind):

1. OS is Red Hat Linux 7.9

2. I can't see any valid jenkins package in the system: ‘rpm -qa | grep -i jenkins’ command doesn’t show anything 

3. Running ‘yum list | grep -i jenkins’ shows only these 3 packages:

python2-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

python2-jenkins-job-builder.noarch

python36-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

What else should I check - do you have any suggestions ?

Since there is no Jenkins package installed, the system must have a jenkins.war file that someone in your organization used to create their own service to run the Jenkins war file.   Your choices include:

• Adapt your custom systemd service to match the systemd implementation in the Jenkins RPM package.  If there are future changes to the Jenkins systemd implementation, you'll need to update your service definition to match those changes
• Replace your custom systemd service that is running jenkins.war with the standard Jenkins RPM package

Mark Waite