[JIRA] [core] (JENKINS-29888) Jenkins stuck when concurrent builds of a same job due to log rotator

[tazi.otmane@yahoo.fr (JIRA)]

Otmane Tazi created an issue

Jenkins / JENKINS-29888 Jenkins stuck when concurrent builds of a same job due to log rotator Issue Type: Bug Assignee: Unassigned Components: core Created: 11/Aug/15 9:14 AM Priority: Minor Reporter: Otmane Tazi Hello, I use Jenkins with parallel builds to checkout many packages. Example: parallel ( build(build("CheckOut-Package", PACKAGE_NAME: "Package1"); build(build("CheckOut-Package", PACKAGE_NAME: "Package2"); ... build(build("CheckOut-Package", PACKAGE_NAME: "Package10"); ) Each of these jobs calls the log rotator when finished. Some time, next log appears: SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467) at hudson.model.Run.execute(Run.java:1808) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:374) In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked. Since log Rotator should not be a blocking task, I added an exception catch in this PR: https://github.com/jenkinsci/jenkins/pull/1790 That was enough on my side to handle this very recurrent problem. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[ogondza@gmail.com (JIRA)]

Oliver Gondža updated an issue

Jenkins / JENKINS-29888 Jenkins stuck when concurrent builds of a same job due to log rotator

Change By: Oliver Gondža

Hello, I use Jenkins with parallel builds to checkout many packages. Example:

{noformat}

parallel ( build(build("CheckOut-Package", PACKAGE_NAME: "Package1"); build(build("CheckOut-Package", PACKAGE_NAME: "Package2"); ... build(build("CheckOut-Package", PACKAGE_NAME: "Package10"); )

{noformat}

Each of these jobs calls the log rotator when finished. Some time, next log appears:

{noformat}

SEVERE: Executor threw an exception java.util.NoSuchElementException         at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)         at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)         at java.util.AbstractMap$2$1.next(AbstractMap.java:385)         at hudson.util.RunList.subList(RunList.java:139)         at hudson.tasks.LogRotator.perform(LogRotator.java:125)         at hudson.model.Job.logRotate(Job.java:467)         at hudson.model.Run.execute(Run.java:1808)         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)         at hudson.model.ResourceController.execute(ResourceController.java:98)         at hudson.model.Executor.run(Executor.java:374)

{noformat}

In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked. Since log Rotator should not be a blocking task, I added an exception catch in this PR: https://github.com/jenkinsci/jenkins/pull/1790 That was enough on my side to handle this very recurrent problem.

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator Code changed in jenkins User: tfennelly Path: .gitignore changelog.html cli/pom.xml core/pom.xml core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Run.java core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly core/src/main/resources/lib/form/confirm.js core/src/main/resources/lib/form/select/select.js core/src/main/resources/lib/layout/layout.jelly plugins/pom.xml pom.xml test/pom.xml test/src/test/groovy/hudson/model/AbstractProjectTest.groovy war/pom.xml http://jenkins-ci.org/commit/jenkins/95ca3da67d217c90d31819ec92e521e2072acd5a Log: Merge branch 'master' into plugin-manager-dependants master: Update the changelog by new merges: [FIXED JENKINS-30569] HistoryWidget: fix JS syntax error [FIXED JENKINS-29014] render API link conditional on getApi() presence JENKINS-21720 JS alert preventig to leave a configuration page even without formulary changes add ctags file 'tags' to .gitignore [maven-release-plugin] prepare release jenkins-1.632 [FIXED JENKINS-29888] Handling all exceptions returned by logRotator JENKINS-30742 Fixed possible NPE in AbstractProject.resolveForCLI()

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon resolved as Fixed

Jenkins / JENKINS-29888

Jenkins stuck when concurrent builds of a same job due to log rotator

Change By: SCM/JIRA link daemon Status: Open Resolved Resolution: Fixed

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator

Code changed in jenkins User: Otmane TAZI Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/1a5d57fdda3dba92543b8f486d3dce9cfb9c3811 Log:

[FIXED JENKINS-29888] Handling all exceptions returned by logRotator

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator

Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/876e8e20505b705d07b561b34863938872240369 Log: Merge pull request #1790 from tototoman/master [FIXED JENKINS-29888] - Handling all LogRotator exceptions

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator

Code changed in jenkins User: Oleg Nenashev Path:

changelog.html http://jenkins-ci.org/commit/jenkins/a79e2f144a932fe0a97b524f79133d1ea5d0fef0 Log:

Update the changelog by new merges:

JENKINS-29888 - https://github.com/jenkinsci/jenkins/pull/1790 JENKINS-30742 - https://github.com/jenkinsci/jenkins/pull/1849 JENKINS-29014 - https://github.com/jenkinsci/jenkins/pull/1850 JENKINS-21720 - https://github.com/jenkinsci/jenkins/pull/1854 JENKINS-30569 - https://github.com/jenkinsci/jenkins/pull/1856

Add Comment

[jenkins@thorzten.de (JIRA)]

Thorsten Möllers reopened an issue

Same issue occured two times just today on our Jenkins server. Version: 1.625.3 (LTS) From jenkins log: Jan 15, 2016 1:33:38 PM hudson.model.Executor finish1

SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467)

at hudson.model.Run.execute(Run.java:1805)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98)

at hudson.model.Executor.run(Executor.java:408) Jan 15, 2016 11:27:15 AM hudson.model.Executor finish1

SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467)

at hudson.model.Run.execute(Run.java:1805)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98)

at hudson.model.Executor.run(Executor.java:408)

Jenkins / JENKINS-29888

Jenkins stuck when concurrent builds of a same job due to log rotator

Change By: Thorsten Möllers Resolution: Fixed Status: Resolved Reopened Assignee: Régis Desgroppes

Add Comment

[rdesgroppes@java.net (JIRA)]

Régis Desgroppes assigned an issue to Unassigned

Jenkins / JENKINS-29888 Jenkins stuck when concurrent builds of a same job due to log rotator

Change By: Régis Desgroppes Assignee: Régis Desgroppes

Add Comment

[rdesgroppes@java.net (JIRA)]

Régis Desgroppes updated an issue

Jenkins / JENKINS-29888 Jenkins stuck when concurrent builds of a same job due to log rotator

Change By: Régis Desgroppes Priority: Minor Critical

Add Comment

[tib@hms.se (JIRA)]

Timmy Brolin commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator Happens somewhat frequently for us as well. Very annoying. (Jenkins ver. 1.625.2)

Add Comment

[luc.cary@gmail.com (JIRA)]

Luc Cary commented on JENKINS-29888

Re: Jenkins stuck when concurrent builds of a same job due to log rotator

For a long time, this issue was making our CI cluster less reliable. We were on jenkins 1.625.2, and our build flow jobs would often hang when running in parallel. This was a bad user experience that made our CI pipelines flaky, since it either caused timeouts or required that we abort the build. This was fixed for my team by upgrading to the latest stable version of jenkins core (1.651.1). I believe any version of jenkins before https://github.com/jenkinsci/jenkins/pull/1790/files was merged will have this bug, but upgrading to a newer jenkins version (any version after 1.633) should make the exception non-fatal and thus address the issue.

Add Comment