[JIRA] (JENKINS-55356) Some workflow jobs fail after restart on Java 11 server

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite created an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server Issue Type: Bug Assignee: Unassigned Components: workflow-job-plugin Created: 2018-12-28 21:20 Environment: Jenkins JDK 11 docker image and current plugins Multibranch Pipeline builds for git plugin, git client plugin, and platformlabeler plugin Priority: Minor Reporter: Mark Waite While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a jenkins-url/safeRestart will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs. Build log output from the failed builds has included messages like (seems to be windows specific): 07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’ Ready to run at Fri Dec 28 07:48:49 MST 2018 07:48:49 Timeout set to expire in 17 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 07:48:50 Failed in branch windows-8-2.150.1 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh and this (seems to fail on Windows and on Linux): 08:48:33 [INFO] --------------------------------[ hpi ]--------------------------------- Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ??? ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ??? Ready to run at Fri Dec 28 08:52:07 MST 2018 08:52:07 Timeout set to expire in 54 min 08:52:07 Timeout set to expire in 54 min 08:52:07 Timeout set to expire in 54 min 08:52:07 Timeout set to expire in 54 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] // timeout [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] // node [Pipeline] // timeout [Pipeline] } 08:52:08 Failed in branch windows-8 [Pipeline] } [Pipeline] // node [Pipeline] } 08:52:08 Failed in branch windows-8-2.150.1 08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572 [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } 08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955 [Pipeline] // node [Pipeline] } 08:57:14 Failed in branch linux-8 [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 08:57:14 Failed in branch linux-8-2.150.1 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh and this (windows and linux): 09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client --- Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Ready to run at Fri Dec 28 09:13:51 MST 2018 09:13:51 Timeout set to expire in 55 min 09:13:51 Timeout set to expire in 55 min 09:13:51 Timeout set to expire in 55 min 09:13:51 Timeout set to expire in 55 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // timeout [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // timeout [Pipeline] // node [Pipeline] // timeout [Pipeline] } 09:13:52 Failed in branch windows-8-2.150.1 [Pipeline] } [Pipeline] } [Pipeline] // node [Pipeline] // node [Pipeline] } 09:13:52 Failed in branch linux-8 [Pipeline] } 09:13:52 Failed in branch windows-8 09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 09:18:58 Failed in branch linux-8-2.150.1 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh The process never started message and the missing workspace message are visible in both the failed git client plugin builds and in the failed git plugin builds. I don't know if the problem can be duplicated on a Java 8 environment. Will try that soon Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server

Change By: Mark Waite Labels: java11-compatibility

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server Change By: Mark Waite

While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a

{code} jenkins-url/safeRestart {code}

will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs. Build log output from the failed builds has included messages like (seems to be windows specific):

{noformat}

07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’ Ready to run at Fri Dec 28 07:48:49 MST 2018 07:48:49 Timeout set to expire in 17 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 07:48:50 Failed in branch windows-8-2.150.1 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh

{noformat}

and this (seems to fail on Windows and on Linux):

{noformat}

{noformat}

and this (windows and linux):

{noformat}

{noformat}

The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

I don't know if the The problem can be duplicated does not seem to repeat on a Java 8 environment , just on a Java 11 environment .   Will try that soon   More experiments to come.

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server Mark Waite could you please also check the Jenkins logs to see if there's more there? Specifically I'm curious if this could actually be some kind of duplicate of JENKINS-55174. Also: could you test again with the fix provided (not yet merged) by Devin Nusbaum in https://github.com/jenkinsci/workflow-support-plugin/pull/86? You can the Incremental release of this PR available at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/3.0-java11-alpha-2-rc689.90369f68d9e9/workflow-support-3.0-java11-alpha-2-rc689.90369f68d9e9.hpi Thank you!

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Baptiste Mathus thanks for asking! The details in this report are using the incremental build from the pull request that Devin Nusbaum had provided. The results with that pull request are much better than without it. Without that pull request I see null pointer exception messages in the Jenkins logs on restart. The null pointer exceptions cause the jobs to fail and are described in JENKINS-55338. JENKINS-55338 is a duplicate of JENKINS-55174. There are no additional entries in the Jenkins log related to the problem as far as I can tell. I think PR-86 should be merged into the workflow support plugin for Java 11. It improves the results by removing the null pointer exception and moves the code onto a newer JBoss marshalling release.

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

It looks like the root failure in all cases is related to the "ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh" error message. It think that error message comes from workflow-durable-task-step, and happens if the workspace for a job is not a directory. Maybe this is a bug with the workspace-related changes in recent versions of branch-api? Mark Waite Can you add the full list of plugins installed in Jenkins and their versions to the description? It would also be great to see the Pipeline jobs that these log snippets come from.

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

@vivek can you please plan analyzing this issue? Thanks!

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus edited a comment on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

@ [~ vivek ] can you please plan analyzing this issue? Thanks!

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

Change By: Mark Waite

ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh

ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh

The problem does * not * seem to repeat on a Java 8 environment, just on a Java 11 environment.    The problem does *not* seem to repeat on a Java 11 environment running on a larger computer.  The failing computer has 8 GB RAM with an older Intel i5 processor, while the passing computer has 32 GB RAM and a newer Intel i5 processor.   More experiments research.  The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case.  However, I haven't yet been able to come duplicate the problem on any other machine .   More investigation soon.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server Devin Nusbaum the link to the list of plugins is a list of plugin files committed to the git repository at the time I ran the test. The Pipeline job definitions are Git client plugin Git plugin Both those Jenkinsfile examples are buildPlugin with a few options. The second example includes a long definition of some PCT / ATH code that is not executed.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Baptiste Mathus yes, Devin Nusbaum is already looking in to it.

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Great thanks!

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus started work on JENKINS-55356

Change By: Baptiste Mathus Status: Open In Progress

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus assigned an issue to Devin Nusbaum

Jenkins / JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

Change By: Baptiste Mathus Assignee: Devin Nusbaum

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server

Change By: Mark Waite

The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.   The problem does * not* seem to repeat less frequently on a Java 11 environment running on a larger computer.  The failing computer has 8 GB RAM with an older Intel i5 processor, while the passing less frequently failing computer has 32 GB RAM and a newer Intel i5 processor.   The 32 GB machine has shown the failure only once.  That failure was during a restart while the agents and the server were very busy. More research.   The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case.  However, I haven't yet been able to duplicate the problem on any other machine.  More investigation soon.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server When the 32 GB i5 machine failed, the message in the failing build log was similar to other Windows "workspace not found" messages: 14:31:36 [INFO] ------------------------------------------------------------------------ Resuming build at Wed Jan 02 14:33:19 MST 2019 after Jenkins restart Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Ready to run at Wed Jan 02 14:33:53 MST 2019 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // timeout [Pipeline] // withEnv [Pipeline] // stage [Pipeline] stage [Pipeline] { (Archive (linux-8-2.150.1)) [Pipeline] } [Pipeline] } [Pipeline] junit [Pipeline] // node 14:33:54 Recording test results [Pipeline] // stage [Pipeline] } 14:33:54 Failed in branch linux-8 [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:33:55 Failed in branch windows-8-2.150.1 14:33:55 No test report files were found. Configuration error? [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:33:55 Failed in branch linux-8-2.150.1 Resuming build at Wed Jan 02 14:44:08 MST 2019 after Jenkins restart Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: Waiting for next available executor on ‘mark-pc4-ssh’ Ready to run at Wed Jan 02 14:44:29 MST 2019 14:44:29 Timeout set to expire in 35 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:44:29 Failed in branch windows-8 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline GitHub has been notified of this commit’s build result Command Close created at at hudson.remoting.Command.<init>(Command.java:68) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1267) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265) at hudson.remoting.Channel.close(Channel.java:1438) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1272) Caused: hudson.remoting.Channel$OrderlyShutdown Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to aws-ubuntu-18-a at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.Channel.call(Channel.java:956) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:144) at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc3-ssh at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to testing-a-mwaite at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:50) at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:23) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) hudson.AbortException: No test report files were found. Configuration error? at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:154) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc4-ssh at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1040) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1273) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85) Finished: FAILURE

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Failures seem to be more frequent when the master is heavily loaded.  I've seen multiple failures on my 32 GB machine now.  Will continue exploring to see if I can identify common failure patterns.

When the 32 GB i5 machine failed, the message in the failing build log was similar to other Windows "workspace not found" messages:

{noformat}

{noformat}

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

It seems that workflow-support 3.0 and 3.1 did not address this. I still see the following report in the Windows agent on my Java 11 server when I safeRestart the Jenkins server during Pipeline execution: [INFO] ------------------------------------------------------- [INFO] Running InjectedTest [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.181 s - in InjectedTest [INFO] Running hudson.plugins.git.BranchTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.374 s - in hudson.plugins.git.BranchTest [INFO] Running hudson.plugins.git.GitAPIBadInitTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in hudson.plugins.git.GitAPIBadInitTest [INFO] Running hudson.plugins.git.GitExceptionTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.534 s - in hudson.plugins.git.GitExceptionTest [INFO] Running hudson.plugins.git.GitLockFailedExceptionTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in hudson.plugins.git.GitLockFailedExceptionTest [INFO] Running hudson.plugins.git.GitObjectTest [INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 s - in hudson.plugins.git.GitObjectTest [INFO] Running hudson.plugins.git.GitToolResolverTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.028 s - in hudson.plugins.git.GitToolResolverTest [INFO] Running hudson.plugins.git.GitToolTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.005 s - in hudson.plugins.git.GitToolTest [INFO] Running hudson.plugins.git.IndexEntryTest [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in hudson.plugins.git.IndexEntryTest [INFO] Running hudson.plugins.git.RevisionTest [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in hudson.plugins.git.RevisionTest [INFO] Running hudson.plugins.git.TagTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in hudson.plugins.git.TagTest [INFO] Running org.jenkinsci.plugins.gitclient.CliGitAPIImplTest missing workspace C:\J\T\workspace\it-client-pipeline-github_master on mark-pc3-ssh

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Yes, there are no changes in any recent Pipeline plugin release I am aware of that would explain or fix this issue. I investigated a few weeks ago but was not able to create a reproduction case. The error message comes from workflow-durable-task-step. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes).

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

That directory exists and looks like a typical workspace. it does contain a directory named .@tmp which was a surprise for me. That directory contains 3 files that with names that start with 'jenkins-gitclient-permission'. I assume some form of temporary file, but don't recognize them as anything specific.

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus updated an issue

Jenkins / JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

Change By: Baptiste Mathus Labels: java11-compatibility scrub triaged

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server

Change By: Mark Waite

While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a {code} jenkins-url/safeRestart {code} will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

Build log output from the failed builds has included messages like (seems to be more common in windows specific , but visible in Linux agents as well ):

{noformat} 07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’ Ready to run at Fri Dec 28 07:48:49 MST 2018

07:48:49 Timeout set to expire in 17 min

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] }

07:48:50 Failed in branch windows-8-2.150.1

[Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline

ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh {noformat} and this (seems to fail on Windows and on Linux): {noformat} 08:48:33 [INFO] --------------------------------[ hpi ]--------------------------------- Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???

???

Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ??? Ready to run at Fri Dec 28 08:52:07 MST 2018 08:52:07 Timeout set to expire in 54 min 08:52:07 Timeout set to expire in 54 min 08:52:07 Timeout set to expire in 54 min

08:52:07 Timeout set to expire in 54 min

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] }

[Pipeline] // stage

[Pipeline] // withEnv [Pipeline] } [Pipeline] }

[Pipeline] // timeout [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] // node [Pipeline] // timeout [Pipeline] } 08:52:08 Failed in branch windows-8 [Pipeline] }

[Pipeline] // node [Pipeline] } 08:52:08 Failed in branch windows-8-2.150.1 08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] }

08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955 [Pipeline] // node [Pipeline] }

08:57:14 Failed in branch linux-8 [Pipeline] }

[Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] }

08:57:14 Failed in branch linux-8-2.150.1

[Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline

ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh {noformat} and this (windows and linux): {noformat} 09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client --- Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ??? Ready to run at Fri Dec 28 09:13:51 MST 2018 09:13:51 Timeout set to expire in 55 min 09:13:51 Timeout set to expire in 55 min 09:13:51 Timeout set to expire in 55 min

09:13:51 Timeout set to expire in 55 min

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] }

[Pipeline] } [Pipeline] // stage [Pipeline] // withEnv

[Pipeline] // withEnv [Pipeline] } [Pipeline] }

[Pipeline] } [Pipeline] // timeout [Pipeline] // stage

[Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // timeout

[Pipeline] // node [Pipeline] // timeout [Pipeline] } 09:13:52 Failed in branch windows-8-2.150.1 [Pipeline] }

[Pipeline] } [Pipeline] // node [Pipeline] // node [Pipeline] } 09:13:52 Failed in branch linux-8 [Pipeline] }

09:13:52 Failed in branch windows-8 09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] }

09:18:58 Failed in branch linux-8-2.150.1

[Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline

ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh {noformat} The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds. The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.

The problem does *seem to repeat less frequently on a Java 11 environment running on a larger computer.  The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor.  The 32 GB machine has shown the failure only once.  That failure was during a restart while the agents and the server were very busy.

The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case.  However, I haven't yet been able to duplicate the problem on any other machine.  More investigation soon.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server Change By: Mark Waite

While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a {code} jenkins-url/safeRestart {code} will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

The problem does * seem to repeat * less frequently * on a Java 11 environment running on a larger computer.  The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor.  The 32 GB machine has shown the failure only once multiple times as well as the smaller computer .  That failure was during a restart while the agents and the server were very busy.

The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case.  However, I haven't yet been able to duplicate the problem on any other machine.  More investigation soon.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

The problem does seem to repeat *less frequently* on a Java 11 environment running on a larger computer.  The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor.  The 32 GB machine has shown the failure multiple times as well as the smaller computer.  That failure was during a restart while the agents and the server were very busy. The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case.   However, I haven ' t yet been able to duplicate ve dupilicated the problem failures on any other machine at least two different machines .   More investigation soon.

Add Comment

[batmat@batmat.net (JIRA)]

Baptiste Mathus commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server > repeat less frequently on a Java 11 environment running on a larger computer Triggered a thought Mark Waite: would you be able to run that test inside a Docker container with memory constraints? I can help with that if that's something you've never played with before. But basically passing like -m 1G will limit the memory available inside the container to 1GB of RAM, which is possibly going to trigger that behavior more immediately, if that is actually related to some memory settings . Cheers!

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

The error message comes from workflow-durable-task-step. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes). Another thing to point out is that FilePath#isDirectory will return false in some failure cases (see the Javadoc for File#isDirectory). Perhaps we should update this line to use NIO methods so it throws exceptions instead of returning false in some cases (another case of JENKINS-47324).

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum edited a comment on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

{quote} The error message comes from [workflow-durable-task-step|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/20dc6a9bc70e8b6ec598d4836b5c20cf57abd8ea/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L345]. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes). {quote} Another thing to point out is that {{FilePath#isDirectory}} will return false in some failure cases (see the [ Javadoc for {{File#isDirectory}} |https://docs.oracle.com/javase/7/docs/api/java/io/File.html#isDirectory( ) ]) . Perhaps we should update [this line|https://github.com/jenkinsci/jenkins/blob/226f7b4c2bedb14b70f12da90db15574e18364d0/core/src/main/java/hudson/FilePath.java#L1653] to use NIO methods so it throws exceptions instead of returning false in some cases (another case of JENKINS-47324).

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum edited a comment on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

{quote} The error message comes from [workflow-durable-task-step|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/20dc6a9bc70e8b6ec598d4836b5c20cf57abd8ea/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L345]. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes). {quote}

Another thing to point out is that {{FilePath#isDirectory}} will return false in some failure cases (see the [Javadoc for {{File#isDirectory}}|https://docs.oracle.com/javase/7/docs/api/java/io/File.html#isDirectory()]). Perhaps we should update [this line|https://github.com/jenkinsci/jenkins/blob/226f7b4c2bedb14b70f12da90db15574e18364d0/core/src/main/java/hudson/FilePath.java#L1653] to use NIO methods so it throws exceptions instead of returning false in some cases (another case of JENKINS-47324).   Edit: I filed [https://github.com/jenkinsci/jenkins/pull/3864] for that issue.

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Double edit: Also to clarify, if {{FilePath#isDirectory}}  _had_ thrown an exception, then [this code|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/20dc6a9bc70e8b6ec598d4836b5c20cf57abd8ea/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L341] would have been called, which would have caused Pipeline to attempt to connect again rather than aborting the build immediately. That issue wouldn't really explain why we are seeing this on Java 11 and not Java 8, but seems worth investigating.

Add Comment

[me@basilcrow.com (JIRA)]

Basil Crow commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

I experienced this failure mode twice on January 16, two weeks after upgrading Jenkins from 2.138.1 LTS (with workflow-job 2.25, workflow-cps 2.54, and workflow-durable-task-step 2.21) to 2.150.1 LTS (with workflow-job 2.31, workflow-cps 2.61, and workflow-durable-task-step 2.27). I am not running Java 11. The job has been running daily and has only failed twice with this failure mode, so the error is transient. ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod on scale-dc2 ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod@2 on dc3 When this error occurred on the 16th, I logged into these machines and checked the given directories on the command line. In both cases the directories existed. So I suspect there may have been some transient I/O error at the time.

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev updated an issue

Jenkins / JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

Change By: Oleg Nenashev Labels: java11-compatibility scrub triaged

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server Devin Nusbaumwhat is the status here? We are about to proceed with Java 11 GA in Jenkins. I do not think it is a blocker, but it would be nice to get your feedback

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-55356

Re: Some workflow jobs fail after restart on Java 11 server

Oleg Nenashev Unchanged from my perspective. Given that Basil Crow mentioned that they have seen the issue on Java 8, it seems like this is not something specific to Java 11. I closed the PR I mentioned in this comment because I was not able to perform the testing necessary to feel confident about the change, and it wasn't clear to me what kinds of failures would have been exposed by switching to NIO. It would probably be safe to reopen it and change but change the behavior to return false if the directory does not exist to be much closer to the original behavior, but I'm not sure what the benefit would be. If anyone is able to come up with a self-contained and consistent reproduction case, then I would be more than happy to take a look, but without any other ideas I am just grasping at straws for now.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite stopped work on JENKINS-55356

Change By: Mark Waite Status: In Progress Open

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev updated JENKINS-55356

Platform SIG meeting: As far as Mark Waite concerned, it is resolved.

Jenkins / JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

Change By: Oleg Nenashev Status: Open Fixed but Unreleased Resolution: Fixed

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev updated JENKINS-55356

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server

Change By: Oleg Nenashev Status: Fixed but Unreleased Resolved

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite closed an issue as Fixed

Jenkins / JENKINS-55356 Some workflow jobs fail after restart on Java 11 server

Change By: Mark Waite Status: Resolved Closed

Add Comment