[JIRA] (JENKINS-59907) sh steps stuck indefinitely after upgrading durable-task to v1.31

[jakub.loucky@gmail.com (JIRA)]

Jakub L created an issue

Jenkins / JENKINS-59907 sh steps stuck indefinitely after upgrading durable-task to v1.31 Issue Type: Bug Assignee: Unassigned Components: durable-task-plugin Created: 2019-10-23 19:18 Environment: Jenkins ver. 2.201 (yum installed, master node only) durable-task plugin v1.31 os.arch: s390x os.name: Linux (RedHat) os.version: 3.10.0-327.el7.s390x Priority: Major Reporter: Jakub L After upgrading to v1.31, the first sh step in a pipeline gets stuck. After few minutes Console Output shows: [Pipeline] sh (Get email of the author of last commit) process apparently never started in /data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) Cannot contact : java.io.FileNotFoundException: File '/data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6/output.txt' does not exist   Eventually, I discovered that a new binary was added in the latest version of this plugin. The script compile-binaries.sh in GitHub suggests that the binary is only built for Linux and MacOS.   Sure enough, when I try to execute the binary myself on an architecture other than amd64, I get: -bash: /data/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64: cannot execute binary file   Are other architectures or operating systems (Windows) not supported anymore? Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[zerkms@zerkms.ru (JIRA)]

Ivan Kurnosov commented on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31 I confirm it affects me as well: Jenkins ver. 2.190.1, docker, kubernetes and other plugins: latest stable version. Jenkins runs on linux (inside a docker container)

Add Comment

[vahid@exten.io (JIRA)]

Vahid Tyau commented on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31

I was also hit by this. Rolling back the durable task plugin to 1.30 got us back up and running.

Add Comment

[vahid@exten.io (JIRA)]

Vahid Tyau edited a comment on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31

I was also hit by this. Rolling back the durable task plugin to 1.30 got us back up and running.

-- edit: OS: Centos 7.6 and 7.7 64bit hosts Architecture: x86_64  Jenkins 2.190.1 docker   Issue started after upgrading the durable task plugin today to 1.31.

Add Comment

[r.fuereder@xortex.com (JIRA)]

Reinhold Füreder commented on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31

I think this issue's title is just one of the symptoms of the new "Durable Task" plugin v1.31 bug: In on of our pipelines we do not use Docker at all and just exeucte an "sh" step (on Jenkins master => psst!) via ssh on another server and it fails right away: Pipeline code: ... boolean skipRenewal = true script.xortex.skipableStage("check need for renewal") { def exitCode = script.sh(returnStatus: true, script: makeRenewLocalCertShellCmd(config.domainName)) // ! This is line 54 where the exception happens ... private static String makeRenewLocalCertShellCmd(String domainName) { return "ssh -t -t MyServer.ACME.com './renewlocalcert ${domainName}'" } Build log (proprietary) with nice exception stacktrace at the end: ... [Pipeline] // stage [Pipeline] stage [Pipeline] { (check need for renewal) [Pipeline] sh [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] ansiColor [Pipeline] { [Pipeline] echo 06:37:13 Pipeline problem: Build failed (check need for renewal) due to: "java.io.IOException: Cannot run program "/var/lib/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64" (in directory "/var/lib/jenkins/workspace/Sandbox/ACME.renewCerts"): error=13, Permission denied" => Please check the "Console/Log Output" => Failure notification will be sent... ... java.io.IOException: error=13, Permission denied at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) at hudson.Proc$LocalProc.<init>(Proc.java:250) at hudson.Proc$LocalProc.<init>(Proc.java:219) at hudson.Launcher$LocalLauncher.launch(Launcher.java:937) at hudson.Launcher$ProcStarter.start(Launcher.java:455) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:230) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:317) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) Caused: java.io.IOException: Cannot run program "/var/lib/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64" (in directory "/var/lib/jenkins/workspace/Sandbox/ACME.renewCerts"): error=13, Permission denied at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at hudson.Proc$LocalProc.<init>(Proc.java:250) at hudson.Proc$LocalProc.<init>(Proc.java:219) at hudson.Launcher$LocalLauncher.launch(Launcher.java:937) at hudson.Launcher$ProcStarter.start(Launcher.java:455) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:230) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:317) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at com.ACME.renewcerts.RenewCertsBuild.build(RenewCertsBuild.groovy:54) at com.ACME.stage.SkipableStage.execute(SkipableStage.groovy:59) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:84) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45) at sun.reflect.GeneratedMethodAccessor284.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:87) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Add Comment

[r.fuereder@xortex.com (JIRA)]

Reinhold Füreder edited a comment on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31

I think this issue's title is just one of the symptoms of the new "Durable Task" plugin v1.31 bug:

* Latest version of Jenkins core (v2.201) running on Ubuntu 16.04 * In on of our pipelines we do not use Docker at all and just exeucte an "sh" step (on Jenkins master :-( => psst!) via ssh on another server and it fails right away: * Pipeline code: {code}

... boolean skipRenewal = true script.xortex.skipableStage("check need for renewal") { def exitCode = script.sh(returnStatus: true, script: makeRenewLocalCertShellCmd(config.domainName)) // ! This is line 54 where the exception happens ... private static String makeRenewLocalCertShellCmd(String domainName) {   return "ssh -t -t MyServer.ACME.com './renewlocalcert ${domainName}'" }

{code} * Build log (proprietary) with nice exception stacktrace at the end: {code}

{code}

Add Comment

[jakub.loucky@gmail.com (JIRA)]

Jakub L commented on JENKINS-59907

Re: sh steps stuck indefinitely after upgrading durable-task to v1.31

This is NOT related to the other two opened issues with v1.31. Those two are likely caused by the new binary durable_task_monitor not being available from within Docker containers. But this issue is caused by the binary not being executable on my HW architecture. I'll clarify the title.

Add Comment