[JIRA] (JENKINS-60286) durable-task v1.33 fails with agent Dockerfile

40 views
Skip to first unread message

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 9:18:02 AM11/26/19
to jenkinsc...@googlegroups.com
Marky Jackson created an issue
 
Jenkins / Bug JENKINS-60286
durable-task v1.33 fails with agent Dockerfile
Issue Type: Bug Bug
Assignee: Unassigned
Components: durable-task-plugin
Created: 2019-11-26 14:17
Environment: Jenkins version 2.190.3
Priority: Blocker Blocker
Reporter: Marky Jackson

Error I get
process apparently never started in /home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-f026a6cb
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
Cannot contact EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5): java.io.FileNotFoundException: File '/home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-f026a6cb/output.txt' does not existHard kill!
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 9:27:03 AM11/26/19
to jenkinsc...@googlegroups.com
Marky Jackson commented on Bug JENKINS-60286
 
Re: durable-task v1.33 fails with agent Dockerfile

I eventually have to hard kill the job.

I tried all applicable workarounds including all noted in previous tickets as workaround to no avail.

 

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 10:09:03 AM11/26/19
to jenkinsc...@googlegroups.com

Here are some questions to help understand what is happening:

  • Was this working for you previously? If so, what plugin versions have been installed/upgraded since then?
  • Does it happen consistently or intermittently?
  • Can you turn on FINE logging for org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep and then post the full stack trace of the exception from the Jenkins system log (the message should contain "is evidently offline now")?
  • Did you try adding -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true as indicated to your Jenkins Java startup settings to see if you get more detailed error information?

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 10:21:04 AM11/26/19
to jenkinsc...@googlegroups.com

Here is the stack trace from the logs:

Nov 26, 2019 3:17:48 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings@3 Nov 26, 2019 3:17:48 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcould not check /home/ec2-user/workspace/ntend-pipeline_customer-settings@3 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:300) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:293) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.lambda$check$3(DurableTaskStep.java:604) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:651) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:604) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:548) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.io.FileNotFoundException: File '/home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-a112d9e1/output.txt' does not exist at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:302) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:300) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 10:22:02 AM11/26/19
to jenkinsc...@googlegroups.com
  • Was this working for you previously? If so, what plugin versions have been installed/upgraded since then?
    • Yes it was on version 1.29 for durable task and core code Jenkins version 2.190.2
  • Does it happen consistently or intermittently?
    • It happens consistently for jobs that have the agent set to use Dockerfiles but no other jobs types
  • Did you try adding -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true as indicated to your Jenkins Java startup settings to see if you get more detailed error information?
    • I will add that now

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 10:39:06 AM11/26/19
to jenkinsc...@googlegroups.com
JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[16:sh]:Owner[monitor-frontend-pipeline/customer-settings/34:monitor-frontend-pipeline/customer-settings #34] Nov 26, 2019 3:32:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) using RemoteLauncher[hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5)] Nov 26, 2019 3:32:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:32:04 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:32:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:04 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:05 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:05 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcalling close with nl=true Nov 26, 2019 3:32:06 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[20:sh]:Owner[monitor-frontend-pipeline/customer-settings/34:monitor-frontend-pipeline/customer-settings #34] Nov 26, 2019 3:32:06 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) using RemoteLauncher[hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5)] Nov 26, 2019 3:32:06 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:32:07 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:07 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:32:07 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcalling close with nl=true Nov 26, 2019 3:32:08 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[25:sh]:Owner[monitor-frontend-pipeline/customer-settings/34:monitor-frontend-pipeline/customer-settings #34] Nov 26, 2019 3:32:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) using org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@5148818d; decorates RemoteLauncher[hudson.remoting.Channel@13b6e486:EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5)] Nov 26, 2019 3:32:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:32:08 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:32:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:08 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:09 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:09 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:10 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:10 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:10 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:10 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:11 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:11 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:12 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:12 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:13 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:13 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:16 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:16 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:18 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:18 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:20 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:20 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:23 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:23 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:26 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:26 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:30 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:30 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:34 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:34 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:40 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:40 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:46 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:46 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:32:54 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:32:54 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:33:04 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:33:04 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:33:15 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:33:15 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:33:29 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:33:29 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:33:44 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:33:44 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:33:59 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:33:59 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:34:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:34:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:34:29 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:34:29 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:34:44 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:34:44 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:34:59 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:34:59 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:35:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:35:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:35:29 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:35:29 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:35:44 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:35:44 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:35:59 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:35:59 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:36:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:36:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:36:29 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:36:29 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:36:44 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:36:44 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:36:59 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:36:59 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/ntend-pipeline_customer-settings on EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) Nov 26, 2019 3:37:02 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[12:sh]:Owner[QA-agent-smoke/515:QA-agent-smoke #515] Nov 26, 2019 3:37:02 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) using RemoteLauncher[hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113)] Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:37:03 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcalling close with nl=true Nov 26, 2019 3:37:03 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[16:sh]:Owner[QA-agent-smoke/515:QA-agent-smoke #515] Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) using RemoteLauncher[hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113)] Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:37:03 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcalling close with nl=true Nov 26, 2019 3:37:03 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[18:sh]:Owner[QA-agent-smoke/515:QA-agent-smoke #515] Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) using RemoteLauncher[hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113)] Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:37:03 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:37:03 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcalling close with nl=true Nov 26, 2019 3:37:08 PM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepJENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[32:sh]:Owner[QA-agent-smoke/515:QA-agent-smoke #515] Nov 26, 2019 3:37:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunching task against hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) using org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@703fae2b; decorates RemoteLauncher[hudson.remoting.Channel@63cb1c8:EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113)] Nov 26, 2019 3:37:08 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskSteplaunched task Nov 26, 2019 3:37:09 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcreating filtered stream Nov 26, 2019 3:37:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:09 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:09 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:09 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:10 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:10 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:10 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:10 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:11 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:11 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:12 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:12 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:13 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:13 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) seems to be online so using /home/ec2-user/workspace/QA-agent-smoke Nov 26, 2019 3:37:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepstill running in /home/ec2-user/workspace/QA-agent-smoke on EC2 (ec2) - builder-backend-j8 (i-0327f0750aa305113) Nov 26, 2019 3:37:14 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepEC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings Nov 26, 2019 3:37:14 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStepcould not check /home/ec2-user/workspace/ntend-pipeline_customer-settings Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:300) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:293) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.lambda$check$3(DurableTaskStep.java:604) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:651) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:604) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:548) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.io.FileNotFoundException: File '/home/ec2-user/workspace/ntend-pipeline_customer-settings@tmp/durable-40a66013/output.txt' does not exist at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:302) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:300) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 10:40:03 AM11/26/19
to jenkinsc...@googlegroups.com

The 2nd above stack trace is with  -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true turned on and doesn't show anything new

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 10:49:03 AM11/26/19
to jenkinsc...@googlegroups.com

Marky Jackson LAUNCH_DIAGNOSTICS prints information to the build log, not the Jenkins system log. Can you post the build log from a failing build after you enabled that property?

jglick@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 10:49:04 AM11/26/19
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
 

Possibly a regression from the new native wrapper.

I would not recommend using agent docker (or agent dockerfile). These ultimately rely on the withDockerContainer step and rather complex injection of docker exec into commands, which can easily break when there are issues with volume mounts and so on. Much safer and more transparent to use a plain label and run CLI docker commands (build, run) from sh.
Change By: Jesse Glick
Component/s: docker-workflow-plugin

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 10:51:03 AM11/26/19
to jenkinsc...@googlegroups.com

Possibly a regression from the new native wrapper.

The new wrapper is disabled by default due to various problems, so if there is a regression, I suspect that something subtle was changed in the fallback code.

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 10:52:02 AM11/26/19
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
 
Change By: Devin Nusbaum
Error I get
process apparently never started in /home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-f026a6cb
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
Cannot contact EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5): java.io.FileNotFoundException: File '/home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-f026a6cb/output.txt' does not existHard kill!


Stack trace:

{noformat}

Nov 26, 2019 3:17:48 PM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5) seems to be online so using /home/ec2-user/workspace/ntend-pipeline_customer-settings@3
Nov 26, 2019 3:17:48 PM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
could not check /home/ec2-user/workspace/ntend-pipeline_customer-settings@3
    Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (ec2) - new-builder-backend-j8 (i-0d91350838c6340f5)
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
        at hudson.remoting.Channel.call(Channel.java:957)
        at hudson.FilePath.act(FilePath.java:1072)
        at hudson.FilePath.act(FilePath.java:1061)
        at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:300)
        at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.getOutput(FileMonitoringTask.java:293)
        at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.lambda$check$3(DurableTaskStep.java:604)
        at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:651)
        at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:604)
        at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:548)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.io.FileNotFoundException: File '/home/ec2-user/workspace/ntend-pipeline_customer-settings@3@tmp/durable-a112d9e1/output.txt' does not exist

    at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
    at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:302)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1.invoke(FileMonitoringTask.java:300)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
    at hudson.remoting.UserRequest.perform(UserRequest.java:212)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:369)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

{noformat}

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 11:05:04 AM11/26/19
to jenkinsc...@googlegroups.com
Marky Jackson commented on Bug JENKINS-60286
 
Re: durable-task v1.33 fails with agent Dockerfile

Devin Nusbaum here is the build log from the console. I eventually have to hard kill the job because it just hangs at the missing file:

Started by user mark.j...@sysdig.com 15:51:39 Connecting to https://api.github.com using xxxxxxx/****** (zzzzzzzzz) Obtained Jenkinsfile from xxxxxxx Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on EC2 (ec2) - ${WORKER_NAME} (ec2ID) in /home/${USER}/workspace/ntend-pipeline_customer-settings [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout using credential ${TOKEN} Cloning the remote Git repository Cloning repository ${REPO} > git init /home/${USER}/workspace/ntend-pipeline_customer-settings # timeout=10 } Fetching upstream changes from ${REPO} > git --version # timeout=10 } using GIT_SSH to set credentials ${TOKEN} > git fetch --tags --progress -- ${REPO} +refs/heads/*:refs/remotes/origin/* # timeout=10 } Checking out Revision ${REVISION} (customer-settings) Commit message: "Reverting my changes" > git config remote.origin.url ${REPO} # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url ${REPO} # timeout=10 Fetching upstream changes from ${REPO} using GIT_SSH to set credentials ${TOKEN} > git fetch --tags --progress --${REPO} +refs/heads/customer-settings:refs/remotes/origin/customer-settings # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f ${REVISION} # timeout=10 > git rev-list --no-walk ${REVISION} # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Agent Setup) [Pipeline] isUnix [Pipeline] readFile [Pipeline] sh + docker build -t c7c9f201c0dbb37091ac75c689812beae3bf54b4 -f Dockerfile . WARNING: Error loading config file: /home/${USER}/.docker/config.json: EOF Sending build context to Docker daemon 116.3MBStep 1/13 : FROM ubuntu:latest ---> 775349758637 Step 2/13 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> 08bb415b8561 Step 3/13 : RUN apt-get update -y && apt-get install apt-transport-https ca-certificates curl gnupg git python-pip python2.7 python2.7-dev groff-base build-essential parallel jq grep awscli -y --no-install-recommends ---> Using cache ---> 2d2f267ee29f Step 4/13 : RUN groupadd ${USER} -g 500 ---> Using cache ---> c1508155cdc1 Step 5/13 : RUN useradd -m -d /home/${USER} -g ${USER} -u 500 ${USER} ---> Using cache ---> 12ad008b1a8a Step 6/13 : RUN curl -sSL https://dl.google.com/linux/linux_signing_key.pub | apt-key add - && echo "deb [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list ---> Using cache ---> 1e4d342560b2 Step 7/13 : RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - ---> Using cache ---> 7ae11025dc8e Step 8/13 : RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list ---> Using cache ---> 6f40d90e286a Step 9/13 : ENV YARN_VERSION=1.17.3-1 ---> Using cache ---> 818b40b458e3 Step 10/13 : RUN apt-get update && apt-get install -y yarn=$YARN_VERSION --no-install-recommends ---> Using cache ---> 88de9a37077f Step 11/13 : RUN apt-get update && apt-get install -y google-chrome-stable ---> Using cache ---> f57b038effc4 Step 12/13 : ENV NODE_VERSION=10.16.3 ---> Using cache ---> 4c946687c3a9 Step 13/13 : RUN curl https://deb.nodesource.com/node_10.x/pool/main/n/nodejs/nodejs_$NODE_VERSION-1nodesource1_amd64.deb > node.deb && dpkg -i node.deb && rm node.deb ---> Using cache ---> 412afce72a56 Successfully built 412afce72a56 Successfully tagged c7c9f201c0dbb37091ac75c689812beae3bf54b4:latest [Pipeline] } [Pipeline] // stage [Pipeline] isUnix [Pipeline] sh + docker inspect -f . c7c9f201c0dbb37091ac75c689812beae3bf54b4 WARNING: Error loading config file: /home/${USER}/.docker/config.json: EOF . [Pipeline] withDockerContainer EC2 (ec2) - ${LABEL} ($EC2-ID) does not seem to be running inside a container $ docker run -t -d -u 500:500 -w /home/${USER}/workspace/ntend-pipeline_customer-settings -v /home/$USER}/workspace/ntend-pipeline_customer-settings:/home/${USER}/workspace/ntend-pipeline_customer-settings:rw,z -v /home/${USER}/workspace/ntend-pipeline_customer-settings@tmp:/home/${USER}/workspace/ntend-pipeline_customer-settings@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** c7c9f201c0dbb37091ac75c689812beae3bf54b4 cat $ docker top 96ac75a6873563bc96ddef48328caaa3a641987e2d9d7ce10b86db07c2ee6876 -eo pid,comm [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $NPM_AUTH_TOKEN or $GITHUB_ACCESS_TOKEN [Pipeline] { [Pipeline] sh process apparently never started in /home/${USER}/workspace/ntend-pipeline_customer-settings@tmp/durable-d4083a8e (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) Cannot contact EC2 (ec2) - ${LABEL} (${EC2-ID}): java.io.FileNotFoundException: File '/home/${USER}/workspace/ntend-pipeline_customer-settings@tmp/durable-d4083a8e/output.txt' does not exist Hard kill!GitHub has been notified of this commit’s build resultFinished: ABORTED

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 11:26:02 AM11/26/19
to jenkinsc...@googlegroups.com

?? I would not recommend using agent docker (or agent dockerfile). These ultimately rely on the withDockerContainer step and rather complex injection of docker exec into commands, which can easily break when there are issues with volume mounts and so on. Much safer and more transparent to use a plain label and run CLI docker commands (build, run) from sh.??

Oh that is painful given our online documentation almost seems to promote the containerized approach (i.e. https://jenkins.io/doc/book/pipeline/docker/)

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 11:38:02 AM11/26/19
to jenkinsc...@googlegroups.com

Marky Jackson Can you please format your messages/stack traces so they aren't just one long line?

jglick@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 11:43:02 AM11/26/19
to jenkinsc...@googlegroups.com

Oh I definitely promote using containers to run builds—just not with this plugin that I wrote much of and now deeply regret.

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 11:45:03 AM11/26/19
to jenkinsc...@googlegroups.com

Devin Nusbaum formatting is not working for some reason.

 

Jesse Glick I totally know this feeling

dnusbaum@cloudbees.com (JIRA)

unread,
Nov 26, 2019, 12:01:02 PM11/26/19
to jenkinsc...@googlegroups.com

Marky Jackson You mean formatting is not working in Jira? I think if you just use the non-WYSIWYG editor and surround whatever your text is in {noformat} it will work fine. It looks like your build log still contains:

[Pipeline] sh process apparently never started in /home/${USER}/workspace/ntend-pipeline_customer-settings@tmp/durable-d4083a8e (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Which I think still means that the LAUNCH_DIAGNOSTICS is still false. Did you restart Jenkins after setting the property? If you don't want to restart Jenkins, I think you can run "org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true" from the script console to enable it temporarily (not sure if Groovy will let you change the field though since it is private).

marky.r.jackson@gmail.com (JIRA)

unread,
Nov 26, 2019, 12:03:03 PM11/26/19
to jenkinsc...@googlegroups.com

I did a restart Devin Nusbaum. I did also just try running it in the script console

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 2:44:03 PM12/4/19
to jenkinsc...@googlegroups.com

Marky Jackson Once you got LAUNCH_DIAGNOSTICS enabled, did the build log have any additional information? If not, can you downgrade to Durable Task Plugin 1.30 (and Pipeline: Nodes and Processes 2.34) to see if things work correctly in that version? CC Carroll Chiou, potentially related to JENKINS-60107

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 2:51:03 PM12/4/19
to jenkinsc...@googlegroups.com
Devin Nusbaum edited a comment on Bug JENKINS-60286
[~jequals5] Once you got LAUNCH_DIAGNOSTICS enabled, did the build log have any additional information? If not, can you downgrade to Durable Task Plugin 1.30 (and Pipeline: Nodes and Processes 2.34) to see if things work correctly in that version? CC [~carroll], potentially related to JENKINS-60107 , or maybe JENKINS-60054 unless that is Kubernetes-specific?

marky.r.jackson@gmail.com (JIRA)

unread,
Dec 4, 2019, 3:29:03 PM12/4/19
to jenkinsc...@googlegroups.com

Devin Nusbaum at this point it appears someone upgraded a plugin which in turn upgraded 1.29 to 1.33 and that broke all the dependancies.

I moved everything back to native docker calls as Jesse Glick alluded to

 

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 3:55:03 PM12/4/19
to jenkinsc...@googlegroups.com

Marky Jackson Ok, if you end up with any additional information from LAUNCH_DIAGNOSTICS, or if you can figure out whether 1.30 has the same problem, feel free to add the info here. I am going to close the ticket.

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 3:55:04 PM12/4/19
to jenkinsc...@googlegroups.com
Devin Nusbaum closed an issue as Incomplete
 
Change By: Devin Nusbaum
Status: Open Closed
Resolution: Incomplete
Reply all
Reply to author
Forward
0 new messages