[JIRA] (JENKINS-55308) intermittent "terminated" messages using sh in Pipelines

147 views
Skip to first unread message

gc875@claypack.ca (JIRA)

unread,
Dec 21, 2018, 6:22:02 PM12/21/18
to jenkinsc...@googlegroups.com
gc875 created an issue
 
Jenkins / Bug JENKINS-55308
intermittent "terminated" messages using sh in Pipelines
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: docker, durable-task-plugin
Created: 2018-12-21 23:21
Environment: Test env:
* Jenkins 2.138.2 LTS
* Jenkins master is a docker image based on jenkins/jenkins:2.138.2-alpine
* Specific plugins are baked into the image using /usr/local/bin/install-plugins.sh and plugins.txt shown below
* Master runs on AWS ECS.
* Slaves are AWS EC2 instances

plugins.txt:
active-directory:2.8
ansicolor:0.5.2
aws-cloudwatch-logs-publisher:1.2.0
build-timeout:1.19
cloudbees-folder:6.6
credentials-binding:1.16
command-launcher:1.2
docker-workflow:1.17
ec2:1.40.1
git-client:2.7.3
git:3.9.1
gitlab-plugin:1.5.10
htmlpublisher:1.17
jdk-tool:1.1
pipeline-utility-steps:2.1.0
role-strategy:2.9.0
ssh-agent:1.17
ssh-slaves:1.28.1
timestamper:1.8.10
workflow-aggregator:2.6
ws-cleanup:0.35
Priority: Minor Minor
Reporter: gc875

Testing Jenkins 2.138.2 LTS, Jenkins pipelines that use sh intermittently throw the following message in the console log …

 

sh: line 1:  4449 Terminated              sleep 3

 

 … and sometimes this … 

 

sh: line 1: 13136 Terminated              { while [ ( -d /proc/$pid -o ! -d /proc/$$ ) -a -d '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0' -a ! -f '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0/jenkins-result.txt' ]; do    touch '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0/jenkins-log.txt'; sleep 3;{{done; }}}

 

Jenkins master runs from a Docker image based on jenkins/jenkins:2.138.2-alpine with specific plugins baked into the image by /usr/local/bin/install-plugins.sh

The message originates in durable-task-plugin, which must be a dependency of one of the plugins.txt plugins.  

Two important observations:

1) The issue does not occur when starting with the base jenkins/jenkins:2.138.2-alpine image and manually installing plugins via UI. That might suggest the issue is around how install-plugins.sh installs plugins and/or dependencies. 

2) The issue does not occur on our production image, which is also 2.138.2-alpine + plugins built 2018-10-11. Rebuilding the the same image from the same Dockerfile results in different installed plugins. Makes me think results using install-plugins.sh are not deterministic.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jonas.d.lindstrom@netent.com (JIRA)

unread,
Jan 18, 2019, 10:21:03 AM1/18/19
to jenkinsc...@googlegroups.com
Jonas Lindström commented on Bug JENKINS-55308
 
Re: intermittent "terminated" messages using sh in Pipelines

We run the jenkins-alpine docker image and we also get these messages. I tried switching to the Debian image and the intermittent "terminated" message are still there, so I don't think it's specifically related to the OS.

This seems to have been introduced after v1.22 of durable-task-plugin. I guess the error is lurking somewhere in this line in BourneShellScript.java:

            cmd = String.format("pid=$$; { while [ \\( -d /proc/$pid -o \\! -d /proc/$$ \\) -a -d '%s' -a \\! -f '%s' ]; do touch '%s'; sleep 3; done } & jsc=%s; %s=$jsc %s '%s' > '%s' 2> '%s'; echo $? > '%s.tmp'; mv '%s.tmp' '%s'; wait", 
... more args ...

It's difficult to downgrade to 1.22 since so many other plugins depend on later versions of this plugin.

jonas.d.lindstrom@netent.com (JIRA)

unread,
Jan 18, 2019, 10:21:04 AM1/18/19
to jenkinsc...@googlegroups.com
Jonas Lindström edited a comment on Bug JENKINS-55308
We run the jenkins-alpine docker image and we also get these messages. I tried switching to the Debian image (2.150.2) and the intermittent "terminated" message are still there, so I don't think it's specifically related to the OS.


This seems to have been introduced after v1.22 of durable-task-plugin. I guess the error is lurking somewhere in this line in BourneShellScript.java:
{code:java}

            cmd = String.format("pid=$$; { while [ \\( -d /proc/$pid -o \\! -d /proc/$$ \\) -a -d '%s' -a \\! -f '%s' ]; do touch '%s'; sleep 3; done } & jsc=%s; %s=$jsc %s '%s' > '%s' 2> '%s'; echo $? > '%s.tmp'; mv '%s.tmp' '%s'; wait",
... more args ...{code}

It's difficult to downgrade to 1.22 since so many other plugins depend on later versions of this plugin.

jenkins@gyoo.com (JIRA)

unread,
Jan 23, 2019, 3:29:03 AM1/23/19
to jenkinsc...@googlegroups.com
Jean-Paul G updated an issue
 
Change By: Jean-Paul G
Component/s: docker

jenkins@gyoo.com (JIRA)

unread,
Jan 23, 2019, 3:31:02 AM1/23/19
to jenkinsc...@googlegroups.com
Jean-Paul G commented on Bug JENKINS-55308
 
Re: intermittent "terminated" messages using sh in Pipelines

Not specific to docker.

Have the very same issue when running sh step with parallel

jenkins@gyoo.com (JIRA)

unread,
Jan 23, 2019, 3:32:02 AM1/23/19
to jenkinsc...@googlegroups.com
Jean-Paul G edited a comment on Bug JENKINS-55308
Not specific to docker.

Have the very same issue when running sh step with parallel , on ssh slave with Jenkins 2.156

Adam@vonNieda.org (JIRA)

unread,
Feb 5, 2019, 2:40:02 PM2/5/19
to jenkinsc...@googlegroups.com

I've got the same issue when running sh step without parallel, on ssh slave with Jenkins 2.150.2 LTS, I'm not running on docker.

connor.yu@cspi.com (JIRA)

unread,
Feb 13, 2019, 4:38:02 PM2/13/19
to jenkinsc...@googlegroups.com

I am seeing this as well. Upgraded to 2.46.3 yesterday and started seeing this in my simple pipeline.  

sh: line 1: 27323 Terminated              sleep 3

 

karl.dean.parry@gmail.com (JIRA)

unread,
Feb 18, 2019, 10:25:04 AM2/18/19
to jenkinsc...@googlegroups.com

I am seeing this as well. We have just seen it randomly. We do get it when we run a custom clean up script that deletes the @tmp/@script/@script@tmp folders 

sh: line 1: 27323 Terminated              sleep 3

mhall@tivo.com (JIRA)

unread,
Feb 19, 2019, 2:24:03 PM2/19/19
to jenkinsc...@googlegroups.com

Also seeing this. Jenkins 2.150.3 and durable-task 1.29, which was an upgrade from 1.25.

mkozell@magento.com (JIRA)

unread,
Feb 27, 2019, 5:58:05 PM2/27/19
to jenkinsc...@googlegroups.com

I upgraded Pipeline: Job (workflow-job) plugin from 2.25 to 2.31 and now see the issue.  I'm running durable-task 1.27.

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 14, 2019, 6:04:02 AM3/14/19
to jenkinsc...@googlegroups.com
Devin Nusbaum assigned an issue to Devin Nusbaum
 
Change By: Devin Nusbaum
Assignee: Carlos Sanchez Devin Nusbaum

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 14, 2019, 6:15:02 AM3/14/19
to jenkinsc...@googlegroups.com
Devin Nusbaum commented on Bug JENKINS-55308
 
Re: intermittent "terminated" messages using sh in Pipelines

The sleep 3 is part of the execution of a shell step. A background process touches a specific file on the agent every 3 seconds, and the Jenkins master checks the timestamp on that file as a proxy to know whether the script is still running or not. It seems based on the reports here that something is causing that process to be killed on some systems, but I don't have any ideas about what it could be offhand.

Does anyone have a Pipeline that can reproduce the issue consistently? If so, please add it here (simplifying as much as possible0 and specify the agent OS and sh implementation (e.g., dash, bash) for the agents running the sh steps.

mayqueen91@gmail.com (JIRA)

unread,
Mar 19, 2019, 10:10:04 AM3/19/19
to jenkinsc...@googlegroups.com

I can confirm that I'm seeing this issue constantly on pipeline with parallel tasks. Here is the code:

properties([parameters([string(defaultValue: '', description: '', name: 'LABEL_WITH_MANY_NODES', trim: false)])])

def nodes_and_labels = env.LABEL_WITH_MANY_NODES.split(',')

def nodes = nodes_and_labels.collect {
  nodesWithLabel(it)
}.flatten().toSet()

echo "Launching job on nodes: ${nodes}"

def tasks = nodes.collectEntries {
  ["${it}": runJob(it)]
}

@NonCPS
def nodesWithLabel(name) {
    return jenkins.model.Jenkins.instance.nodes.findAll { node ->
      if (node.computer.isOffline()) { return false }
      return node.getAssignedLabels().collect { it.name }.contains(name)
    }.collect { node ->
      node.getNodeName()
    }
}

def runJob(aNode) {
    return {
        node(aNode) {
            stage("Test" + aNode) {
                sh """                """
            }        
        }
    }
}

parallel tasks

mayqueen91@gmail.com (JIRA)

unread,
Mar 19, 2019, 10:12:06 AM3/19/19
to jenkinsc...@googlegroups.com
Natalia Natalia edited a comment on Bug JENKINS-55308
I can confirm that I'm seeing this issue constantly on pipeline with parallel tasks. Here is the code:


{code:java}

properties([parameters([string(defaultValue: '', description: '', name: 'LABEL_WITH_MANY_NODES', trim: false)])])

def nodes_and_labels = env.LABEL_WITH_MANY_NODES.split(',')

def nodes = nodes_and_labels.collect {
  nodesWithLabel(it)
}.flatten().toSet()

echo "Launching job on nodes: ${nodes}"

def tasks = nodes.collectEntries {
  ["${it}": runJob(it)]
}

@NonCPS
def nodesWithLabel(name) {
    return jenkins.model.Jenkins.instance.nodes.findAll { node ->
      if (node.computer.isOffline()) { return false }
      return node.getAssignedLabels().collect { it.name }.contains(name)
    }.collect { node ->
      node.getNodeName()
    }
}

def runJob(aNode) {
    return {
        node(aNode) {
            stage("Test" + aNode) {
                sh """                """
            }        
        }
    }
}

parallel tasks{code}


the system is MacOSX, High Sierra

mayqueen91@gmail.com (JIRA)

unread,
Mar 19, 2019, 10:19:04 AM3/19/19
to jenkinsc...@googlegroups.com
Mac OS X  10.13.6
the system is MacOSX BuildVersion: 17G65

$ /bin/bash --version
GNU bash , High Sierra version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 22, 2019, 9:56:03 AM3/22/19
to jenkinsc...@googlegroups.com

Natalia Natalia Thanks for posting your Pipeline! Here are some further questions to try to help narrow down what could be happening:

  • Just to confirm, your agents are running OS X?
  • You say you see the issue constantly, but can you be more specific (do you see the issue every other build, every 10th build, ...)?
  • Approximately how many parallel tasks are you running at once (10s, 100s, ...)?
  • How resource-intensive is your shell script (is it possibly triggering memory or CPU limits that would cause the Kernel to start killing processes)?
  • How are the agents connected to the Jenkins master (SSH, JNLP), and are they static machines or are they dynamically provisioned by the Jenkins master through something like the Docker Swarm or EC2 plugins?
  • If they are static machines, are they running anything else or are they only used as Jenkins agents? Assuming they are static, I would see if there are any particular agents where the shell script seems to be getting terminated consistently based on previous failed, and log onto those machines and check general system logs and running processes to see if there is anything that looks like it might be responsible for killing the process.

mayqueen91@gmail.com (JIRA)

unread,
Mar 22, 2019, 12:41:04 PM3/22/19
to jenkinsc...@googlegroups.com

Devin Nusbaum

  • My agents are Apple nodes, MacMinis
  • every run
  • I have around 20 nodes gathered in one label and it's just one same task for all of the nodes
  • shell code exactly like the above - keeping only sh """ """" produces the error in every run; so I would say it doesn't matter at all
  • agents are connected through ssh, they are static physical machines in the racks
  • they are only used as Jenkins agents

I'm not sure if you are getting the problem - the shell code is executed, basically it all works as expected. The only 'not-working' part for me is the noise in the log. It started when we upgraded core and plugins to the newest version.

mayqueen91@gmail.com (JIRA)

unread,
Mar 22, 2019, 12:52:02 PM3/22/19
to jenkinsc...@googlegroups.com
Natalia Natalia edited a comment on Bug JENKINS-55308
[~dnusbaum]  
* My agents are Apple nodes, MacMinis
* every run
* I have around 20 nodes gathered in one label and it's just one same task for all of the nodes
* shell code exactly like the above - keeping only sh """ """" produces the error in every run; so I would say it doesn't matter at all
* agents are connected through ssh, they are static physical machines in the racks
* they are only used as Jenkins agents


I'm not sure if you are getting the problem - the shell code is executed, basically it all works as expected. The only 'not-working' part for me is the noise in the log. It started when we upgraded core and plugins to the newest version.


EDIT: I also have other nodes provisioned in the cloud (manually), connected through ssh, just tested on these and it's the same issue.

# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
$ /bin/bash --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

mayqueen91@gmail.com (JIRA)

unread,
Mar 22, 2019, 12:53:03 PM3/22/19
to jenkinsc...@googlegroups.com
Natalia Natalia edited a comment on Bug JENKINS-55308
[~dnusbaum]  
* My agents are Apple nodes, MacMinis
* every run
* I have around 20 nodes gathered in one label and it's just one same task for all of the nodes
* shell code exactly like the above - keeping only sh """ """" produces the error in every run; so I would say it doesn't matter at all
* agents are connected through ssh, they are static physical machines in the racks
* they are only used as Jenkins agents

I'm not sure if you are getting the problem - the shell code is executed, basically it all works as expected. The only 'not-working' part for me is the noise in the log. It started when we upgraded core and plugins to the newest version.

EDIT: I also have other nodes provisioned in the cloud (manually), connected through ssh, just tested on these and it's the same issue.

{code}
# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
$ /bin/bash --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

{code}

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 22, 2019, 1:01:08 PM3/22/19
to jenkinsc...@googlegroups.com

the shell code is executed, basically it all works as expected. The only 'not-working' part for me is the noise in the log.

Thanks for clarifying, I didn't realize that things were working apart from the log message and that you were able to reproduce the issue by literally running a blank sh step. I'll try to create a unit test that can reproduce the problem based on your info when I have some time.

wl2776@gmail.com (JIRA)

unread,
Apr 5, 2019, 10:36:04 AM4/5/19
to jenkinsc...@googlegroups.com

I see this error on Jenkins 2.164.1

One my pipeline job started failing today with unclear reason. 

Both successful jobs and failed jobs contain "Terminated"  in console logs

vennila.perumal@aspiresys.com (JIRA)

unread,
Apr 24, 2019, 7:46:03 AM4/24/19
to jenkinsc...@googlegroups.com

hi,

I am also having the same problem where the console log is contains the "sh: line 1:  xxx Terminated              sleep 3" message unwantedly.

May I know how to get rid of this ?

Jenkins version :  2.150

 

dhanshree.kadam@ericsson.com (JIRA)

unread,
Apr 25, 2019, 6:30:03 AM4/25/19
to jenkinsc...@googlegroups.com
Dhanshree Kadam assigned an issue to Dhanshree Kadam
 
Change By: Dhanshree Kadam
Assignee: Devin Nusbaum Dhanshree Kadam

dhanshree.kadam@ericsson.com (JIRA)

unread,
Apr 25, 2019, 6:31:02 AM4/25/19
to jenkinsc...@googlegroups.com
Dhanshree Kadam assigned an issue to Devin Nusbaum
Change By: Dhanshree Kadam
Assignee: Dhanshree Kadam Devin Nusbaum

jonas.d.lindstrom@netent.com (JIRA)

unread,
May 10, 2019, 6:45:03 AM5/10/19
to jenkinsc...@googlegroups.com
Jonas Lindström commented on Bug JENKINS-55308
 
Re: intermittent "terminated" messages using sh in Pipelines

Mike Kozell was on to something: the bug is not just in the durable-task-plugin, although the symptoms come from there. It is introduced when you upgrade workflow-job. I have managed to pinpoint it down to a specific version.

The following setup does not trigger the bug. All are at latest version at the time of writing, except workflow-job:

  • durable-task 1.29
  • workflow-api 2.33
  • workflow-durable-task-step 2.30
  • workflow-step-api 2.19
  • workflow-support 3.3
  • workflow-job 2.25

Upgrading workflow-job to 2.27 or later triggers the bug. (2.26 does not exist.)

jonas.d.lindstrom@netent.com (JIRA)

unread,
May 10, 2019, 6:45:08 AM5/10/19
to jenkinsc...@googlegroups.com
Jonas Lindström updated an issue
 
Change By: Jonas Lindström
Component/s: workflow-job-plugin

jonas.d.lindstrom@netent.com (JIRA)

unread,
May 10, 2019, 6:48:03 AM5/10/19
to jenkinsc...@googlegroups.com
Jonas Lindström edited a comment on Bug JENKINS-55308
 
Re: intermittent "terminated" messages using sh in Pipelines
[~mkozell] was on to something: the bug is not just in the durable-task-plugin, although the symptoms come from there. It is introduced when you upgrade workflow-job. I have managed to pinpoint it down to a specific version.

The following setup does not trigger the bug (relevant plugins listed) . All are at latest version at the time of writing, except workflow-job:
* durable-task 1.29
* workflow-api 2.33
* workflow-durable-task-step 2.30
* workflow-step-api 2.19
* workflow-support 3.3
* *workflow-job 2.25*


Upgrading workflow-job to 2.27 or later triggers the bug. (2.26 does not exist.)

avi@scylladb.com (JIRA)

unread,
May 14, 2019, 9:12:05 AM5/14/19
to jenkinsc...@googlegroups.com

I am seeing the same problem, parallel jobs running on Linux agents.  I can share the pipeline code if it will help.

martin.wiklundh@ericsson.com (JIRA)

unread,
May 14, 2019, 9:34:02 AM5/14/19
to jenkinsc...@googlegroups.com

We see the same problem intermittently, in the afternoons only. Appeared without any known change in our Jenkins setup. Network/storage issues may produce the same error.

kierzo@kierzo.com (JIRA)

unread,
Jun 27, 2019, 5:32:03 AM6/27/19
to jenkinsc...@googlegroups.com

Hey,

Had this issue on the my Mac build.

My fix was to set the JAVA_HOME variable which was missing on the machine.

I created and added this file on the mac : "~/.bash_profile"

added this line....

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home

rebooted the mac and its fixed it.

your "jdk-11.0.2.jdk" folder may be different so just replace that with whatever is in there.

if you run the command:

echo ${JAVA_HOME}

 

and its empty this could be the cause.

 

Hope this helps others.

 

Cheers Kieron.

 

 

jsosic@gmail.com (JIRA)

unread,
Jul 23, 2019, 8:47:04 AM7/23/19
to jenkinsc...@googlegroups.com

It seems that adding $JAVA_HOME in Global properties indeed fixes this!

Since I run CentOS 7, this is what I've added:

Name: JAVA_HOME

Value: /usr/lib/jvm/jre-1.8.0-openjdk

cuong.tran@gmail.com (JIRA)

unread,
Jul 23, 2019, 10:13:03 PM7/23/19
to jenkinsc...@googlegroups.com

Was this property added to the agent or the master?

ericchou19831101@msn.com (JIRA)

unread,
Sep 3, 2019, 9:32:06 AM9/3/19
to jenkinsc...@googlegroups.com
Wen Zhou commented on Bug JENKINS-55308

if the bug is really introduced by the workflow-job plugin, can we expect a fix in the new release?

 

prodriguezguillamon@cloudbees.com (JIRA)

unread,
Oct 18, 2019, 10:07:04 AM10/18/19
to jenkinsc...@googlegroups.com

The issue could be reproduced by running this simple Pipeline:

Pipeline script
pipeline {
 agent { label 'rhel6' }
  stages {
   stage('build') {
    steps {
     echo "Hello World!"
     sh "echo Hello from the shell"
}
}
}
}

Console output:

[Pipeline] Start of Pipeline
[Pipeline] node
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
Hello World!
[Pipeline] sh
+ echo Hello from the shell
Hello from the shell
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
sh: line 1: 18285 Terminated sleep 3
Finished: SUCCESS

Environment:

  • CloudBees Jenkins Platform - Master 2.176.2.3

  • workflow-job:2.32 'Pipeline: Job'

It started happening after workflow-job was updated from 2.25 to 2.32
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

prodriguezguillamon@cloudbees.com (JIRA)

unread,
Oct 18, 2019, 10:08:03 AM10/18/19
to jenkinsc...@googlegroups.com
Pablo Rodriguez edited a comment on Bug JENKINS-55308
The issue could be reproduced by running this simple Pipeline:
{code:java}
Pipeline script
pipeline {
agent { label ' rhel6 nodelinux ' }

  stages {
   stage('build') {
    steps {
     echo "Hello World!"
     sh "echo Hello from the shell"
}
}
}
}
{code}
Console output:
{code:java}

[Pipeline] Start of Pipeline
[Pipeline] node
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
Hello World!
[Pipeline] sh
+ echo Hello from the shell
Hello from the shell
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
sh: line 1: 18285 Terminated sleep 3
Finished: SUCCESS
{code}
Environment:
* CloudBees Jenkins Platform - Master [2.176.2.3|https://release-notes.cloudbees.com/release/1/2.176.2.3]
*
workflow-job:2.32 'Pipeline: Job'

It started happening after workflow-job was updated from 2.25 to 2.32

cchiou@cloudbees.com (JIRA)

unread,
Oct 22, 2019, 1:54:04 PM10/22/19
to jenkinsc...@googlegroups.com

The sleep 3 process was introduced when the heartbeat check feature was added in version 1.16 of durable-task, which translates to 2.18 of workflow-durable-task-step, which translates to 2.27 of workflow-job.

Unless I missed a comment, it looks like the pipeline behaves as expected outside of "Terminated sleep 3" line.

At least in Pablo Rodriguez’s example, this might be a race condition between the sleep 3 process and when the script completes. Like Devin Nusbaum mentioned earlier, the sleep 3 process is used to touch the output log file to show the script is still alive. However, when the script completes, it will write to a separate result file. A watcher service is checking for that result file every 100ms. Once that result file is found, results are transmitted and everything related to that specific step’s workspace is purged. It might be possible that the output file gets cleaned up right after the sleep 3 process checks if the file still exists, but before it gets touched again?

There will be a new release of durable-task that will remove the sleep 3 process so this line won't pop up anymore.

cchiou@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 2:11:03 PM10/24/19
to jenkinsc...@googlegroups.com
Carroll Chiou edited a comment on Bug JENKINS-55308
The sleep 3 process was introduced when the heartbeat check feature was added in version 1.16 of durable-task, which translates to 2.18 of workflow-durable-task-step, which translates to 2.27 of workflow-job.

Unless I missed a comment, it looks like the pipeline behaves as expected outside of "Terminated sleep 3" line.

At least in [~parogui]’s [example|https://issues.jenkins-ci.org/browse/JENKINS-55308?focusedCommentId=377961&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-377961], this might be a race condition between the sleep 3 process and when the script completes. Like [~dnusbaum] [mentioned earlier|https://issues.jenkins-ci.org/browse/JENKINS-55308?focusedCommentId=362789&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-362789], the sleep 3 process is used to touch the output log file to show the script is still alive.  However, when the script completes, it will write to a separate result file. A watcher service is checking for that result file every 100ms. Once that result file is found, results are transmitted and everything related to that specific step’s workspace is purged. It might be possible that the output file gets cleaned up right after the sleep 3 process checks if the file still exists, but before it gets touched again?

There will be is a new release of durable-task (1.31) that will remove removes the sleep 3 process so this line won't pop up anymore.

cchiou@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 2:22:04 PM10/24/19
to jenkinsc...@googlegroups.com
Carroll Chiou assigned an issue to Carroll Chiou
 
Change By: Carroll Chiou
Assignee: Devin Nusbaum Carroll Chiou

cchiou@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 2:25:08 PM10/24/19
to jenkinsc...@googlegroups.com
Carroll Chiou started work on Bug JENKINS-55308
 
Change By: Carroll Chiou
Status: Open In Progress

cchiou@cloudbees.com (JIRA)

unread,
Oct 24, 2019, 2:28:04 PM10/24/19
to jenkinsc...@googlegroups.com
Carroll Chiou edited a comment on Bug JENKINS-55308
The sleep 3 process was introduced when the heartbeat check feature was added in version 1.16 of durable-task, which translates to 2.18 of workflow-durable-task-step, which translates to 2.27 of workflow-job.

Unless I missed a comment, it looks like the pipeline behaves as expected outside of "Terminated sleep 3" line.

At least in [~parogui]’s [example|https://issues.jenkins-ci.org/browse/JENKINS-55308?focusedCommentId=377961&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-377961], this might be a race condition between the sleep 3 process and when the script completes. Like [~dnusbaum] [mentioned earlier|https://issues.jenkins-ci.org/browse/JENKINS-55308?focusedCommentId=362789&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-362789], the sleep 3 process is used to touch the output log file to show the script is still alive.  However, when the script completes, it will write to a separate result file. A watcher service is checking for that result file every 100ms. Once that result file is found, results are transmitted and everything related to that specific step’s workspace is purged. It might be possible that the output file gets cleaned up right after the sleep 3 process checks if the file still exists, but before it gets touched again?

There is a new release of durable-task (1.31) that removes the sleep 3 process so this line won't pop up anymore.
Update: I  have  not been able to reproduce this issue, so I can't say for certain if this issue is resolved. Technically, it should, but it could be possible that the new version  just changes the behavior of this bug.

aalvarez@forhims.com (JIRA)

unread,
Oct 30, 2019, 1:58:03 PM10/30/19
to jenkinsc...@googlegroups.com

Can confirm that after upgrading durable-task to the latest version, the Terminated messages are gone from our jobs.

cchiou@cloudbees.com (JIRA)

unread,
Oct 30, 2019, 2:35:05 PM10/30/19
to jenkinsc...@googlegroups.com

A Alvarez I'm curious, if you are running 1.33, are you passing in the FORCE_BINARY_WRAPPER=true system property?

aalvarez@forhims.com (JIRA)

unread,
Oct 30, 2019, 4:21:03 PM10/30/19
to jenkinsc...@googlegroups.com

Hi Carroll Chiou not as far as I'm aware although this could be a default or coming from another plugin. All we did was upgrading the plugin and the messages disappeared from the console output when running shell commands within Pipeline scripts

cchiou@cloudbees.com (JIRA)

unread,
Oct 30, 2019, 4:51:04 PM10/30/19
to jenkinsc...@googlegroups.com

A Alvarez thanks, that's good to know
Reply all
Reply to author
Forward
0 new messages