[JIRA] (JENKINS-50379) Jenkins kills long running sh script with no output

[evan.ward@nrl.navy.mil (JIRA)]

Evan Ward created an issue

Jenkins / JENKINS-50379 Jenkins kills long running sh script with no output Issue Type: Bug Assignee: Unassigned Components: durable-task-plugin Created: 2018-03-23 19:53 Environment: Jenkins ver. 2.107.1 on CentOS 7 Priority: Minor Reporter: Evan Ward I have a Jenkins pipeline that runs a shell script that takes about 5 minutes and generates no output. The job fails and I'm seeing the following in the output: wrapper script does not seem to be touching the log file in /home/jenkins/workspace/job_Pipeline@2@tmp/durable-595950a5 (--JENKINS-48300--: if on a laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=300) script returned exit code -1 Based on JENKINS-48300 it seems that Jenkins is intentionally killing my script while it is still running. IMHO it is a bug for Jenkins to assume that a shell script will generate output every n seconds for any finite n. As a workaround I've set -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL to one hour. But what happens when I have a script that takes an hour and one minute!? Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output Does it work when you echo whatever ; yourscript.sh instead of just the latter?

Add Comment

[nullify005@gmail.com (JIRA)]

Lee Webb commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Travis does this sort of thing too, if there's no output for a while it just assumes the process is hung then stops the build. If you don't want to mess with Jenkins something like the following shell snippet can help. It forks your long running process & echo's dots to the console as long as it's still running: # suppress command output unless there is a failure function quiet() { if [[ $- =~ x ]]; then set +x; XTRACE=1; fi if [[ $- =~ e ]]; then set +e; ERREXIT=1; fi tmp=$(mktemp) || return # this will be the temp file w/ the output echo -ne "quiet running: ${@} " ts_elapsed=0 ts_start=$(date +%s) "${@}" > "${tmp}" 2>&1 & cmd_pid=$! while [ 1 ]; do if [ `uname` == 'Linux' ]; then ps -q ${cmd_pid} > /dev/null 2>&1 running=${?} else ps -ef ${cmd_pid} > /dev/null 2>&1 running=${?} fi if [ "${running}" -eq 0 ]; then echo -ne '.' sleep 3 continue fi break done wait ${cmd_pid} ret=${?} ts_end=$(date +%s) let "ts_elapsed = ${ts_end} - ${ts_start}" if [ "${ret}" -eq 0 ]; then echo -ne " finished with code ${ret} in ${ts_elapsed} secs, last lines were:\n" tail -n 4 "${tmp}" else cat "${tmp}" fi rm -f "${tmp}" if [ "${ERREXIT}" ]; then unset ERREXIT; set -e; fi if [ "${XTRACE}" ]; then unset XTRACE; set -x; fi return "${ret}" # return the exit status of the command }

Add Comment

[evan.ward@nrl.navy.mil (JIRA)]

Evan Ward commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Daniel Beck the script initially generates some output to show that it started and then generates no output for a long time. I think this has the same effect as your suggestion of using echo.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Evan Ward I expect so. Thanks for the clarification.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

I see this issue on scripts which do generate some output, but it happens that parts of the script take some time to run: in my case I'm compiling a kernel module and even when the make output is sent to the console, sometimes individual steps take longer than the timeout...

Add Comment

[shreedhara.isc@gmail.com (JIRA)]

shreedhara H commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Hi Evan Ward, We are also facing the same issue, can you please help us to know how to change Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL interval.

Add Comment

[evan.ward@nrl.navy.mil (JIRA)]

Evan Ward commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Set it in the JVM arg line on master.

Add Comment

[max.ivanch@gmail.com (JIRA)]

Max Ivanch commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Hi there,   Same issue I run task with high load disk tasks. I put durable plugin in "none" option, I tried HEARTBEAT_CHECK_INTERVAL but it doesn't work for me. To have solution I have created additional mount point in jenkins slave. But IMHO I would prefer to have option to disable it at all.

Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

[18332563813@163.com (JIRA)]

lei rou commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

I have the same problem some times, and how to change the HEARTBEAT_CHECK_INTERVAL?

Add Comment

[sr.professional88@gmail.com (JIRA)]

swarnendu roy commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Hi There,   I am also facing this issue now in our environment. If there is any work around for this.   ```wrapper script does not seem to be touching the log file in /home/****/workspace/demo@tmp/durable-549a8a8c (JENKINS-48300: if on a laggy filesystem, consider -Dorg.****ci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=300)```

Add Comment

[tamas.gal@me.com (JIRA)]

Tamas Gal commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Same for us. Out of nowhere jobs are killed on our Jenkins nodes. Manually setting the heartbeat check interval to 300 seems to work for now.

Add Comment

[tamas.gal@me.com (JIRA)]

Tamas Gal edited a comment on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Same for us. Out of nowhere jobs are killed on our Jenkins nodes. Manually setting the heartbeat check interval to 300 seems to work for now.

Btw. on Debian-like machines, you need to edit `/var/default/jenkins` and add the above mentioned variable setting to the line starting with *{{JAVA_ARGS=}}* It should then look something like this: *{{JAVA_ARGS="-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=300 -Djava.awt.headless=true ..."}}*

Add Comment

[sr.professional88@gmail.com (JIRA)]

swarnendu roy commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Hi Guys,  This issue is due to Durable Task Pluggin, In the latest release of Durable task Pluggin 1.25 this has been resolved. reference Link : https://issues.jenkins-ci.org/browse/JENKINS-52881

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

More likely related to JENKINS-48300. Impossible to diagnose merely from this message. The problem is not that your script stops producing output for a while. That is perfectly normal and supported. The problem is that a side process which is supposed to be detecting this fact and touching the log file every three seconds is either not running, or not producing the right timestamp as observed by the Jenkins agent JVM.

Add Comment

[hboeken@me.com (JIRA)]

Hermann Boeken commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

After having updated Jenkins and its plugins, we're experiencing this issue too now. We're now using Jenkins 2.193 and the Durable Task Plugin has version 1.30. wrapper script does not seem to be touching the log file in /local/user_data/s__t/jenkins/workspace/S___K@7@tmp/durable-ad608bf9 (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400) This 'extremely laggy filesystem' is a local hard disk which isn't laggy whatsoever.  About 50% of our jobs get aborted due to this.  Do you have any suggestions how this can get solved without the workaround to redefine the HEARTBEAT_CHECK_INTERVAL?

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

Only by diagnosing and figuring out how to reproduce, so the issue can be fixed.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

> The problem is not that your script stops producing output for a while. That is perfectly normal and supported. The problem is that a side process which is supposed to be detecting this fact and touching the log file every three seconds is either not running, or not producing the right timestamp as observed by the Jenkins agent JVM. Right, so it sounds like we need to investigate why the side process that should be touching the log file isn't working properly.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-50379

Re: Jenkins kills long running sh script with no output

I should have mentioned that JENKINS-25503 would completely reimplement the code involved here, possibly solving this issue (possibly introducing others).

Add Comment