[JIRA] [core] (JENKINS-17116) gracefull job termination

I'm also have problems with this.
In particular, our nodes are running Ubuntu 14.04. We are using Jenkins to run some tests as part of the build. There are a few steps where interruption will cause communication failures, leaked temporary files gigabytes in size, and locks that are not undone. Orphaned process is very bad as well, since this could lead to a new build starting communication on the same channel prior to the previous one terminating.

Like Martin d'Anjou indicated, we would need a timeout parameter, since the allowed timeout for SIGTERM may be about 300sec, which is probably a lot longer than someone implementing this fix may anticipate.

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

erez.k@taboola.com (JIRA)

unread,

Jan 24, 2017, 4:09:02 AM1/24/17

to jenkinsc...@googlegroups.com

Erez Kashi commented on

I have the same issue. When canceling Job, I am trying to signal in side a python script, and cleanup.
It there any workaround for this issue?

akostadinov@java.net (JIRA)

unread,

Jan 24, 2017, 5:17:04 PM1/24/17

to jenkinsc...@googlegroups.com

akostadinov commented on

Erez Kashi, it is possible to:
1. remove the jenkins cookie environment variable
2. run your program in background (output still can go to stdout)
3. launch another background process to check original process PID, such that when it is gone, it would kill the other child gracefully
4. in main process, wait for the other two to complete (make sure second monitor process would exit if the first background process exits)
5. take care to report proper termination status of the program

Not very nice probably but you can script it to run arbitrary shell commands in this way. Also might not be worth the effort. It didn't for me.

akostadinov@java.net (JIRA)

unread,

Jan 24, 2017, 5:33:04 PM1/24/17

to jenkinsc...@googlegroups.com

akostadinov edited a comment on

[~kashierez], it is possible to:

1. remove the jenkins cookie environment variable
2. run your program in background (output still can go to stdout)
3. launch another background process to check original process PID, such that when it is gone, it would kill the other child gracefully
4. in main process, wait for the other two to complete (make sure second monitor process would exit if the first background process exits)
5. take care to report proper termination status of the program

Not very nice probably but you can script it to run arbitrary shell commands in this way. Also might not be worth the effort. It didn't for me.

Forgot to mention: this would only work on UNIX derivatives IIRC.

erez.k@taboola.com (JIRA)

unread,

Jan 25, 2017, 4:15:02 AM1/25/17

to jenkinsc...@googlegroups.com

Erez Kashi commented on

Thanks for the quick response . I will try ...

martin.danjou14@gmail.com (JIRA)

unread,

Feb 7, 2017, 9:53:08 AM2/7/17

to jenkinsc...@googlegroups.com

Martin d'Anjou commented on

I have done more experiments. And I am still not seeing the signal being sent, like Daniel Beck is seeing.

I started with this Java version:

Java1.8.0_77. OS is Fedora release 14 (Laughlin).

The Jenkins console shows:

 
                                                                [freestyle-kill] $ /bin/sh -xe /tmp/hudson3073245061937599649.sh
+ trap 'echo TERM >terminated.txt' TERM
+ sleep 120
Build was aborted
Aborted by martinda
Finished: ABORTED
 
                                                            

Observe that the script is not printing "TERM" to the file, like it does in Daniel's environment.

I also tried these Java versions ones:

Java OpenJDK 1.8.0_111, Red Hat Enterprise Linux Server release 6.6 (Santiago)
Java HotSpot 1.8.0_121, Ubuntu 16.04 LTS (Xenial Xerus)
Java OpenJDK 1.8.0_121, Ubuntu 16.04 LTS (Xenial Xerus)

I captured some logs using OpenJDK 1.8.0_121 on Ubuntu 16.04. In the terminal running Jenkins:

 
                                                                INFO: jenkins-17116/freestyle #4 aborted
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
        at hudson.Proc$LocalProc.join(Proc.java:318)
        at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:135)
        at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:95)
        at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:64)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
        at hudson.model.Build$BuildExecution.build(Build.java:205)
        at hudson.model.Build$BuildExecution.doRun(Build.java:162)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
        at hudson.model.Run.execute(Run.java:1720)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:404)
 
                                                            

In the jenkins log recorder:

 
                                                                Feb 07, 2017 9:42:11 AM FINE hudson.util.ProcessTree
killAll: process=java.lang.UNIXProcess@2a8af379 and envs={HUDSON_COOKIE=2d16a893-7e22-4360-aad0-0931104599a5}
Feb 07, 2017 9:42:11 AM FINE hudson.util.ProcessTree
Recursively killing pid=25054
Feb 07, 2017 9:42:11 AM FINE hudson.util.ProcessTree
Recursively killing pid=25055
Feb 07, 2017 9:42:11 AM FINE hudson.util.ProcessTree
Killing pid=25055
Feb 07, 2017 9:42:11 AM FINE hudson.util.ProcessTree
Killing pid=25054
 
                                                            

None of them traps the signal.

dbeck@cloudbees.com (JIRA)

unread,

Feb 7, 2017, 11:59:04 AM2/7/17

to jenkinsc...@googlegroups.com

Daniel Beck assigned an issue to Unassigned

Jenkins /

Change By:	Daniel Beck
Assignee:	Kohsuke Kawaguchi

robin.jarry@gmail.com (JIRA)

unread,

Feb 17, 2017, 2:16:04 PM2/17/17

to jenkinsc...@googlegroups.com

Robin Jarry assigned an issue to Robin Jarry

Jenkins /

Change By:	Robin Jarry
Assignee:	Robin Jarry

robin.jarry@gmail.com (JIRA)

unread,

Feb 17, 2017, 2:24:01 PM2/17/17

to jenkinsc...@googlegroups.com

Robin Jarry commented on

I have 2 patches that add support for killing launched processes with specific signals.

I'll submit them in a PR asap.

This is unfortunately a *NIX-only solution where signals are supported. For windows I don’t know what to do.

stephan.reiter@gmail.com (JIRA)

unread,

May 1, 2018, 2:31:04 AM5/1/18

to jenkinsc...@googlegroups.com

Stephan Reiter commented on

Check out pull request https://github.com/jenkinsci/jenkins/pull/3414

I added code that will make Jenkins wait for process termination (for up to 30secs, should be made configurable).

Behavior changes are as follows:

On Windows, Jenkins sends Ctrl+C for up to 30secs. If the process hasn't exitted by then, it will be terminated like before.
On Linux, we send SIGTERMs for up to 30secs. If the process is still around after that, we continue as before: we close stdin/stdout/stderr which causes the process to terminate. (Note that we could send SIGKILL.)

Note that Jenkins doesn't use SIGKILL! It uses SIGTERM, but doesn't give the process any time to handle it before closing stdin/stdout/stderr.

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

rm2687@gmail.com (JIRA)

unread,

Jun 11, 2018, 5:18:02 AM6/11/18

to jenkinsc...@googlegroups.com

Rahul Mahajan commented on

Hi,

Any update regarding this issue? I am facing the same issue, where I am able to terminate gracefully using the command line, but when I issue the job using the Jenkins and abort it, it wont end gracefully.

It has to be noted, that if I abort using using the command line of a running job, the graceful termination is visible on Jenkins as well. But not when we abort using Jenkins, which is quite weird.

msinclair@juniper.net (JIRA)

unread,

Jun 11, 2018, 9:02:02 AM6/11/18

to jenkinsc...@googlegroups.com

Mark Sinclair commented on

It maybe a good idea to create a plugin that remaps the abort button to run a script specified in each job. It could be optionally configured to run the script, then do the normal abort process. For example:

1. User abort triggered.

2. Job specific abort script runs without interrupting what the job is currently running. A timeout counter starts simultaneously.

3. At completion of the script or timeout, Jenkins checks the job status. If the job is ready to exit normally, it does so - this will allow final status other than abort. If the job is still running, the normal Jenkins abort procedure takes over.

stephan.reiter@gmail.com (JIRA)

unread,

Jun 11, 2018, 9:07:03 AM6/11/18

to jenkinsc...@googlegroups.com

Stephan Reiter commented on

Well, I am still pursuing the graceful-termination via SIGTERM. An stepping stone for this is ready to be merged into a library that is used by Jenkins for process management on Windows. Now, a new version of that library needs to be bundled with Jenkins and then used in my pull request.

stephan.reiter@gmail.com (JIRA)

unread,

Jun 11, 2018, 9:08:04 AM6/11/18

to jenkinsc...@googlegroups.com

Stephan Reiter edited a comment on

Well, I am still pursuing the graceful-termination via SIGTERM. An A stepping stone for this is ready to be merged into a library that is used by Jenkins for process management on Windows. Now, a new version of that library needs to be bundled with Jenkins and then used in my pull request.

stephan.reiter@gmail.com (JIRA)

unread,

Jun 11, 2018, 9:08:12 AM6/11/18

to jenkinsc...@googlegroups.com

Stephan Reiter edited a comment on

Well, I am still pursuing the graceful-termination via SIGTERM. A stepping stone for this is ready to be merged into a library that is used by Jenkins for process management on Windows. Now After that has happened , a new version of that library needs to be bundled with Jenkins and then used in my pull request.

rm2687@gmail.com (JIRA)

unread,

Jun 11, 2018, 11:06:03 AM6/11/18

to jenkinsc...@googlegroups.com

Rahul Mahajan commented on

Hey, Stephan Reiter can we use the library(if its available) in our existing Jenkins setup? And if yes, how can we do that? Thanks for your help!

stephan.reiter@gmail.com (JIRA)

unread,

Jun 11, 2018, 12:04:02 PM6/11/18

to jenkinsc...@googlegroups.com

Stephan Reiter commented on

Hi Rahul,

The library alone is not enough - Jenkins needs to be recompiled to use it.

WinP is the library I was talking about and needs to get the following change applied: https://github.com/kohsuke/winp/pull/49
After that we can recompile Jenkins with the new WinP library and this change: https://github.com/jenkinsci/jenkins/pull/3414

I am really hoping that we can move faster with those changes. I am usually somewhat patient, but the perceived lack of interest from Jenkins maintainers is hard to understand.

rm2687@gmail.com (JIRA)

unread,

Jun 11, 2018, 1:27:02 PM6/11/18

to jenkinsc...@googlegroups.com

Rahul Mahajan commented on

Thanks Stephan Reiter..Will wait for it, until then we can work around the problem.

stephan.reiter@gmail.com (JIRA)

unread,

Jun 11, 2018, 1:35:03 PM6/11/18

to jenkinsc...@googlegroups.com

Stephan Reiter commented on

Cool, Rahul. We've been working around it for 4 years. But now we got tired of the performance penalty the workarounds introduce for us, so I said: How hard can it be (to fix the problem)? Turns out, not hard. Just getting it done is. sigh

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 31, 2018, 2:19:04 PM8/31/18

to jenkinsc...@googlegroups.com

Oleg Nenashev started work on

Change By:	Oleg Nenashev
Status:	Open In Progress

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 31, 2018, 2:19:10 PM8/31/18

to jenkinsc...@googlegroups.com

Oleg Nenashev updated

Jenkins /

Change By:	Oleg Nenashev
Status:	In Progress Review

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 31, 2018, 2:19:15 PM8/31/18

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

I believe it is going to land in the next weekly

hashar@free.fr (JIRA)

unread,

Sep 14, 2018, 4:03:04 PM9/14/18

to jenkinsc...@googlegroups.com

Antoine Musso commented on

The merge https://github.com/jenkinsci/jenkins/commit/d8eac92ee9a1c19bf145763589f1c152607bf3ed is in tag jenkins-2.141

martin.danjou14@gmail.com (JIRA)

unread,

Sep 15, 2018, 9:04:08 AM9/15/18

to jenkinsc...@googlegroups.com

Martin d'Anjou commented on

with Jenkins 2.141, I ran the bash script, and the python script and there is no change. Jenkins still leaks processes, and still the signals are not trapped by the user script. The is one difference though, the first click on the terminate button (the red [x]) does not kill the job immediately, but that seems to change nothing

martin.danjou14@gmail.com (JIRA)

unread,

Sep 15, 2018, 9:06:05 AM9/15/18

to jenkinsc...@googlegroups.com

Martin d'Anjou edited a comment on

with With Jenkins 2.141, I ran the [ bash script |#comment-177337] , and the [python script|#comment-182402] and there is no change. Jenkins still leaks processes, and still the signals are not trapped by the user script. The is one difference though, the first click on the terminate button (the red { color:#de350b} { { [x]}} {color} ) does not kill the job immediately, but that seems to change nothing .

martin.danjou14@gmail.com (JIRA)

unread,

Sep 15, 2018, 9:07:03 AM9/15/18

to jenkinsc...@googlegroups.com

Martin d'Anjou edited a comment on

With Jenkins 2.141, I ran the [ bash script |https://issues.jenkins-ci.org/browse/JENKINS-17116?focusedCommentId=177337&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-177337] , and the [python script|#comment-182402] and there is no change. Jenkins still leaks processes, and still the signals are not trapped by the user script. The is one difference though, the first click on the terminate button (the red {color:#de350b}{{[x]}}{color}) does not kill the job immediately, but that seems to change nothing.

o.v.nenashev@gmail.com (JIRA)

unread,

Sep 15, 2018, 9:40:03 AM9/15/18

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Martin d'Anjou Leaking of processes is unrelated to this fix.

Usual causes:

You use 32bit-Java on a 64bit machine
You use tool wrappers like Cygwin which mess up the process tree in Windows (See the Cygwin Process Killer plugin)
The processes are spawned without inheriting Build reference variables, so the library cannot pick them up if parent processes are already aborted, and the process is orphaned

I suggest creating a separate issue if none of the above is your case

vicmagana@hotmail.com (JIRA)

unread,

Nov 20, 2018, 2:39:02 PM11/20/18

to jenkinsc...@googlegroups.com

Victor Magana commented on

Hello, I'm seeing an error in the hudson.util.ProcessTree logger, "External Ctrl+C execution failed for process pid=3872. Ctrl+C process exited with code -1073741515: Failed to attach to the console". Is there any option/parameter that needs to be set for this to attach and send the Ctrl+C signal. I'm running Jenkins Server on Windows 7x64 version 2.150. Running a Windows batch job on the local master that executes a python script. Also ran as Execute Python Script job, same error. Thanks for any help.

Failed to send CTRL+C to pid=3872
org.jvnet.winp.WinpException: External Ctrl+C execution failed for process pid=3872. Ctrl+C process exited with code -1073741515: Failed to attach to the console (see the AttachConsole WinAPI call). error=0 at winp.cpp:59

at org.jvnet.winp.Native.sendCtrlC(Native Method)
at org.jvnet.winp.Native.sendCtrlC(Native.java:90)
at org.jvnet.winp.WinProcess.sendCtrlC(WinProcess.java:93)
at hudson.util.ProcessTree$WindowsOSProcess.killSoftly(ProcessTree.java:538)
at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:517)
at hudson.util.ProcessTree.killAll(ProcessTree.java:168)
at hudson.Proc$LocalProc.destroy(Proc.java:384)
at hudson.Proc$LocalProc.join(Proc.java:357)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)

martin.danjou14@gmail.com (JIRA)

unread,

Nov 27, 2018, 8:50:37 AM11/27/18

to jenkinsc...@googlegroups.com

Martin d'Anjou commented on

The TERM signal is sent when the job runs on the Jenkins master, but when it runs on a node, the signal is not received (or not sent?).

martin.danjou14@gmail.com (JIRA)

unread,

Nov 27, 2018, 8:50:45 AM11/27/18

to jenkinsc...@googlegroups.com

Martin d'Anjou edited a comment on

The TERM signal is sent trapped by the freestyle script when the job runs on the Jenkins master, but when it runs on a node, the signal is not received (or not sent?).

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 7, 2019, 12:10:02 PM1/7/19

to jenkinsc...@googlegroups.com

Karl Parry commented on

We are experiencing an issue that seems to be related to Martin d'Anjou comment. A job is recieving the TERM signal but the node is still running the job and even triggering "child" jobs without ever reporting these jobs back to master.

We are running these jobs via an SSH slave plugin using Jenkins Pipeline scripts (running Jenkins 2.150.1 LTS and ssh slave plugin 1.29.1)

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 7, 2019, 12:16:03 PM1/7/19

to jenkinsc...@googlegroups.com

Karl Parry edited a comment on

We are experiencing an issue that seems to be related to [~deepchip] comment. A job is recieving the TERM signal but the node is still running the job and even triggering "child" jobs without ever reporting these jobs back to master.

We are running these jobs via an SSH slave plugin using Jenkins Pipeline scripts (running Jenkins 2.150.1 LTS and ssh slave plugin 1.29.1)

{color:#FF0000}_additional edit_{color}: noticed my SSH Slave plugin is out of date - I will update to 1.29.4 tomorrow in case this is the cause but based on changelog doesn't seem to relate.

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 7, 2019, 12:17:03 PM1/7/19

to jenkinsc...@googlegroups.com

Karl Parry edited a comment on

We are experiencing an issue that seems to be related to [~deepchip] comment. A job is recieving the TERM signal but the node is still running the job and even triggering "child" jobs without ever reporting these jobs back to master.

We are running these jobs via an SSH slave plugin using Jenkins Pipeline scripts (running Jenkins 2.150.1 LTS and ssh slave plugin 1.29.1)

{color:# FF0000 ff0000 }_additional edit_{color}: noticed my SSH Slave plugin is out of date - I will update to 1.29.4 tomorrow in case this is the cause but based on changelog doesn't seem to relate.

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 7, 2019, 12:28:04 PM1/7/19

to jenkinsc...@googlegroups.com

Karl Parry edited a comment on

We are experiencing an issue that seems to be related to [~deepchip] comment. A job is recieving the TERM signal but the node is still running the job and even triggering "child" jobs without ever reporting these jobs back to master.

We are running these jobs via an SSH slave plugin using Jenkins Pipeline scripts (running Jenkins 2.150.1 LTS and ssh slave plugin 1.29.1)

{color:#ff0000}_additional edit_{color}: noticed my SSH Slave plugin is out of date - I will update to 1.29.4 tomorrow in case this is the cause but based on changelog doesn't seem to relate.

{color:#0747a6}Additional details{color}: The job we noticed this behaviour on was an application ant build that was stopped just after triggering from a GitHub Enterprise Push Event. That job was stopped, then re-triggered using different parameters. The re-trigger seemed to complete then the first job re-triggered but there is no record of the re-trigger in Jenkins build history of either the main build job or the child job that was triggered. The only reason we know it triggered is the artifact was uploaded to our artifact server (sonatype nexus) and notified our tracking system (its a curl command in the child job it triggers).

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 8, 2019, 3:36:05 AM1/8/19

to jenkinsc...@googlegroups.com

Karl Parry edited a comment on

We are experiencing an Final Edit - our issue that seems to be was identified not related to [~deepchip] comment. A job is recieving the TERM signal kill but the node is still running the job and even a rogue jenkins triggering "child" jobs without ever reporting these jobs back to master.

We are running these jobs via an SSH slave plugin using Jenkins Pipeline scripts (running Jenkins 2.150.1 LTS and ssh slave plugin 1.29.1)

{color:#ff0000}_additional edit_{color}: noticed my SSH Slave plugin is out of date - I will update to 1.29.4 tomorrow in case this is the cause but based on changelog doesn't seem to relate.

{color:#0747a6}Additional details{color}: The job we noticed this behaviour on was an application ant build that was stopped just after triggering from a GitHub Enterprise Push Event. That duplicate job was stopped, then re-triggered using different parameters. The re-trigger seemed to complete then the first job re-triggered but there is no record of the re-trigger in Jenkins build history of either the main build job or the child job that was triggered. The only reason we know it triggered is the artifact was uploaded to our artifact server (sonatype nexus) and notified our tracking system (its a curl command in the child job it triggers).

karl.dean.parry@gmail.com (JIRA)

unread,

Jan 8, 2019, 3:36:14 AM1/8/19

to jenkinsc...@googlegroups.com

Karl Parry updated an issue

Jenkins /

Change By:	Karl Parry
Comment:	Final Edit - our issue was identified not related to the TERM kill but a rogue jenkins triggering a duplicate job