[JIRA] (JENKINS-53315) Timeout step should support a closure to execute prior to killing body

75 views
Skip to first unread message

regs@akom.net (JIRA)

unread,
Aug 29, 2018, 8:02:02 AM8/29/18
to jenkinsc...@googlegroups.com
Alexander Komarov created an issue
 
Jenkins / Improvement JENKINS-53315
Timeout step should support a closure to execute prior to killing body
Issue Type: Improvement Improvement
Assignee: Unassigned
Components: workflow-basic-steps-plugin
Created: 2018-08-29 12:01
Priority: Minor Minor
Reporter: Alexander Komarov

Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it difficult to perform any automated debugging on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.

Currently:

 

try {
  timeout(time: 1, unit: 'HOURS') {
     sh "java IntermittentlySlowProcess"
  }
} catch (t) {
    //It's too late to, for example, send a "kill -3" to the slow/hung java process
}

 

What I'd propose

(and I'm willing to try to make a PR if this seems reasonable):

 

timeout(time: 1, unit: 'HOURS', beforeKill: {
   sh "killall -3 java" //for example
}) {
   sh "java IntermittentlySlowProcess"
}

 

 

"beforeKill" can be used for clean shutdown of complex tasks, analysis of problems, etc.

Thoughts welcome.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

regs@akom.net (JIRA)

unread,
Aug 29, 2018, 8:03:01 AM8/29/18
to jenkinsc...@googlegroups.com
Alexander Komarov updated an issue
Change By: Alexander Komarov
Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 


You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
h3. Currently:

 
{code:java}

try {
  timeout(time: 1, unit: 'HOURS') {
     sh "java IntermittentlySlowProcess"
  }
} catch (t) {
    //It's too late to, for example, send a "kill -3" to the slow/hung java process
}
{code}
 
h3. What I'd propose


(and I'm willing to try to make a PR if this seems reasonable):

 
 {code:java}

timeout(time: 1, unit: 'HOURS', beforeKill: {
   sh "killall -3 java" //for example
}) {
   sh "java IntermittentlySlowProcess"
}
{code}
 The new  

 

 "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

Thoughts welcome.

regs@akom.net (JIRA)

unread,
Aug 29, 2018, 8:03:02 AM8/29/18
to jenkinsc...@googlegroups.com

regs@akom.net (JIRA)

unread,
Aug 29, 2018, 12:01:02 PM8/29/18
to jenkinsc...@googlegroups.com
Alexander Komarov updated an issue
Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 


You can argue that this should  It would be done outside nice to be able to get some information about the state of Jenkins affairs before things are killed , but, too often, Jenkins is the only place this is visible due and maybe even perform safe shutdown steps prior to frequency of execution or differences in the environment kill .

h3. Currently: 
{code:java}
try {
  timeout(time: 1, unit: 'HOURS') {
     sh "java IntermittentlySlowProcess"
  }
} catch (t) {
    //It's too late to, for example, send a "kill -3" to the slow/hung java process
}
{code}
h3. What I'd propose

(and I'm willing to try to make a PR if this seems reasonable):
{code:java}
timeout(time: 1, unit: 'HOURS', beforeKill: {
   sh "killall -3 java" //for example
}) {
   sh "java IntermittentlySlowProcess"
}
{code}
The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

One workaround may be to wrap whatever you are running and trap signals, but that's ugly and error-prone (and will likely cause zombies)

 Thoughts welcome.

 

 

regs@akom.net (JIRA)

unread,
Oct 5, 2018, 1:19:02 PM10/5/18
to jenkinsc...@googlegroups.com
Alexander Komarov commented on Improvement JENKINS-53315
 
Re: Timeout step should support a closure to execute prior to killing body

Jesse Glick can I get some pointers on how to implement this?

I've tried using workflow-cps-plugin (ParallelStep) for guidance, but I don't seem to be able to figure it out.  It's trivial to implement a step with a body, but how does a step take a second body as a parameter and then properly invoke it? 

I've tried making the parameter a Closure, a CpsClosure(2), etc.  None of these options seem right.  Do I need to implement a custom context and something similar to newBodyInvoker(BodyReference)? Then how do I convert my Closure to a BodyReference?

Anyway, any help would be appreciated (or a reason why this is a dumb idea).

jglick@cloudbees.com (JIRA)

unread,
Oct 5, 2018, 1:41:01 PM10/5/18
to jenkinsc...@googlegroups.com

Before you dive into things which are somewhere between ridiculously hard and perhaps impossible—why are you doing this? If you just want a thread dump from a Java program before it gets killed, you can do that with a few lines of Bash, with no changes to your Pipeline script or special Jenkins knowledge. trap TERM and kill -QUIT the Java PID before killing it for good.

regs@akom.net (JIRA)

unread,
Oct 5, 2018, 3:49:01 PM10/5/18
to jenkinsc...@googlegroups.com

Jesse Glick thanks for the feedback.  I have thought of using a trap wrapper but it just seemed like an additional layer of hackery and possible future debugging effort (it's also not cross-platform).  Perhaps I can wrap it in a groovy script or java instead, or something along these lines.

Since you say that it's too hard to do what I planned, I'll hold off.  Feel free to close, although I'm curious if anyone else is interested in the same functionality (doesn't seem like it so far).

regs@akom.net (JIRA)

unread,
Oct 5, 2018, 5:25:02 PM10/5/18
to jenkinsc...@googlegroups.com

Also, it appears to be impossible to trap the SIGTERM that Jenkins sends using bash (the trap works, but the subprocess, eg gradle, is simultaneously killed anyway).

jglick@cloudbees.com (JIRA)

unread,
Oct 5, 2018, 6:05:03 PM10/5/18
to jenkinsc...@googlegroups.com

Well, it is cross-platform to the extent that the Bourne shell is. I do not know offhand what the Powershell equivalent would be.

Not sure why you would be having an issue with the subprocess being killed. Try unsetting the JENKINS_SERVER_COOKIE environment variable and see if that helps. It should not be necessary because of this, but it has been a long time since I wrote this.

regs@akom.net (JIRA)

unread,
Oct 5, 2018, 7:51:03 PM10/5/18
to jenkinsc...@googlegroups.com

It appears to be the way Bash works.  The very first example here shows that although it is possible to trap the signal and prevent the bash shell from exiting, it is not possible to prevent the currently executing shell command (sleep in this case) from also receiving the signal, hence the while loop.  

I was able to create a groovy wrapper script using the approach suggested here and that works correctly (both options in this answer work).

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 8:20:02 AM10/8/18
to jenkinsc...@googlegroups.com

Summary of findings:

I was able to work around this using a few different methods:

  • A shell script that traps and cleans up before kill
  • A groovy script that traps and cleans up before kill
  • A pure pipeline DSL block that uses parallel closure to do the same without an external script (at the cost of uglier output)

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 8:22:03 AM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov closed an issue as Won't Do
 
Change By: Alexander Komarov
Status: Open Closed
Resolution: Won't Do

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 8:22:04 AM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov edited a comment on Improvement JENKINS-53315
 
Re: Timeout step should support a closure to execute prior to killing body
h3. Summary of findings:

I was able to work around this using [a few different methods|https://github.com/akomakom/jenkins-scripts/tree/master/pipeline-timeout-prekill]:
* A * shell script * that traps and cleans up before kill
* A * groovy script * that traps and cleans up before kill
* A pure pipeline * DSL block script* that uses parallel closure to do the same without an external script (at the cost of uglier output)

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 8:23:02 AM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov edited a comment on Improvement JENKINS-53315
It appears to be the way Bash works.  The very first example [here|https://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.html] shows that although it is possible to trap the signal and prevent the bash shell from exiting, it is not possible to prevent the currently executing shell command (sleep in this case) from also receiving the signal, hence the while loop.  

I was able to create a groovy wrapper script using the approach [suggested here|https://stackoverflow.com/a/8851685/1735931] and that works correctly (both options in this answer work).

UPDATE: the trick with bash is to [background the main process|https://unix.stackexchange.com/a/146770/199293]... 

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 9:16:02 AM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov edited a comment on Improvement JENKINS-53315
h3. Summary of findings:

I was able to work around this using [a few different methods|https://github.com/akomakom/jenkins-scripts/tree/master/pipeline-timeout-prekill]:
* A *shell script* that traps and cleans up before kill (not cross-platform)
* A *groovy script* that traps and cleans up before kill  
* A pure pipeline *DSL script* that uses parallel closure to do the same without an external script (at the cost of uglier output)

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 11:40:02 AM10/8/18
to jenkinsc...@googlegroups.com

So if I follow correctly, Jenkins was behaving correctly to the extent that its behavior followed what Ctrl-C in a shell would do (send SIGTERM to subprocesses), and you were able to write a Bourne shell script with a little difficulty that intercepted this signal as sent to the subprocess and did something else before running a clean shutdown? If so, would you mind posting your script (or, better, filing it on stackoverflow.com)?

I should have mentioned earlier that Java itself has a friendlier solution to this problem: shutdown hooks. A hook could do whatever you liked, including getting all stack traces or dumping threads including more sophisticated information such as locks. With such a hook inserted into your program (if you indeed control it, or can write in-process plugins for it), there is no need for any special wrapper script.

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 11:58:01 AM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov edited a comment on Improvement JENKINS-53315
h3. Summary of findings:

I was able to work around this using [a few different methods|https://github.com/akomakom/jenkins-scripts/tree/master/pipeline-timeout-prekill]: <-- Link
* A *shell script* that traps and cleans up before kill (not cross-platform)
* A *groovy script* that traps and cleans up before kill 
* A pure pipeline *DSL script* that uses parallel closure to do the same without an external script (at the cost of uglier output)

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 11:58:02 AM10/8/18
to jenkinsc...@googlegroups.com

I'll try to make the link in my previous comment more visible.  I have 4 solutions now, with this one being the one I'm using:

https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill-pure-dsl-failfast.groovy

 

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 12:00:01 PM10/8/18
to jenkinsc...@googlegroups.com
Alexander Komarov edited a comment on Improvement JENKINS-53315
I'll try to make the link in my previous comment more visible.  I have 4 solutions now, with this one being the one I'm using:

[https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill-pure-dsl-failfast.groovy]

 The only thing I would wish to change would be to silence or reduce the   *waitUntil* output... 

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 12:00:02 PM10/8/18
to jenkinsc...@googlegroups.com

To be clear, I strongly recommend solving this at the native process level rather than with Pipeline tricks.

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 1:26:02 PM10/8/18
to jenkinsc...@googlegroups.com

Why, specifically?  Is the DSL behavior expected to change?

They all have downsides over the built-in that this ticket originally requests:

  • bash script won't work in Windows.  
  • groovy script fixes that, but both make quoting for a complex sub-process invocation very hard to understand (eg 'bash -c "sleep 5"') because they have to accept the commands as parameters.
  • pipeline tricks don't have either limitation, but they do add unnecessary verbosity to the output.

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 2:11:02 PM10/8/18
to jenkinsc...@googlegroups.com

Bourne shell scripts will indeed not be available in most Windows flavors, but I assume an analogous feature is easy to write in Powershell. Or, again, depending on the Java program in question, you can intercept SIGTERM in a cross-platform manner using Java APIs. In fact, at least on Linux, you can dispense with the timeout step altogether!

sh 'timeout 11 timeout -sQUIT 10 java -jar …/jdk5/demo/jfc/SwingSet2/SwingSet2.jar'

As to why you should always prefer native solutions to Pipeline script tricks, this is a subject for innumerable conference talks. Just trust me that you are better off keeping the Pipeline script to the bare minimum necessary to orchestrate actions on the Jenkins service, boiling your actual build down to a sh './run' if at all possible.

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 3:27:02 PM10/8/18
to jenkinsc...@googlegroups.com

Jesse Glick at your suggestion I switched from using pipeline tricks to the bash script (which I tested only locally): https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill.sh

It does not work in Jenkins.  It works fine in local testing (pkill --signal SIGTERM xxxx or Ctrl-C) but not in a Jenkins timeout step - the trap never runs, termination is instant.

This leads me to believe that Jenkins actually sends a SIGTERM to the whole process tree right away.

In other words, there is still no good solution aside for pipeline tricks.

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 3:45:02 PM10/8/18
to jenkinsc...@googlegroups.com

Interestingly, when I use the linux "timeout" that you mention instead of the timeout step, traps work fine... So the linux solution is a combination of the shell script and timeout:

timeout 10 bash pipeline-timeout-prekill.sh './gradlew ....'

Which is a lot of parts...  but OK - now I'll just need to find an equivalent for windows (we do cross-platform testing on many platforms, that's why I wanted to solve this in pipeline code).

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 3:47:02 PM10/8/18
to jenkinsc...@googlegroups.com

As I said before, yes Jenkins might be sending SIGTERM to the whole process tree, not only the entry script. This might be worked around (untested) via

JENKINS_SERVER_COOKIE=suppress java -jar …

since it uses this environment variable to identify some processes. I have forgotten the details at this point. (JENKINS-28182)

The “good solutions” are

  • Java shutdown hooks
  • using /usr/bin/timeout

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 3:49:01 PM10/8/18
to jenkinsc...@googlegroups.com

And no you do not need a separate pipeline-timeout-prekill.sh if you are using /usr/bin/timeout. Look at my example again. That one-liner sends SIGQUIT after ten seconds, then waits one more second for the thread dump to appear, and sends a SIGTERM.

regs@akom.net (JIRA)

unread,
Oct 8, 2018, 3:59:01 PM10/8/18
to jenkinsc...@googlegroups.com

Thanks, I did see that, but a "kill -3" on the main gradle process isn't going to help me - I need to get the thread dump from the hung tests which are running a separate process launched by gradle.  My current approach is to run "jstack" on every java process on the slave (we have a single executor policy)

I also just tried setting JENKINS_SERVER_COOKIE on the child process and that doesn't help with killing subprocesses, but I haven't had time to research further.  /usr/bin/timeout does work in combination with my trap script.

jglick@cloudbees.com (JIRA)

unread,
Oct 8, 2018, 4:12:03 PM10/8/18
to jenkinsc...@googlegroups.com

get the thread dump from the hung tests which are running a separate process launched by gradle

If that is all you wanted, you may be barking up the wrong tree. The JUnit Timeout rule, for example, applies a per-test-case timeout (which is likely to be more robust and easier to manage that a per-build timeout) and automatically displays a thread dump for hung tests.

regs@akom.net (JIRA)

unread,
Jun 5, 2019, 10:41:02 AM6/5/19
to jenkinsc...@googlegroups.com

Sadly that's not all I'm after. In the case of complex integration tests, the tests may be hanging due to a hung external process (ie a server) started as part of the test. That is the process I need the stacktrace for, not the test. Otherwise, Timeout rule would work great.

jglick@cloudbees.com (JIRA)

unread,
Jun 5, 2019, 12:09:03 PM6/5/19
to jenkinsc...@googlegroups.com

In the case of JUnit, you would probably want to write a simple TestRule (if one does not already exist) which is responsible for launching the external process, sending it SIGQUIT if the test fails, and finally sending in SIGTERM in all cases.

To your original proposal, if you really have to do this inside Pipeline script for some reason, try something like

timeout(time: 1, unit: 'HOURS') {
  def ok = false
  try {
    sh 'java IntermittentlySlowProcess'
    ok = true
  } finally {
    if (!ok) {
      sh 'killall -3 java'
    }
  }
}
Reply all
Reply to author
Forward
0 new messages