Capturing output of an aborting build?

9 views
Skip to first unread message

Stephan Bergmann

unread,
Apr 3, 2020, 8:27:04 AM4/3/20
to jenkins...@googlegroups.com
LibreOffice's Jenkins instance at <https://ci.libreoffice.org/> uses
<https://plugins.jenkins.io/build-timeout/> to abort hanging builds.

At least on Linux, the actual build is wrapped in a kill-wrapper process
to make sure that Jenkins' attempt at terminating a to-be-aborted build
reliably kills all processes spawned during that build. (See
<https://git.libreoffice.org/lode/+/0da57d7ad08a7ad7a6916a32a1333cfa8be05265>
"Make sure tb_slave_wrapper kills all spawned processes". The
kill-wrapper spawns the actual build in a new process group, and when it
receives SIGTERM from Jenkins it sends SIGKILL to that whole process group.)

Now, I thought it would be a good idea if the kill-wrapper could print
some information about any remaining process before it kills them. That
way, we could get a clue as to why a build hung and had to be aborted by
Jenkins. (See
<https://git.libreoffice.org/lode/+/bded43937c6efc82efc5924820a281c8a1ead5ba>
"kill-wrapper: pstree of hung processes".)

However, this does not work as intended, and the pstree output is not
captured in the build's console output (see e.g.
<https://ci.libreoffice.org/job/gerrit_linux_clang_dbgutil/56251/console>
which merely ends with

> Build timed out (after 15 minutes). Marking the build as aborted.
> Build was aborted
> Finished: ABORTED

for a forcefully hung build). I assume that the timing inside Jenkins
is such that aborting a build causes the capturing of the build's
stdout/-err output to stop before the spawned process has terminated.
But I did not get very far trying to understand what happens by reading
the Jenkins source code.

Is there a known solution to this problem (by changing the configuration
of the job, say, or by modifying the Jenkins source code)?

Reply all
Reply to author
Forward
0 new messages