[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step

Issue Type:	Bug
Assignee:	Unassigned
Components:	core
Created:	2019-08-30 16:04
Environment:	Windows Jenkins 2.176.1
Priority:	Major
Reporter:	Marat Radchenko

Windows
Jenkins 2.176.1

Create pipeline:

 
                                                                        node() {
  bat "ping 127.0.0.1 -n 100000"
}

Run pipeline
Abort pipeline
View build log

Expected: pipeline aborts fast and without any issues

Actual (66% reproducibility):

It takes pipeline 20s to abort
Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline
"Click here to forcibly terminate running steps" link is still visible even after the build has finished

Issue analysis:

There is a race condition between 2 minute timer in hudson.util.ProcessTree.WindowsOSProcess#killSoftly introduced for JENKINS-17116 and 20s timer in org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.Execution#stop. It is possible for DurableTaskStep to pretend that step was cancelled while it fact process is still running. Because of this race condition, it is possible to trick Jenkins into thinking that build has finished while if fact there are still processes running in workspace and potentially locking files there (this happens to us in practice).
org.jvnet.winp.WinProcess#sendCtrlC that is used in hudson.util.ProcessTree.WindowsOSProcess#killSoftly is NOT a proper way to terminate processes. Many apps do not interpret CTRL+C as a shutdown signal. (cmd.exe being the most important one here, because running bat in pipeline involved TWO cmd.exe - one running jenkins-wrapper.bat and second running jenkins-main.bat. Why you're not using TerminateProcess function from WinAPI?
There's a race condition between gathering of process list in hudson.util.ProcessTree.Windows#Windows constructor and killing of the processes, during which build can produce new processes that will not be attempted to be killed.
Usage of JENKINS_NODE_COOKIE to find what processes to kill is unreliable because 1) processes are free to alter their environment 2) CreateProcessA allows to pass custom environment variables 3) It has unpredictable order 4) It doesn't match Jenkins behavior on Linux

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

marat@slonopotamus.org (JIRA)

unread,

Aug 30, 2019, 12:07:02 PM8/30/19

to jenkinsc...@googlegroups.com

Marat Radchenko updated an issue

Jenkins /

Change By:	Marat Radchenko

# Windows
# Jenkins 2.176.1
# Create pipeline:
{code}node() {

bat "ping 127.0.0.1 -n 100000"
}

{code}
# Run pipeline
# Abort pipeline
# View build log

Expected: pipeline aborts fast and without any issues

Actual (66% reproducibility):

# It takes pipeline 20s to abort
# Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline
# "Click here to forcibly terminate running steps" link is still visible even after the build has finished

Issue analysis:
# There is a race condition between 2 minute timer in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} introduced for JENKINS-17116 by [PR#3414|https://github.com/jenkinsci/jenkins/pull/3414] and 20s timer in {{org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.Execution#stop}}. It is possible for {{DurableTaskStep}} to pretend that step was cancelled while it fact process is still running. Because of this race condition, it is possible to trick Jenkins into thinking that build has finished while if fact there are still processes running in workspace and potentially locking files there (this happens to us in practice).
# {{org.jvnet.winp.WinProcess#sendCtrlC}} that is used in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} is NOT a proper way to terminate processes. Many apps do not interpret CTRL+C as a shutdown signal. ({{cmd.exe}} being the most important one here, because running {{bat}} in pipeline involved TWO {{cmd.exe}} - one running {{jenkins-wrapper.bat}} and second running {{jenkins-main.bat}}. Why you're not using [TerminateProcess function|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess] from WinAPI?
# There's a race condition between gathering of process list in {{hudson.util.ProcessTree.Windows#Windows}} constructor and killing of the processes, during which build can produce new processes that will not be attempted to be killed.
# Usage of {{JENKINS_NODE_COOKIE}} to find what processes to kill is unreliable because 1) processes are free to alter their environment 2) [CreateProcessA|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa] allows to pass custom environment variables 3) It has unpredictable order 4) It doesn't match Jenkins behavior on Linux

marat@slonopotamus.org (JIRA)

unread,

Sep 3, 2019, 11:04:01 AM9/3/19

to jenkinsc...@googlegroups.com

Marat Radchenko updated an issue

Jenkins /

Change By:	Marat Radchenko
Component/s:	durable-task-plugin

marat@slonopotamus.org (JIRA)

unread,

Sep 23, 2019, 8:40:03 AM9/23/19

to jenkinsc...@googlegroups.com

Marat Radchenko commented on

Re: Jenkins fails to properly abort "bat" step

All that I described in this issue can be reproduced by running org.jenkinsci.plugins.workflow.steps.durable_task.ShellStepTest#abort test on Windows. Sometimes it quickly passes. Sometimes it idles with 20s timeout. Sometimes it fails to kill ping process.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

marat@slonopotamus.org (JIRA)

unread,

Sep 23, 2019, 9:35:01 AM9/23/19

to jenkinsc...@googlegroups.com

Marat Radchenko updated an issue

Jenkins /

Change By:	Marat Radchenko
Component/s:	workflow-durable-task-step-plugin
Component/s:	core
Component/s:	durable-task-plugin

marat@slonopotamus.org (JIRA)

unread,

Sep 23, 2019, 12:08:02 PM9/23/19

to jenkinsc...@googlegroups.com

Marat Radchenko updated an issue

Jenkins /

Change By:	Marat Radchenko

# Windows
# Jenkins 2.176.1
# Create pipeline:
{code}node() {
bat "ping 127.0.0.1 -n 100000"
}
{code}
# Run pipeline
# Abort pipeline
# View build log

Expected: pipeline aborts fast and without any issues

Actual ( 66% reproducibility is less than 100% ):

# It takes pipeline 20s to abort
# Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline
# "Click here to forcibly terminate running steps" link is still visible even after the build has finished

# Sometimes ping processes are NOT terminated even when build has aborted.

Issue analysis:
# There is a race condition between 2 minute timer in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} introduced for JENKINS-17116 by [PR#3414|https://github.com/jenkinsci/jenkins/pull/3414] and 20s timer in {{org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.Execution#stop}}. It is possible for {{DurableTaskStep}} to pretend that step was cancelled while it fact process is still running. Because of this race condition, it is possible to trick Jenkins into thinking that build has finished while if fact there are still processes running in workspace and potentially locking files there (this happens to us in practice).
# {{org.jvnet.winp.WinProcess#sendCtrlC}} that is used in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} is NOT a proper way to terminate processes. Many apps do not interpret CTRL+C as a shutdown signal. ({{cmd.exe}} being the most important one here, because running {{bat}} in pipeline involved TWO {{cmd.exe}} - one running {{jenkins-wrapper.bat}} and second running {{jenkins-main.bat}}. Why you're not using [TerminateProcess function|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess] from WinAPI?
# There's a race condition between gathering of process list in {{hudson.util.ProcessTree.Windows#Windows}} constructor and killing of the processes, during which build can produce new processes that will not be attempted to be killed.
# Usage of {{JENKINS_NODE_COOKIE}} to find what processes to kill is unreliable because 1) processes are free to alter their environment 2) [CreateProcessA|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa] allows to pass custom environment variables 3) It has unpredictable order 4) It doesn't match Jenkins behavior on Linux

marat@slonopotamus.org (JIRA)

unread,

Sep 24, 2019, 10:27:02 AM9/24/19

to jenkinsc...@googlegroups.com

Marat Radchenko commented on

Re: Jenkins fails to properly abort "bat" step

See comments to PR#4216 for additional technical analysis.

o.v.nenashev@gmail.com (JIRA)

unread,

Oct 7, 2019, 8:15:02 PM10/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated an issue

Jenkins /

Change By:	Oleg Nenashev
Component/s:	core

o.v.nenashev@gmail.com (JIRA)

unread,

Oct 7, 2019, 8:15:03 PM10/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated

The fix was released in Jenkins 2.199

Jenkins /

Change By:	Oleg Nenashev
Status:	Open Fixed but Unreleased
Resolution:	Fixed
Released As:	Jenkins 2.199

o.v.nenashev@gmail.com (JIRA)

unread,

Oct 7, 2019, 8:15:04 PM10/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated

Jenkins /

Change By:	Oleg Nenashev
Status:	Fixed but Unreleased Resolved

o.v.nenashev@gmail.com (JIRA)

unread,

Oct 7, 2019, 8:16:03 PM10/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated an issue

Jenkins /

Change By:	Oleg Nenashev
Labels:	lts-candidate

marat@slonopotamus.org (JIRA)

unread,

Oct 8, 2019, 2:13:02 AM10/8/19

to jenkinsc...@googlegroups.com

Marat Radchenko commented on