[JIRA] (JENKINS-37769) Too long stop docker container(default docker stop timeout=10s)

217 views
Skip to first unread message

trex.boolat@gmail.com (JIRA)

unread,
Aug 29, 2016, 12:13:03 PM8/29/16
to jenkinsc...@googlegroups.com
Oleksii Trekhov updated an issue
 
Jenkins / Improvement JENKINS-37769
Too long stop docker container(default docker stop timeout=10s)
Change By: Oleksii Trekhov
Summary: Stop Too long stop docker container too long (default docker stop timeout=10s)
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

trex.boolat@gmail.com (JIRA)

unread,
Aug 30, 2016, 4:49:02 AM8/30/16
to jenkinsc...@googlegroups.com
Oleksii Trekhov commented on Improvement JENKINS-37769
 
Re: Too long stop docker container(default docker stop timeout=10s)

Update - i use last docker v1.12 and Ubuntu 16.04 x64 LTS

jglick@cloudbees.com (JIRA)

unread,
Aug 31, 2016, 1:59:01 PM8/31/16
to jenkinsc...@googlegroups.com

If stop is hitting the default 10s timeout, something deeper is wrong. I doubt that simply passing --time 0 is correcting that.

gabealmer@gmail.com (JIRA)

unread,
Sep 1, 2016, 4:32:03 PM9/1/16
to jenkinsc...@googlegroups.com

Do you really need that cat to run?

$ time docker run -t -d --name test ubuntu
d0b48a93c51f4d56a5bcb206bb2cad24cdf16546118a320c2201ee13f068e218

real    0m0.376s
user    0m0.008s
sys     0m0.009s

And it is running

$ docker ps
CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS              PORTS                                                          NAMES
d0b48a93c51f        ubuntu                                     "/bin/bash"              8 seconds ago       Up 7 seconds                                                                       test

And if a stop it, it runs almost right away:

$ time docker stop test
test

real    0m0.325s
user    0m0.007s
sys     0m0.007s

So can we skip passing cat as command when running a container?

bart.vanbrabant@inmanta.com (JIRA)

unread,
Sep 6, 2016, 3:44:02 AM9/6/16
to jenkinsc...@googlegroups.com

I am seeing a similar problem on a CentOS with Jenkins 2.21 and docker 1.12.1. When I use a pipline with an inside shell command, the container is stopped after +/-10s. In a simple script like this:

node {
    stage('Checkout') {
        checkout scm
    }

    stage('Unit Tests') {
        img = docker.image("fedora:24")
        img.inside {
            sh "sleep 120"
        }
    }
}

The unit tests stage only runs for +/- 10s instead of 120s

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 4:38:04 PM9/12/16
to jenkinsc...@googlegroups.com

Oleksii Trekhov I can reproduce the pause. I am still trying to determine why it is pausing. From what I can tell so far, the container is killed right away (presumably after the SIGTERM), so I am wondering why it is then waiting for the timeout to expire, supposedly to send a SIGKILL. Perhaps this is a bug in Docker. Needs more investigation.

Gabor Almer

can we skip passing cat as command when running a container?

The problem is determining which images that would apply to.

Bart Vanbrabant your problem seems to be unrelated, far more serious, and specific to the fedora image (perhaps). I think it has something to do with different procfs handling. Please file it separately.

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 7:22:01 PM9/12/16
to jenkinsc...@googlegroups.com

At first I thought it was this problem with systemd, but --stop-signal=RTMIN+3 does not work. Anyway the overridden --entrypoint means that systemd is not in play.

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 7:36:01 PM9/12/16
to jenkinsc...@googlegroups.com

Not following what is going on. If you run

bash -c 'trap "echo term" EXIT; sleep 120'

and then kill this bash process from another terminal window, you see

term
Terminated

immediately. But if you run

docker run --rm --name test ubuntu bash -c 'trap "echo term" EXIT; sleep 120'

and from another window

docker stop test

you will see

term

and then ten seconds will elapse before the container exits. Somehow the TERM signal is getting sent, but not properly handled. (sleep is still running during this time.)

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 8:01:01 PM9/12/16
to jenkinsc...@googlegroups.com

Stranger still:

docker run --rm --name test ubuntu sleep infinity

Now docker exec test ps fauxwww confirms that sleep is PID 1, yet docker exec test kill -9 1 does not do anything. docker stop test will pause for ten seconds. I suspect that somehow signal delivery is just getting blocked to PID 1. On the other hand, with

docker run --rm --name test --stop-signal=KILL ubuntu sleep infinity

you will see that docker stop test works promptly. Perhaps in this case the stop command is just being clever and knows that it makes no sense to wait after sending SIGKILL, so behaves like --time=0?

Really not sure what is going on. Passing --time 0 does correct the symptom, without addressing the underlying problem. But it is less than clear that a clean TERM is being delivered to any spawned processes even in the current state—it seems that TERM is delivered to the entrypoint ((cat) but does nothing.

I am (dimly) aware of zombie reaping issues but that does not seem to be relevant in this case.

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 8:18:02 PM9/12/16
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Progress Review

jglick@cloudbees.com (JIRA)

unread,
Sep 12, 2016, 8:18:02 PM9/12/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Improvement JENKINS-37769
 
Change By: Jesse Glick
Status: Open In Progress

scm_issue_link@java.net (JIRA)

unread,
Oct 5, 2016, 10:32:02 AM10/5/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon commented on Improvement JENKINS-37769
 
Re: Too long stop docker container(default docker stop timeout=10s)

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java
src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java
http://jenkins-ci.org/commit/docker-workflow-plugin/0fed7a702751f743eb3603092e1731f8979930a5
Log:
[FIXED JENKINS-37769] Failing to stop containers cleanly, so at least stopping them quickly.
Reply all
Reply to author
Forward
0 new messages