Jenkins docker-workflow plugin - docker container killed prematurely while using ".inside"

[Mike Wilkerson]

Hi all—

I'm trying to run steps inside a docker container via .inside, but the container is killed shortly after running the commands. It appears that the docker-workflow plugin is not waiting for the exec command behind the scenes, and is stopping the container prematurely.

A little context, first:

• Jenkins slave is running on a CentOS Linux release 7.1.1503 VM.
• Using the official debian:7 docker image.
• Docker version 1.9.1, build a34a1d5
  
• Jenkins 1.639
• This might be unrelated, but it's probably worth mentioning:
• The default CentOS Docker install was running the daemon like /usr/bin/docker daemon -H fd://
• I wasn't able to get the docker-workflow plugin to connect to the daemon with that configuration, so I modified the service file to instead run as /usr/bin/docker daemon -H tcp://localhost:2375
• I could then successfully use the plugin by running inside a docker.withServer('tcp://localhost:2375', '') { ... } block.

Reproducing the issue

Anything running in an sh block that finishes very quickly is fine:

node('docker') {

    docker.withServer('tcp://localhost:2375', '') {

        docker.image('debian:7').inside {

            sh 'printenv'

        }

    }

}

However, a longer-running command will always fail (Jenkins build fails, complaining that the script exited with error code -1):

node('docker') {

    docker.withServer('tcp://localhost:2375', '') {

        docker.image('debian:7').inside {

            sh 'sleep 300'

        }

    }

}

Here are some interesting lines from /var/log/messages:

Jan 11 15:25:24 jenkins-build-centos7 docker: time="2016-01-11T15:25:24.290960195-05:00" level=info msg="POST /v1.21/containers/02b6dbc8af81794efe1aeeb43f861e47a803bb3672a9991a63120632127df247/exec"

Jan 11 15:25:24 jenkins-build-centos7 docker: time="2016-01-11T15:25:24.293203526-05:00" level=info msg="POST /v1.21/exec/39fd717f8dd9797e9a967edc995e0f0e29ed19bcd1837e28410dcd0667f56253/start"

Jan 11 15:25:25 jenkins-build-centos7 docker: time="2016-01-11T15:25:25.350407441-05:00" level=info msg="POST /v1.21/containers/02b6dbc8af81794efe1aeeb43f861e47a803bb3672a9991a63120632127df247/stop?t=10"

Jan 11 15:25:35 jenkins-build-centos7 docker: time="2016-01-11T15:25:35.351254649-05:00" level=info msg="Container 02b6dbc8af81794efe1aeeb43f861e47a803bb3672a9991a63120632127df247 failed to exit within 10 seconds of SIGTERM - using the force"

Jan 11 15:25:35 jenkins-build-centos7 docker: time="2016-01-11T15:25:35.359447435-05:00" level=info msg="GET /v1.21/exec/39fd717f8dd9797e9a967edc995e0f0e29ed19bcd1837e28410dcd0667f56253/json"

It looks like the docker-workflow plugin is immediately issuing a stop call to the API, right after the exec call.

Attempt at a solution

I noticed that the docker-workflow plugin calls the image by passing the cat command:

docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/docker-test -v /home/jenkins/workspace/docker-test:/home/jenkins/workspace/docker-test:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** debian:7 cat

...which would cause the container to run cat and then immediately exit. I created a simple dockerfile, based off of the debian:7 image, and set the ENTRYPOINT to ["ping", "localhost"], just to have a process that would run indefinitely, even if docker-workflow supplied cat as the command to docker run. Unfortunately this changed nothing, and I still noticed the same API call to stop in /var/log/messages.

I'm at a loss here. Anyone seen anything like this before?

Thanks,

Mike Wilkerson

[Mike Wilkerson]

Slight correction—Looks like cat will keep the docker container running as long as the -i or -t flag is passed.