[JIRA] (JENKINS-49278) cat command in docker agents not detected correctly

[hans@starefossen.com (JIRA)]

Hans Kristian Flaatten commented on JENKINS-49278

Re: cat command in docker agents not detected correctly JENKINS-41316 is the root cause of this error. Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[benjamin.heilbrunn@sap.com (JIRA)]

Benjamin Heilbrunn commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Nicolas De Loof, thanks for the quick fix!

Add Comment

[vwfoxguru@gmail.com (JIRA)]

Scott Williams commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

I suspect JENKINS-49385 is a duplicate issue, but it is not resolved with the 15.1 update.  The cat command still instantly fails with an argument delimited entrypoint in the Dockerfile (see comments therein).

Add Comment

[vwfoxguru@gmail.com (JIRA)]

Scott Williams commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Note that with an argument delimited entrypoint, the command given (either via CMD or docker run command) will be appended after the entrypoint.

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright assigned an issue to Chisel Wright

Jenkins / JENKINS-49278

cat command in docker agents not detected correctly

Change By: Chisel Wright Assignee: Nicolas De Loof Chisel Wright

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright assigned an issue to Unassigned

Jenkins / JENKINS-49278 cat command in docker agents not detected correctly

Change By: Chisel Wright Assignee: Chisel Wright

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly  I was about to type a comment, and weird things happened with focus and Jenkins shortcuts; I ended up (accidentally) assigning this to myself - apologies for that. I've set to "Unassigned".

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright assigned an issue to Nicolas De Loof

Jenkins / JENKINS-49278

cat command in docker agents not detected correctly

Change By: Chisel Wright Assignee: Nicolas De Loof

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly We're (still) seeing failures after updating to 1.15.1: $ docker run -t -d -u 497:495 -w /tmp/jenkins-prime/agent/workspace/er-G2PEKDXUWD77U -v /tmp/jenkins-prime/agent/workspace/er-G2PEKDXUWD77U:/tmp/jenkins-prime/agent/workspace/er-G2PEKDXUWD77U:rw,z -v /tmp/jenkins-prime/agent/workspace/er-G2PEKDXUWD77U@tmp:/tmp/jenkins-prime/agent/workspace/er-G2PEKDXUWD77U@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** docker.endpoint/servicectl:9.6.0 cat $ docker top b6170c275fb28d292a179ecec4d40c2452091b44074360dec0280ba08f65c3dc -eo pid,comm ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.

Add Comment

[dannyguinther@gmail.com (JIRA)]

Danny Guinther commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

I'm also still seeing this issue after upgrading to 1.15.1:   $ docker top d720fd88f4be924edf76e1d55bc058d82dce45f8cb4a34eb2db492b380fe1ab5 -eo pid,comm ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Alternatively you can force image entrypoint to be disabled by adding option `–entrypoint=''`.     Here's the result of `docker top d720fd88f4be924edf76e1d55bc058d82dce45f8cb4a34eb2db492b380fe1ab5 -eo pid,comm`     $ docker top d720fd88f4be924edf76e1d55bc058d82dce45f8cb4a34eb2db492b380fe1ab5 -eo pid,comm PID COMMAND 123094 init 123133 cat 123212 sh 123218 sh 123219 script.sh 123221 test 123252 npm 123431 sleep     `cat` is clearly in the list.   I also tried removing the `–init` option passed to the container in case `cat` needed to be first in the list, it didn't resolve the issue.    Please let me know if I can offer any other diagnostic info.

Add Comment

[michal.matyjek@gmail.com (JIRA)]

Michal Matyjek commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Seeing same as Chisel Wright and Danny Guinther - the original error message from the issue description.

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Nicolas De Loof: After upgrading to 1.15.1 I get this error in declarative pipeline: java.io.IOException: Failed to run top '788913bdd127e9689e54cf2c699b0d99cc17e7469575bd4d541b4c61394c5ce4'. Error: Error response from daemon: Unexpected pid '111969kworker/u256:2': strconv.Atoi: parsing "111969kworker/u256:2": invalid syntax at org.jenkinsci.plugins.docker.workflow.client.DockerClient.listProcess(DockerClient.java:140) at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:185)

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

[~ndeloof]: After upgrading to 1.15.1 I get this error in declarative pipeline: {noformat}

java.io.IOException: Failed to run top '788913bdd127e9689e54cf2c699b0d99cc17e7469575bd4d541b4c61394c5ce4'. Error: Error response from daemon: Unexpected pid '111969kworker/u256:2': strconv.Atoi: parsing "111969kworker/u256:2": invalid syntax at org.jenkinsci.plugins.docker.workflow.client.DockerClient.listProcess(DockerClient.java:140)

at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:185){noformat} This happens on Docker for Windows 17.12.0-ce (Git commit c97c6d6).

Add Comment

[nicolas.deloof@gmail.com (JIRA)]

Nicolas De Loof commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Thomas Johansen sounds like a docker bug, maybe watch https://github.com/moby/moby/issues/34282

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Nicolas De Loof: Yep, I upgraded Docker for Windows to 18.02, and the problem is gone. Thanks!

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

18.02? https://download.docker.com/linux/static/stable/x86_64/ has: Index of /linux/static/stable/x86_64/ ../ docker-17.03.0-ce.tgz 2017-03-01 11:11 27M docker-17.03.1-ce.tgz 2017-03-28 04:46 27M docker-17.03.2-ce.tgz 2017-06-28 03:35 27M docker-17.06.0-ce.tgz 2017-06-28 05:17 29M docker-17.06.1-ce.tgz 2017-08-18 02:35 29M docker-17.06.2-ce.tgz 2017-09-05 10:39 29M docker-17.09.0-ce.tgz 2017-09-27 01:47 29M docker-17.09.1-ce.tgz 2017-12-08 12:22 29M docker-17.12.0-ce.tgz 2017-12-27 09:52 33M

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Chisel Wright: It's from the edge channel (https://download.docker.com/linux/static/edge/x86_64/).

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Does this mean the docker-workflow-plugin has been updated to (silently?) require non-Stable versions of Docker?

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Chisel Wright: The problem seems to be isolated to Docker for Windows. I don't have the problem on 17.12 on Ubuntu.

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

My earlier comment was missing the version and OS - sorry. 17.12.0-ce on Centos/AWS (2017.09)

Add Comment

[nicolas.deloof@gmail.com (JIRA)]

Nicolas De Loof commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Chisel Wright no, this just means docker runtime has some issues, and some are fixed in recent releases. docker-workflow-plugin uses docker CLI and expect it to run by the reference documentation.

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

For the record; I also get the invalid syntax error on Docker for Windows 18.02 now. It is obviously related to the large pids bug.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Release notes ought to mention that 1.15.1 works for docker.image('maven:3.5.0').inside but not docker.image('maven:3.3.9').inside, for example. I presume that is a result of this commit by Carlos Sanchez.

Add Comment

[sachin.gupta771@gmail.com (JIRA)]

sachin gupta reopened an issue

I'm still facing the same issue with Jenkins version 2.161, I have search many issues opened and closed without any proper resolution to this problem, please provide the suitable resolution

Jenkins / JENKINS-49278

cat command in docker agents not detected correctly

Change By: sachin gupta Resolution: Fixed Status: Resolved Reopened

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly We've frozen our version of this plugin at 1.14 because of the behavioural change that broke our pipelines. (by frozen I mean, we just have to remember to deselect it when updating all the other plugins)

Add Comment

[sachin.gupta771@gmail.com (JIRA)]

sachin gupta commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Hello Chisel Wright, can you share the complete name of the plugin, Sorry I'm new to Jenkins and docker and just performing some poc for learning perspectives.

Add Comment

[chisel@chizography.net (JIRA)]

Chisel Wright commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

It's this one: https://plugins.jenkins.io/docker-workflow

Add Comment

[sachin.gupta771@gmail.com (JIRA)]

sachin gupta commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Thanks Chisel Wright, I had just downgraded the same plugin before your reply and my builds worked fine.

Add Comment

[andrew@nicols.co.uk (JIRA)]

Andrew Nicols commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

I suspect that this is the same as JENKINS-54389, and also JENKINS-39748. Any longer-running entrypoint will cause the job to fail because the cat command doesn't run until the end of the entrypoint. Additionally, requiring an image to run cat could break the image. For example, if I wish to start an image whose CMD is apache-foreground then I expect apache to be running, not cat. I'm not going to serve any web responses with cat. I think the docker-workflow plugin either needs to: respect HEALTHCHECKs allow for either no CMD override, or a specified CMD override to be provided when running the image allow for a wait period before checking that the expected command is running Basically, don't run cat by default, instead respect the healthcheck.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Any further behavioral changes in this area are likely to be incompatible and unlikely to be entertained. withDockerContainer was designed for “tool-only” images with no particular designated entrypoint (typically falling back to bash or something), since its entire purpose is to provide a passive container—basically a predefined filesystem layout—in which you can run sh steps. Images which expect to be running a specific process like Apache are not supported. You can use withRun, or (better) just run whatever docker commands you like directly.

Add Comment

[nicolas.deloof@gmail.com (JIRA)]

Nicolas De Loof assigned an issue to Unassigned

Jenkins / JENKINS-49278

cat command in docker agents not detected correctly

Change By: Nicolas De Loof Assignee: Nicolas De Loof

Add Comment

[stz.randhahn@extaccount.com (JIRA)]

Dirk D. commented on JENKINS-49278

Re: cat command in docker agents not detected correctly What about providing a additional switch like "useImageCMD" (default false) which ensures that the CMD from the image is used and is not overwritten with "cat"?

Add Comment

[boitoletre@gmail.com (JIRA)]

Ad commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Jesse Glick It is very undestandable that `withDockerContainer` is expected for tool-only images, and as a matter of fact that is exactly what our organization uses it for. Yet, such tool-only container can very easily have an ` ENTRYPOINT` which needs to completely run before the actual sh steps are executed. In our case, we have a Docker container that we use as a Conan (https://conan.io/) tool. Yet, this container has a entrypoint which runs the usual `docker-entrypoint.sh` that is notably responsible for setting up the different remote repositories and log-in to use them. Yet, we observe spurious failure because of the race condition: sometimes the entrypoint had time to complete before the sh steps are executed, sometimes it did not. The problem is clearly addressed by the Docker guidelines: `ENTRYPOINT` is for the container steps that should not be replaced by the client, and `CMD` is exactly for client override. It does not seem reasonable for Docker to execute commands while racing with `ENTRYPOINT`, and it is becoming such a major problem for us that we will have to walk away from this solution if the situation remains as it currently is.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

For specialized use cases like this you should not use withDockerContainer. Just run whatever docker commands you need directly from a sh (or indirectly via some script).

Add Comment

[boitoletre@gmail.com (JIRA)]

Ad edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

Well, we are using the Docker agent syntax: {code:java} agent { docker { image 'ag/ubuntu-conan' args '''-v $DOCKERCONFIG_FOLDER/ag/ubuntu-conan.env:/dockerconfig.env } } stage('Use the tool') {   steps {     sh 'conan install whatever'   } }{code} Is not that the exact intended "tool-only" use case you were mentioning above? If not, what is the expected use-case for these kind of agents then?

Add Comment

[boitoletre@gmail.com (JIRA)]

Ad commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Well, we are using the Docker agent syntax:

agent { docker { image 'ag/ubuntu-conan'

args '''-v $DOCKERCONFIG_FOLDER/ag/ubuntu-conan.env:/dockerconfig.env } } stage('Use the tool') { steps { sh 'conan install whatever' } }

Is not that the exact intended "tool-only" use case you were mentioning above? If not, what is the expected use-case for these kind of agents then?

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

agent docker is just sugar for withDockerContainer. The expected use case is anything that happens to work the first time you try it. There is really no further guarantee than that.

Add Comment

[boitoletre@gmail.com (JIRA)]

Ad commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

For specialized use cases like this you should not use withDockerContainer. Just run whatever docker commands you need directly from a sh (or indirectly via some script). We are revisiting our use of docker agents in our CI pipeline. Jesse Glick we are considering following your advice above, thus removing the docker agents and instead run `docker` commands in `sh` steps directly on the master. Yet, our current setup is that we have a stage with many steps running on the agent (having the different steps showing nicely in Jenkins UI). How could we get the same result by executing the docker command directly? (i.e. executing discrete Jenkins `steps` in the same docker container, without restarting the container since it would loose its state).

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

Ad that is indeed a missing feature in Pipeline. I have hijacked JENKINS-44847 to discuss this.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [jenkins.log|[^jenkins.log]),] However, in my case the container process was not detected at all by docker top.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [jenkins  https://issues . log|[^ jenkins -ci . org/secure/attachment/50348/50348_jenkins. log ] ) ,] .

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see  https://issues [jenkins . log|[^ jenkins -ci . org/secure/attachment/50348/50348_jenkins. log ]] ).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [ title jenkins.log| [^ https://issues. jenkins -ci . org/secure/attachment/50348/50348_jenkins. log] ] ).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [ title ^ jenkins.log |https://issues.jenkins-ci.org/secure/attachment/50348/50348_jenkins.log ]).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [ ^ link title|https://issues. jenkins -ci . org/secure/attachment/50348/50348_jenkins. log]).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem (described in WEBSITE-726, see [ link title|https://issues. ^ jenkins -ci . org/secure/attachment/50348/50348_jenkins. log]).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[gan.ainm@gmx.com (JIRA)]

Gan Ainm edited a comment on JENKINS-49278

Re: cat command in docker agents not detected correctly

I encountered a similar problem with the *cdrx/pyinstaller-linux:python2* container (described in WEBSITE-726, see [ ^ corresponding logfile|https://issues. jenkins -ci . org/secure/attachment/50348/50348_jenkins. log]).

However, in my case the container process was not detected at all by {{docker top}}.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-49278

Re: cat command in docker agents not detected correctly

I fixed the Python tutorial error reported by Gan Ainm by using `sh 'docker run ...'` to replace the docker image reference from the Declarative Pipeline. See the pull request for more details.

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)