[JIRA] (JENKINS-55577) docker agent using a custom Dockerfile is broken with docker buildkit

1 view
Skip to first unread message

thomas@deepomatic.com (JIRA)

unread,
Jan 14, 2019, 8:29:02 AM1/14/19
to jenkinsc...@googlegroups.com
Thomas Riccardi created an issue
 
Jenkins / Bug JENKINS-55577
docker agent using a custom Dockerfile is broken with docker buildkit
Issue Type: Bug Bug
Assignee: Unassigned
Components: docker-workflow-plugin
Created: 2019-01-14 13:28
Environment: ubuntu 16.04
docker 18.09.0
jenkins 2.150.1
docker-workflow-plugin 1.17
Labels: docker
Priority: Minor Minor
Reporter: Thomas Riccardi

When enabling buildkit in docker (either via `/etc/docker/daemon.json` or a global env var in jenkins: DOCKER_BUILDKIT=1), the job fails just after building the docker image used for the agent:

 https://jenkins.io/doc/book/pipeline/docker/#dockerfile

[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Agent Setup)
[Pipeline] isUnix
[Pipeline] readFile
[Pipeline] sh
[workspace] Running shell script
+ docker build -t xxxx -f Dockerfile .

... buildkit logs ...

[Pipeline] dockerFingerprintFrom
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] End of Pipeline
java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(DockerClient.java:220)
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:133)
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:85)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Analysis:

jenkins tries to get information about the base image used in the Dockerfile: `dockerFingerprintFrom`. It does so by assuming the base image is available in `docker images`, as it was just used to build the agent image.

(This has a small timing issue if a system-wide `docker image prune` happens between the build and the call to `dockerFingerprintFrom`.)

This worked great with the old docker builder, which effectively did a `docker pull` of the base image, but buildkit does not behave like that: it keeps things completely internally.

$ cat Dockerfile
FROM ubuntu:18.04
$ DOCKER_BUILDKIT=1 docker build .
[+] Building 1.7s (5/5) FINISHED                                                                                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                         1.6s
 => [1/1] FROM docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
 => => resolve docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
 => exporting to image                                                                                                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                                                                                                 0.0s
 => => writing image sha256:645e081eb3fdb8c828216921411fc0c08335e4fd098c151b44aba797fa334839                                                                                                                                            0.0s
$ docker inspect ubuntu:18.04
[]
Error: No such object: ubuntu:18.04
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

k.scherban@gmail.com (JIRA)

unread,
Feb 26, 2019, 8:27:02 AM2/26/19
to jenkinsc...@googlegroups.com
Konstantin Shcherban commented on Bug JENKINS-55577
 
Re: docker agent using a custom Dockerfile is broken with docker buildkit

Faced the same issue.

  • jenkins version: 2.138.1
  • docker engine version: 18.09.2, API version: 1.39, Git commit: 6247962
  • docker-workflow-plugin version: 1.17

Following step fails

docker.withRegistry(registry) {
   docker.build(image_repo + ":$tag", '-f docker/Dockerfile .').push()
   docker.image(image_repo + ":$tag").push(shortTag)
}

thomas@deepomatic.com (JIRA)

unread,
Feb 26, 2019, 8:45:04 AM2/26/19
to jenkinsc...@googlegroups.com

As a workaround, I simply disabled the whole function that tried to collect the base image id: https://github.com/Deepomatic/docker-workflow-plugin/commit/826b06d2188b01f9a91be5130bb9eec30831c731

It's OK for me as I don't use any of these jenkins collected metadata; it should not be merged.

I pushed the .hpi file I built there: https://github.com/Deepomatic/docker-workflow-plugin/releases/tag/deepomatic-docker-workflow-1.18-SNAPSHOT
Built using these commands:

docker run -it --rm -v $PWD:/app -v $PWD/.m2:/root/.m2 maven:3-jdk-8 bash
cd /app
mvn -DskipTests clean install
# result at
ls .m2/repository/org/jenkins-ci/plugins/docker-workflow/1.18-SNAPSHOT/docker-workflow-1.18-SNAPSHOT.hpi

No issue encountered since we deployed that version more than a month ago.

jonesbusy@gmail.com (JIRA)

unread,
Mar 26, 2019, 5:04:03 PM3/26/19
to jenkinsc...@googlegroups.com

Same issue for us Jenkins 2.164.1 and docker-workflow-plugin version: 1.17 and docker 18.09.3 with buildkit

 

```

[Pipeline] End of Pipeline java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'

```

jonesbusy@gmail.com (JIRA)

unread,
Mar 26, 2019, 5:04:04 PM3/26/19
to jenkinsc...@googlegroups.com
Valentin Delaye edited a comment on Bug JENKINS-55577
Same issue for us Jenkins 2.164.1 and docker-workflow-plugin version: 1.17 and docker 18.09.3 with buildkit

 
``` {code}

[Pipeline] End of Pipeline java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'

``` {code}

f.grutschus@lubyte.de (JIRA)

unread,
Apr 29, 2019, 1:04:03 PM4/29/19
to jenkinsc...@googlegroups.com

Instead of removing the functionality we just need to pull the last used image explicit before running docker build? Correct?

thomas@deepomatic.com (JIRA)

unread,
Apr 29, 2019, 1:17:04 PM4/29/19
to jenkinsc...@googlegroups.com

Valentin Delaye
This would be vulnerable to race-conditions: what if the image changed between the two actions ?

f.grutschus@lubyte.de (JIRA)

unread,
Apr 30, 2019, 10:17:02 AM4/30/19
to jenkinsc...@googlegroups.com

That true. So the only option would be to get the sha256 from the output and pull the image by it that digest?

f.grutschus@lubyte.de (JIRA)

unread,
Apr 30, 2019, 10:33:02 AM4/30/19
to jenkinsc...@googlegroups.com
Fabian Grutschus edited a comment on Bug JENKINS-55577

thomas@deepomatic.com (JIRA)

unread,
Apr 30, 2019, 11:27:03 AM4/30/19
to jenkinsc...@googlegroups.com

Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...).
I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...)

Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

f.grutschus@lubyte.de (JIRA)

unread,
Apr 30, 2019, 12:07:02 PM4/30/19
to jenkinsc...@googlegroups.com

buildctl has the option --export-cache=mode=max, which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

peter.niederlag@datenbetrieb.de (JIRA)

unread,
Jul 8, 2019, 5:16:04 AM7/8/19
to jenkinsc...@googlegroups.com
Peter Niederlag updated an issue
 
Change By: Peter Niederlag
Priority: Minor Major

peter.niederlag@datenbetrieb.de (JIRA)

unread,
Jul 8, 2019, 5:17:04 AM7/8/19
to jenkinsc...@googlegroups.com
Peter Niederlag commented on Bug JENKINS-55577
 
Re: docker agent using a custom Dockerfile is broken with docker buildkit

anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.

michael.kriese@visualon.de (JIRA)

unread,
Sep 11, 2019, 4:03:04 AM9/11/19
to jenkinsc...@googlegroups.com

This can be closed as fixed with docker-workflow-plugin 1.19
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

michael.kriese@visualon.de (JIRA)

unread,
Sep 11, 2019, 4:14:04 AM9/11/19
to jenkinsc...@googlegroups.com
Michael Kriese edited a comment on Bug JENKINS-55577
This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released)

https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350
Reply all
Reply to author
Forward
0 new messages