[JIRA] (JENKINS-55354) Permissions issue in remoting image jenkins/slave:latest

4 views
Skip to first unread message

thibault@doubliez.com (JIRA)

unread,
Dec 28, 2018, 2:01:03 PM12/28/18
to jenkinsc...@googlegroups.com
Thibault Doubliez created an issue
 
Jenkins / Bug JENKINS-55354
Permissions issue in remoting image jenkins/slave:latest
Issue Type: Bug Bug
Assignee: Nicolas De Loof
Components: docker-slaves-plugin, remoting
Created: 2018-12-28 19:00
Environment: Jenkins 2.150.1 installed on CentOS 7.6.1810 using RPM package

This build stage is run on the master since Docker Slaves plugin doesn't support running on slaves last time I checked.

Java version:
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

The "jenkins" user on the machine has 996:993 uid:gid, which may be part of the issue? I had problems with that in the past.
Priority: Minor Minor
Reporter: Thibault Doubliez

Using jenkins/slave:latest remoting image in Docker Slaves plugin, our builds stopped working a few days ago with a permissions error when initializing the JAR Cache location. This is the relevant console output for the build:

[Pipeline] dockerNode
Start Docker container to host the build
Unable to find image 'jenkins/slave:latest' locally
latest: Pulling from jenkins/slave
54f7e8ac135a: Already exists
d6341e30912f: Already exists
087a57faf949: Already exists
5d71636fb824: Already exists
9da6b28682cf: Already exists
203f1094a1e2: Already exists
ee38d9f85cf6: Already exists
7f692fae02b6: Already exists
eaa976dc543c: Already exists
173a06ff64cc: Pulling fs layer
28fc185aee23: Pulling fs layer
e02013eddffd: Pulling fs layer
cc49fe331e2e: Pulling fs layer
cc49fe331e2e: Waiting
173a06ff64cc: Verifying Checksum
173a06ff64cc: Download complete
28fc185aee23: Verifying Checksum
28fc185aee23: Download complete
e02013eddffd: Verifying Checksum
e02013eddffd: Download complete
173a06ff64cc: Pull complete
28fc185aee23: Pull complete
e02013eddffd: Pull complete
cc49fe331e2e: Verifying Checksum
cc49fe331e2e: Download complete
cc49fe331e2e: Pull complete
Digest: sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196
Status: Downloaded newer image for jenkins/slave:latest
[12/28/18 01:02:51] Launching agent
$ docker start --interactive --attach 56be9bbd2bba60384bb02e9ba11a2a8cc4bbb954620b4dcc9a03ca9f3399b315
Exception in thread "main" java.io.IOException: Failed to initialize the default JAR Cache location
	at hudson.remoting.JarCache.getDefault(JarCache.java:41)
	at hudson.remoting.ChannelBuilder.withJarCacheOrDefault(ChannelBuilder.java:220)
	at hudson.remoting.Launcher.main(Launcher.java:736)
	at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:691)
	at hudson.remoting.Launcher.run(Launcher.java:373)
	at hudson.remoting.Launcher.main(Launcher.java:283)
Caused by: java.lang.IllegalArgumentException: Root directory not writable: ?/.jenkins/cache/jars
	at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:62)
	at hudson.remoting.JarCache.getDefault(JarCache.java:39)
	... 5 more
Caused by: java.nio.file.AccessDeniedException: /home/jenkins/?
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:767)
	at hudson.remoting.Util.mkdirs(Util.java:231)
	at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:60)
	... 6 more 

 

I filed an issue on GitHub: https://github.com/jenkinsci/docker-slave/issues/45

As I mentioned there, it seems like this commit could be related, as it's changing the uid of the user in the Docker image: https://github.com/jenkinsci/docker-slave/commit/723d12eea77b5d5bc3082612c12e30e422e237f2

Neither the new uid 1000 nor the previous one 10000 matches the uid of the jenkins user where my build is running (uid:gid is 996:993), so I'm not sure if that's really the reason, but for sure the build was still working until Dec 19, and then I noticed it stopped working when I came back from vacation on Dec 26. This commit was done on Dec 22 and is related to user/permissions, so that's what makes me suspicious

As a workaround I changed the remoting image to specific version jenkins/slave:3.27-1, which was published around 2 months ago according to https://hub.docker.com/r/jenkins/slave/tags. This fixed the issue.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

nicolas.deloof@gmail.com (JIRA)

unread,
Apr 24, 2019, 10:30:32 AM4/24/19
to jenkinsc...@googlegroups.com
Nicolas De Loof assigned an issue to Unassigned
Change By: Nicolas De Loof
Assignee: Nicolas De Loof

antoine.tran.11w@gmail.com (JIRA)

unread,
Sep 5, 2019, 7:04:03 AM9/5/19
to jenkinsc...@googlegroups.com
Antoine Tran commented on Bug JENKINS-55354
 
Re: Permissions issue in remoting image jenkins/slave:latest

We have the same issue! Thank you for the workaround, it works!

 

thibault@doubliez.com (JIRA)

unread,
Sep 5, 2019, 5:29:02 PM9/5/19
to jenkinsc...@googlegroups.com

Digging into this issue more recently, I realized that Docker Slaves plugin actually hardcodes the 10000 uid when running the jenkins/slave remoting container, and after the change in jenkins/slave to use 1000 instead of 10000, there's now a mismatch which causes the permission issue.

I opened a PR in Docker Slaves plugin to have that fixed, along with another issue I encountered recently after upgrading to Docker 19.03.0: https://github.com/jenkinsci/docker-slaves-plugin/pull/28

Reply all
Reply to author
Forward
0 new messages