[JIRA] (JENKINS-60133) Default HOME folder in containers does not exist

0 views
Skip to first unread message

pablo.gomezjimenez@dhl.com (JIRA)

unread,
Nov 11, 2019, 11:17:02 AM11/11/19
to jenkinsc...@googlegroups.com
Pablo Gomez created an issue
 
Jenkins / Bug JENKINS-60133
Default HOME folder in containers does not exist
Issue Type: Bug Bug
Assignee: Unassigned
Components: kubernetes-plugin
Created: 2019-11-11 16:16
Environment: Kubernetes plugin 1.21.1
Openshift 3.11
Priority: Major Major
Reporter: Pablo Gomez

I upgraded from Kubernetes plugin 1.16 to 1.21 and suddenly many of our Jenkins slaves stopped working. We found that the issue is caused by the default HOME environment variable that the plugin adds to all containers in Jenkins slave pods in Openshift. https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L104

That variable has a default value of /home/jenkins but if that folder does not exist then the container can fail. 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

aburdajewicz@cloudbees.com (JIRA)

unread,
Jan 22, 2020, 7:20:04 AM1/22/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ commented on Bug JENKINS-60133
 
Re: Default HOME folder in containers does not exist

This is probably a side effect of the change of working directory to /home/jenkins/agent (i.e. https://issues.jenkins-ci.org/browse/JENKINS-58705). If the /home/jenkins does not exist in the image, it will be automatically added during the mounting of the workspace volume but will not be writable.
The DEFAULT_HOME might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines.

aburdajewicz@cloudbees.com (JIRA)

unread,
Jan 23, 2020, 1:24:02 AM1/23/20
to jenkinsc...@googlegroups.com

Pablo Gomez Setting the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder=/home/jenkins/agent should workaround that problem and mock the behavior that was there before JENKINS-58705.

aburdajewicz@cloudbees.com (JIRA)

unread,
Jan 23, 2020, 1:24:03 AM1/23/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-60133
This is probably a side effect of the change of working directory to {{/home/jenkins/agent}} (i.e. https://issues.jenkins-ci.org/browse/ JENKINS-58705). If the {{/home/jenkins}} does not exist in the image,  it will be automatically added during the mounting of the workspace volume but will not be writable.

The {{DEFAULT_HOME}} might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines.

pablo.gomezjimenez@dhl.com (JIRA)

unread,
Jan 23, 2020, 3:15:02 AM1/23/20
to jenkinsc...@googlegroups.com

Thanks Allan, but when you are using some random image from DockerHub in the pod (i.e. Maven image), /home/jenkins does not exist and you cannot control which user that image is using

aburdajewicz@cloudbees.com (JIRA)

unread,
Jan 28, 2020, 6:19:02 AM1/28/20
to jenkinsc...@googlegroups.com

Pablo Gomez Most if images have a user defined and a HOME directory. As an example the maven image's HOME dir is set to /root and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the example.

The real challenge with Openshift is that the arbitraty uids are generated dynamically without a $HOME directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user /home/jenkins. Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug.

Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, /tmp/ ?), use an additional emptyDir as HOME ?
Maybe Vincent Latombe has some ideas ?

aburdajewicz@cloudbees.com (JIRA)

unread,
Jan 28, 2020, 6:30:02 PM1/28/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-60133
[~pgomez] Most if of images have a user defined and a {{HOME}} directory. As an example the maven image's HOME dir is set to [/root|https://github.com/carlossg/docker-maven/blob/master/jdk-8/Dockerfile#L4] and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the [example|https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven-yaml.groovy].

The real challenge with Openshift is that the arbitraty uids are generated dynamically without a {{$HOME}} directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user {{/home/jenkins}}. Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug.


Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, {{/tmp/}} ?), use an additional {{emptyDir}} as {{HOME}} ?
Maybe [~vlatombe] has some ideas ?

vincent@latombe.net (JIRA)

unread,
Feb 18, 2020, 5:37:02 AM2/18/20
to jenkinsc...@googlegroups.com

These problems make me think the default should be to do absolutely nothing (essentially, remove https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L291-L296).

On Openshift, if you want to run an image from DockerHub, you should set up the HOME variable pointing to a volume backed by an empty dir to make sure the random uid has access. This can be done already using envvars and volumes or using the yaml spec.

But if you have proper images setup following OpenShift recommendations and it behaves properly with a random UID, then the plugin shouldn't do anything.

vincent@latombe.net (JIRA)

unread,
Feb 18, 2020, 5:50:03 AM2/18/20
to jenkinsc...@googlegroups.com
Vincent Latombe assigned an issue to Vincent Latombe
 
Change By: Vincent Latombe
Assignee: Vincent Latombe

vincent@latombe.net (JIRA)

unread,
Feb 18, 2020, 5:50:03 AM2/18/20
to jenkinsc...@googlegroups.com
Vincent Latombe started work on Bug JENKINS-60133
 
Change By: Vincent Latombe
Status: Open In Progress

pablo.gomezjimenez@dhl.com (JIRA)

unread,
Feb 18, 2020, 7:39:02 AM2/18/20
to jenkinsc...@googlegroups.com
Pablo Gomez commented on Bug JENKINS-60133
 
Re: Default HOME folder in containers does not exist

Yes, most of the images have a HOME folder but that folder is not usually /home/jenkins. And if you use different images each one can have different home folders.

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 20, 2020, 6:30:02 PM2/20/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ edited a comment on Bug JENKINS-60133
[~pgomez] Setting the system property {{-Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder .defaultHome =/home/jenkins/agent}} should workaround that problem and mock the behavior that was there before JENKINS-58705.

vincent@latombe.net (JIRA)

unread,
Feb 27, 2020, 4:38:03 AM2/27/20
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Progress Review
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

vincent@latombe.net (JIRA)

unread,
Mar 2, 2020, 4:31:03 AM3/2/20
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Review Fixed but Unreleased
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages