[JIRA] (JENKINS-52750) dir step creates a <dirname>@tmp directory at <dirname> level.

[jlpinardon@java.net (JIRA)]

jlpinardon created an issue

Jenkins / JENKINS-52750 dir step creates a <dirname>@tmp directory at <dirname> level. Issue Type: Task Assignee: Unassigned Components: pipeline Created: 2018-07-26 12:08 Environment: Jenkins 2.121.2 on a CentOS 7 Labels: pipeline Priority: Major Reporter: jlpinardon In order to install some internal tools within /opt/tools through a Jenkins job, I have created a /opt/tools directory belonging to jenkins:jenkins and where user jenkins only (the user running the slave) has rwx rights. Trying something like : stages { steps('xyz') { dir('/opt/tools') { sh "pwd" } } } Fails with an exception ending with : java.nio.file.AccessDeniedException: /opt/iotb@tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at hudson.FilePath.mkdirs(FilePath.java:3098) at hudson.FilePath.access$900(FilePath.java:209) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) It appears that Jenkins tries to create a tools@tmp directory at the same level as tools. Yet, there is absolutely no reason for the tools root directory to be writable for any user. And as far as /opt is concerned here, it must no be writable for anybody else than root.   Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

[jlpinardon@java.net (JIRA)]

jlpinardon updated an issue

Jenkins / JENKINS-52750 dir step creates a <dirname>@tmp directory at <dirname> level.

Change By: jlpinardon In order to install some internal tools within {{/opt/tools}} through a Jenkins job, I have created a {{/opt/tools}} directory belonging to _{{jenkins:jenkins}}_ and where user {{jenkins}} only (the user running the slave) has rwx rights. Trying something like : {code:java}

stages {   steps('xyz') {     dir('/opt/tools') {        sh "pwd"     }   } }

{code}

Fails with an exception ending with :

{code:java} java.nio.file.AccessDeniedException: /opt/ iotb tools @tmp

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at hudson.FilePath.mkdirs(FilePath.java:3098) at hudson.FilePath.access$900(FilePath.java:209) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

{code} It appears that Jenkins tries to create a tools@tmp directory at the same level as {{tools}}. Yet, there is absolutely no reason for the {{tools}} root directory to be writable for any user. And as far as *_{{/opt}}_* is concerned here, _it must no be writable for anybody else than root_.

Add Comment

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer updated an issue

Jenkins / JENKINS-52750 dir step creates a <dirname>@tmp directory at <dirname> level.

Change By: Andrew Bayer Component/s: workflow-basic-steps-plugin Component/s: pipeline

Add Comment

[jielpe-cblv38@protonmail.com (JIRA)]

jlpinardon updated an issue

Jenkins / JENKINS-52750 dir step creates a <dirname>@tmp directory at <dirname> level.

Change By: jlpinardon

In order to install some internal tools within {{/opt/tools}} through a Jenkins job, I have created a {{/opt/tools}} directory belonging to _{{jenkins:jenkins}}_ and where user {{jenkins}} only (the user running the slave) has rwx rights. Trying something like : {code:java} stages {   steps('xyz') {     dir('/opt/tools') {        sh "pwd"     }   } } {code} Fails with an exception ending with : {code:java}

java.nio.file.AccessDeniedException: /opt/tools@tmp

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at hudson.FilePath.mkdirs(FilePath.java:3098) at hudson.FilePath.access$900(FilePath.java:209) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} It appears that Jenkins tries to create a tools@tmp directory at the same level as {{tools}}. Yet, there is absolutely no reason for the {{tools}} root directory to be writable for any user.

And as far as *_{{/opt}}_* is concerned here, for sure _it must no not be writable for anybody else than root_. Additionnally, such _@tmp_ directory is not removed once the build is achieved. Even though it seems that the directory is empty, I think that Jenkins should remove it to give back a clean environment.

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level. The dir step is generally used for workspace-relative paths where the agent is assumed to have full control over the directory, and Jenkins will use sibling directories for various things such as temp directories or shared libraries. If you want to install things in the absolute directory /opt/tmp on the agent, I would do it in a script in the sh step so that you have full control over what will happen and so that the Pipeline does not run with /opt/tmp as its workspace directory at any point. You might be able to use the Workspace Cleanup Plugin to clear out the temp directory, but to take a step back, I think it would make more sense to just use an agent that already has those tools installed in the first place, i.e. some kind of container or pre-built VM image, so that you aren't changing your agent configuration inside of Jenkins at all.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum edited a comment on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

The {{dir}} step is generally used for workspace-relative paths where the agent is assumed to have full control over the directory, and Jenkins will use sibling directories for various things such as temp directories or shared libraries. If you want to install things in the absolute directory {{/opt/tmp}} on the agent, I would do it in a script in with the {{sh}} step so that you have full control over what will happen and so that the Pipeline does not run with {{/opt/tmp}} as its workspace directory at any point. You might be able to use the [Workspace Cleanup Plugin|https://plugins.jenkins.io/ws-cleanup] to clear out the temp directory, but to take a step back, I think it would make more sense to just use an agent that already has those tools installed in the first place, i.e. some kind of container or pre-built VM image, so that you aren't changing your agent configuration inside of Jenkins at all.

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey updated an issue

Jenkins / JENKINS-52750

dir step creates a <dirname>@tmp directory at <dirname> level.

Change By: Vivek Pandey Labels: pipeline pipeline-triaged

Add Comment

[ivan.martinez.rodriguez@ericsson.com (JIRA)]

Ivan Martinez commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level. In the scenario where you have a pipeline where you have 2 stages, each of them executed in a docker container where the USER of the container is "root". You would have the following problem. Imagine that Stage A executes something like: sh ("mkdir -p foo/bar"). This will generate the directories foo and bar with user and group permissions of "root". Stage B executes using dir("foo/bar") the following command : sh ("touch somefile.txt").  Due to the fact that dir() tries to create a @tmp directory within "foo/bar" the pipeline will fail prompting something like this: Exception: java.nio.file.AccessDeniedException: </path/to/the/workspace...>/foo/bar/@tmp I omit the workspace path as it is not relevant. Why is this @tmp creation actually needed?

Add Comment

[dnusbaum@cloudbees.com (JIRA)]

Devin Nusbaum commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

Why is this @tmp creation actually needed? The @tmp directory is created by the sh step to store things like the log output and return code of the script (see all uses of controlDir in FileMonitoringTask.java), and is created as a sibling to the workspace directory, since the agent is assumed to have write access to the workspace and its parent directory. The problem is that the dir step actually changes the workspace for everything inside of its body, so the shell step ends up creating a temp directory adjacent to the directory specified by dir, and if your permissions are such that creating directories next to the one specified by dir fails, the sh step will fail as well. I don't know if there is a way for FileMonitoringController to use the original workspace rather than the directory specified by dir when creating the temp directory, but if so that might be a way to fix this (I think only the current workspace is available, although maybe we could change the dir step to provide the original value in some way. I wonder if looking at what the parallel step does for the workspace would be useful here.). As a workaround you should be able to use cd in your sh script instead of the dir step.

Add Comment

[jielpe-cblv38@protonmail.com (JIRA)]

jlpinardon edited a comment on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

So, if I correctly understand the answer above, the point is that Jenkins needs some space to record its own stuff when a job is running. IMHA (but perhaps there are some cases I don't imagine where it is not possible ?) the better place to do that is the _build workspace_, whatever the current working directory is. At least, jenkins is sure to be able (i.e. have rwx rights) to create its temp directory. I guess that it should not be too difficult for a jenkins code to retrieve the aboslute path of the build workspace. Besides, _as a jenkins admin_, I wouldn't still see this temporary directory once the build is achieved. _Please, let this place as clean as you found it and sweep your own dust_ ! ;). Especially given that this temp dir looks to be empty in most cases when the build is over. Best Reards Regards

Add Comment

[jielpe-cblv38@protonmail.com (JIRA)]

jlpinardon commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

So, if I correctly understand the answer above, the point is that Jenkins needs some space to record its own stuff when a job is running. IMHA (but perhaps there are some cases I don't imagine where it is not possible ?) the better place to do that is the build workspace, whatever the current working directory is. At least, jenkins is sure to be able (i.e. have rwx rights) to create its temp directory.

I guess that it should not be too difficult for a jenkins code to retrieve the aboslute path of the build workspace.

Besides, as a jenkins admin, I wouldn't still see this temporary directory once the build is achieved. Please, let this place as clean as you found it and sweep your own dust ! . Especially given that this temp dir looks to be empty in most cases when the build is over. Best Reards Add Comment

[pgarrett@us.ibm.com (JIRA)]

Philip Garrett commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

The behavior is both undocumented (as far as I can tell), and surprising, since normally "change current directory" would not be a mutating operation.   Also, using the name of the directory requested by the user compounds the confusion, because it's easy to miss the @tmp suffix, or think it's an artifact of the exception rendering. I didn't really notice the @tmp suffix until after I had been fighting with this problem for a while, believing Jenkins was trying to create a directory that already existed.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

Do not use dir for this purpose. Simply start your sh script with cd /opt/tools.

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

[pkruk@parasoft.com (JIRA)]

Piotr Krukowiecki commented on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

A have a pipeline which builds and creates installer/.zip archive. I'm using "dir" to change directory to a folder and execute python script there. The problem I encountered is that now the installer/zip contains additional empty folder@tmp. Is such usage of "dir" wrong? Seems normal to me. Maybe "dir" could automatically delete the "folder@tmp" after it finishes... Or simply create is in some location for temporary files, if jenkins provides such thing. If not, maybe jenkins could be extended to provide it.

Add Comment

[pkruk@parasoft.com (JIRA)]

Piotr Krukowiecki edited a comment on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

A have a pipeline which builds and creates installer/.zip archive. I'm using "dir" to change directory to a folder and execute python script there. The problem I encountered is that now the installer/zip contains additional empty folder@tmp. Is such usage of "dir" wrong? Seems normal to me.

Maybe Edit: I've read the explanation that Devin wrote again, and I see that it's not " dir cd " could automatically delete that created the " folder@tmp , but rather the " after sh" step (or in my case - the "bat" step probably). Is that the only step that does it finishes ? It's a bit problematic . Now I'm affraid to use "cd", because who knows how will it affect other steps? And it's a very useful step .. . Or simply create If "sh"/"bat" is the only such step, maybe it could create the temp folder in some location for temporary files, if jenkins provides such thing. If not, maybe jenkins could be extended to provide it.

Add Comment

[pkruk@parasoft.com (JIRA)]

Piotr Krukowiecki edited a comment on JENKINS-52750

Re: dir step creates a <dirname>@tmp directory at <dirname> level.

A have a pipeline which builds and creates installer/.zip archive. I'm using "dir" to change directory to a folder and execute python script there. The problem I encountered is that now the installer/zip contains additional empty folder@tmp.

Edit: I've read the explanation that Devin wrote again, and I see that it's not " cd dir " that created the folder@tmp, but rather the "sh" step (or in my case - the "bat" step probably). Is that the only step that does it? It's a bit problematic. Now I'm affraid to use " cd dir ", because who knows how will it affect other steps? And it's a very useful step...

If "sh"/"bat" is the only such step, maybe it could create the temp folder in some location for temporary files, if jenkins provides such thing. If not, maybe jenkins could be extended to provide it.

Add Comment