[JIRA] [durable-task-plugin] (JENKINS-27152) Store sh control files outside of workspace

22 views
Skip to first unread message

jglick@cloudbees.com (JIRA)

unread,
Oct 28, 2015, 9:58:03 AM10/28/15
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
 
Jenkins / Improvement JENKINS-27152
Store sh control files outside of workspace
Change By: Jesse Glick
Summary: Store  bourneshell  sh  control files outside of workspace
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

jw_jenkins-ci@headissue.com (JIRA)

unread,
Nov 14, 2015, 1:47:03 PM11/14/15
to jenkinsc...@googlegroups.com
Jens Wilke commented on Improvement JENKINS-27152
 
Re: Store sh control files outside of workspace

Just came across this issue, since I am trying to cleanup the workspace with this code sequence:

sh 'find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +'

My intention is to clean everything but keep the .git directory. However, this fails, because it also tries to remove the temporary directory, which e.g. contains the script file that it is currently executing, like: ./.4cba202f/script.sh

Since the directory is arbitrary, there is no easy workaround.

I think a gradual improvement would be to give the directory at least consistent name or prefix, like .jenkins-tmp/4cba202f/.... This way it is possible to stay away from it.

jw_jenkins-ci@headissue.com (JIRA)

unread,
Nov 16, 2015, 3:00:02 PM11/16/15
to jenkinsc...@googlegroups.com
Jens Wilke edited a comment on Improvement JENKINS-27152
Just came across this issue, since I am trying to cleanup the workspace with this code sequence:

{code}

sh 'find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +'
{code}


My intention is to clean everything but keep the {{.git}} directory. However, this fails, because it also tries to remove the temporary directory, which e.g. contains the script file that it is currently executing, like: {{./.4cba202f/script.sh}}

Since the directory is arbitrary, there is no easy workaround.

I think a gradual improvement would be to give the directory at least  a  consistent name or prefix, like {{.jenkins-tmp/4cba202f/...}}. This way it is possible to stay away from it.

jw_jenkins-ci@headissue.com (JIRA)

unread,
Nov 16, 2015, 3:10:02 PM11/16/15
to jenkinsc...@googlegroups.com

The PR changes the tmp directory to something like .jenkins-4cba202f.

I think having a separate directory like .jenkins-tmp is not quite easy since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct? This leads automatically to the more complex solution of the tmp-space problem....

The PR is good for me. However, I would suggest to make a longer more descriptive prefix, e.g. .jenkins-tmp-workflow-step-sh-.... So whenever somebody steps on a directory like this, it is obvious who is to blame

jw_jenkins-ci@headissue.com (JIRA)

unread,
Nov 16, 2015, 3:14:03 PM11/16/15
to jenkinsc...@googlegroups.com

BTW, just for some amusement, my current solution to cleanup the workspace, looks like this: 

sh 't=`dirname $0`; t=`basename $t`; find -mindepth 1 -maxdepth 1 \\! -name .git \\! -name $t -exec rm -rf {} +'

jglick@cloudbees.com (JIRA)

unread,
Nov 16, 2015, 8:09:01 PM11/16/15
to jenkinsc...@googlegroups.com

since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct?

Right, that is why I did not try that.

BTW as a workaround you might try

dir('.git') {sh 'cd ..; find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +'}

martin.danjou14@gmail.com (JIRA)

unread,
Nov 25, 2015, 11:45:02 AM11/25/15
to jenkinsc...@googlegroups.com

I just ran into this problem (use case below), and I plan to workaround the problem with this line in .gitignore:

.[a-z0-9][a-z0-9]*/**

My use case is with the Gradle axion-release plugin. The plugin has a verifyRelease task that prevents the release from occurring if there are untracked files in the git repo. Since I run gradle inside an sh step, the temporary files cause the gradle task to fail, and the release fails:

[Workflow] sh
[my-workflow] Running shell script
+ ./gradlew verifyRelease
:verifyRelease
Looking for uncommitted changes.. FAILED

Staged changes:

Unstaged changes:
    added: .29913d6a/script.sh
    added: .29913d6a/jenkins-log.txt
    added: .29913d6a/pid

:verifyRelease FAILED

...
[Workflow] End of Workflow
ERROR: script returned exit code 1
Finished: FAILURE

cobexer@gmail.com (JIRA)

unread,
Nov 27, 2015, 8:59:02 AM11/27/15
to jenkinsc...@googlegroups.com

Any news on this?
I need to use git clean -fdx to have reliable builds and fast turnaround times (avoiding a full clone / checkout every time).
AFAIU the proposed PR would not yet fix this?

jglick@cloudbees.com (JIRA)

unread,
Nov 30, 2015, 3:59:02 PM11/30/15
to jenkinsc...@googlegroups.com

Christoph Obexer no, the proposed PR would not handle the case of git clean -fdx calls. On the other hand that case can be handled by passing the appropriate extension to GitSCM rather than running this Git command manually.

cobexer@gmail.com (JIRA)

unread,
Dec 2, 2015, 10:15:04 AM12/2/15
to jenkinsc...@googlegroups.com

Jesse Glick you mean I could pass extensions: [[$class: 'CleanBeforeCheckout']] to checkout scm somehow? I tried those:

node { bq. scm.extensions.add([$class: 'CleanBeforeCheckout']) bq. scm.extensions.add(new CleanBeforeCheckout()) bq. def myscm = scm; bq. myscm.extensions + [new CleanBeforeCheckout()] bq. checkout scm bq. }

none of them fails(after allowing the field access, ...), but none of them works either. I do that in a multibranch workflow btw, so I need to use checkout scm AFAIK. scm.extensions always only contains one item and never the one I add to the list.

Any Ideas?

jglick@cloudbees.com (JIRA)

unread,
Dec 2, 2015, 5:44:01 PM12/2/15
to jenkinsc...@googlegroups.com

Not sure about usage from workflow-multibranch. Really AbstractGitSCMSource needs to be able to configure extensions. In principle

scm.extensions.add(new hudson.plugins.git.extensions.impl.CleanBeforeCheckout())
checkout scm

ought to work but I have not tested it.

scm_issue_link@java.net (JIRA)

unread,
Dec 2, 2015, 5:51:03 PM12/2/15
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
http://jenkins-ci.org/commit/durable-task-plugin/d652a99ee2820c33d84e366069a9b14e3a0e2ca3
Log:
Merge pull request #11 from jglick/control-dir-prefix-JENKINS-27152

JENKINS-27152 Ameliorate problem by using a predictable control directory name

Compare: https://github.com/jenkinsci/durable-task-plugin/compare/0a836c1cec26...d652a99ee282

scm_issue_link@java.net (JIRA)

unread,
Dec 2, 2015, 5:51:03 PM12/2/15
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java

http://jenkins-ci.org/commit/durable-task-plugin/79990a9f6b5e7daf731bc46f535cd09bb8c2cff3
Log:
JENKINS-27152 Ameliorate problem by at least using a predictable control directory name.
This makes it easy to add to .gitignore and the like.

cobexer@gmail.com (JIRA)

unread,
Dec 3, 2015, 12:03:05 PM12/3/15
to jenkinsc...@googlegroups.com

Jesse Glick scm.extensions can't be modified apparently because somewhere inside the list is actually a hudson.util.CopyOnWriteList...
trying scm.extensions.data = scm.extensions.data << new hudson.plugins.git.extensions.impl.CleanBeforeCheckout() causes this exception:
{{Caused by: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified method hudson.util.CopyOnWriteList leftShift hudson.plugins.git.extensions.impl.CleanBeforeCheckout
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:73)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:15)
at WorkflowScript.run(WorkflowScript:40)}}
which can't be approved...

I'll create a separate issue for the scm.extensions problem to not spam this issue any longer

jglick@cloudbees.com (JIRA)

unread,
Dec 4, 2015, 1:55:02 PM12/4/15
to jenkinsc...@googlegroups.com

Christoph Obexer well CopyOnWriteList.add just replaces its internal array list, but that should be transparent. So something else is amiss.

jglick@cloudbees.com (JIRA)

unread,
Jan 26, 2016, 6:47:02 AM1/26/16
to jenkinsc...@googlegroups.com

Without introducing a new explicit API, it would probably suffice to have every plugin consent to keep temporary files inside the slave FS root, with that whole root being mounted by Image.inside.

jglick@cloudbees.com (JIRA)

unread,
Feb 19, 2016, 2:49:05 PM2/19/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Improvement JENKINS-27152
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Feb 19, 2016, 11:12:04 PM2/19/16
to jenkinsc...@googlegroups.com

Working on a series of patches to use …/workspace/folder/job@tmp as the root of temporary files for a workspace …/workspace/folder/job, and making docker-workflow mount that location. Seems to be working so far, though I had to improve the fix of JENKINS-25678. Remaining to be done: CliGitAPIImpl; FileBinding (JENKINS-32943); AgentServer (

JENKINS-32624 ).

cleclerc@cloudbees.com (JIRA)

unread,
Feb 22, 2016, 9:51:06 AM2/22/16
to jenkinsc...@googlegroups.com

Some use cases:

(?)Linux user home

Maybe interesting to locate the linux user.home under this workspace working folder. /!\ It may break many things if a plugin does this without 'collaborating' with other plugins but it could be a "job config" or a "jenkins global config".

credentials-binding-plugin

ssh-agent

config-file-provider

Maven Pipeline Step

  • Would need to create a tmp folder to prepend to the PATH in which we would create an ephemeral "mvn" file that would inject the proper command line parameters that cannot be passed as environment variables (--settings, --global-settings, --debug, --errors, --batch-mode...). See Sonatype >>6.1. Maven Command Line Options
  • it would be interesting to optionally specify a local-repo under this specific folder (see -Dmaven.repo.local=...)

Gradle Pipeline Step

jglick@cloudbees.com (JIRA)

unread,
Feb 22, 2016, 10:04:04 AM2/22/16
to jenkinsc...@googlegroups.com

Maybe interesting to locate the Linux user.home under this workspace working folder.

$HOME that is. Seems too dangerous, and anyway users may reasonably be expecting configuration files in ~slaveacct/ to be honored.

.ssh/config can only be defined under $HOME

If true, this is a limitation of SSH we would need to work around, if any plugin wishes to modify that configuration. According to man ssh, this is not true:

-F configfile
Specifies an alternative per-user configuration file. If a configuration file is given on the command line, the system-wide configuration file (/etc/ssh/ssh_config) will be ignored. The default for the per-user configuration file is ~/.ssh/config.

BTW Docker used to have such a mandated location for registry credentials, but as of newer versions of the client this is no longer true.

jglick@cloudbees.com (JIRA)

unread,
Feb 22, 2016, 10:09:03 AM2/22/16
to jenkinsc...@googlegroups.com

Plan to file a core PR to not only introduce a formal API for this temporary location (currently planned as a helper method in WorkspaceList), but also to consider it in some places, like workspace cleanup.

cleclerc@cloudbees.com (JIRA)

unread,
Feb 22, 2016, 1:38:02 PM2/22/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc edited a comment on Improvement JENKINS-27152
Some use cases:

h3. (?)Linux user home(?)

Maybe interesting to locate the linux {{user.home}} under this workspace working folder. /!\ It may break many things if a plugin does this without 'collaborating' with other plugins but it could be a "job config" or a "jenkins global config".

* {{.ssh/config}} can only be defined under {{$HOME}}. Are there some config that we would like to define "per job"? See http://www.cyberciti.biz/faq/create-ssh-config-file-on-linux-unix/
* git: most git configuration can be passed through environment variables and thus it is not required to specify a per workspace setup. Moreover, the {{PREFIX}} environment variable allow to define the "system wide" {{$PREFIX/etc/gitconfig}} https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

h3. credentials-binding-plugin

* Use a "secrets" folder to store file based credentials: see [FileBinding.secretsDir(workspace)|https://github.com/jenkinsci/credentials-binding-plugin/blob/3cd54c0a07fcc36ded63a39a2455f1d7337a8934/src/main/java/org/jenkinsci/plugins/credentialsbinding/impl/FileBinding.java#L93-L93]

h3. ssh-agent

* Creates a tmp file for the local socket in java.io.tmp: [AgentServer.createLocalSocketAddress()|https://github.com/jenkinsci/ssh-agent-plugin/blob/576fc86a7fa2c646e030f50abf67bdf3e0cefdcb/src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java#L136-L136]
{{File socket = File.createTempFile("jenkins", ".jnr");}}

h3. config-file-provider

* Creates a tmp folder to store config files [ManagedFileUtil. createTempFile()|https://github.com/jenkinsci/config-file-provider-plugin/blob/d69c1f5a6e400b54d356ba30cb42cd725db72025/src/main/java/org/jenkinsci/plugins/configfiles/buildwrapper/ManagedFileUtil.java#L57-L57]

h3. Maven Pipeline  Step  Integration

* Would need to create a tmp folder to prepend to the PATH in which we would create an ephemeral "mvn" file that would inject the proper command line parameters that cannot be passed as environment variables ({{\-\-settings}}, {{\-\-global-settings}}, {{\-\-debug}}, {{\-\-errors}}, {{\-\-batch\-mode}}...). See [Sonatype >>6.1. Maven Command Line Options|https://books.sonatype.com/mvnref-book/reference/running-sect-options.html]
* (?) it would be interesting to optionally specify a local-repo under this specific folder (see {{\-Dmaven.repo.local=...}})

h3. Gradle Pipeline Step

* Maybe interesting to specify a custom {{GRADLE_USER_HOME}}, see [Gradle >> Chapter 11. The Build Environment|https://docs.gradle.org/current/userguide/build_environment.html]
* (?) maybe interesting to have a per workspace local.repo

jglick@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 2:57:01 PM2/26/16
to jenkinsc...@googlegroups.com

Should also have a Pipeline step akin to pwd but for this temporary location, to make it easy to add other temporary files with custom meanings. Maybe even File.createTempFile semantics as an option.

cleclerc@cloudbees.com (JIRA)

unread,
Feb 28, 2016, 3:42:03 PM2/28/16
to jenkinsc...@googlegroups.com

As you mention java.io.File#createTempFile, java.io.tmpdir seem to be a "static" value defined at startup time (java.io.File.TempDirectory#tmpdir). Due to this "static" behavior, it does not seem possible to have a job specific java.io.tmpdir, located under job@tmp.

Could we mount the java.io.tmpdir in docker-pipeline containers to ensure that temporary files created with java.io.File#createTempFile are visible in docker-pipeline steps? Shall we discuss this in another Jira issue?

jglick@cloudbees.com (JIRA)

unread,
Feb 29, 2016, 12:09:02 PM2/29/16
to jenkinsc...@googlegroups.com

CliGitAPIImpl is a special case. It ignores the Launcher provided to SCM.checkout, and thus works unmodified even inside withContainerStep: the command is run outside the container, directly by the agent. It ought to be adjusted to use the official temporary directory, but it is not strictly necessary.

cleclerc@cloudbees.com (JIRA)

unread,
Feb 29, 2016, 12:13:04 PM2/29/16
to jenkinsc...@googlegroups.com

Jesse Glick by "official temporary directory", do you mean java.io.tmpdir and java.io.File#createTempFile or do you mean PR #33 workspace@tmp?

jglick@cloudbees.com (JIRA)

unread,
Feb 29, 2016, 5:26:05 PM2/29/16
to jenkinsc...@googlegroups.com

Cyrille Le Clerc the latter.

I think everything is now written with the exception of the proposed Pipeline step.

cleclerc@cloudbees.com (JIRA)

unread,
Mar 1, 2016, 7:43:02 AM3/1/16
to jenkinsc...@googlegroups.com

> I think everything is now written

Do you need assistance to test all the plugins with their pull request (ssh-agent, config-file-provider, credentials-binding)?

> with the exception of the proposed Pipeline step.

I'm not sure I understand which plugin you are referring to? CliGitAPIImpl?

cleclerc@cloudbees.com (JIRA)

unread,
Mar 1, 2016, 7:46:06 AM3/1/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc edited a comment on Improvement JENKINS-27152
>  Cyrille Le Clerc the latter.

To clarify, this mean that {{java.io.tmpdir}} will not be mounted by the docker-pipeline plugin in Docker containers launched by {{docker.image('my-image').inside\{...\}}}, and that files created in plugin through {{java.io.File#createTempFile()}} will not be visible in docker containers. Correct?

>  I think everything is now written 


Do you need assistance to test all the plugins with their pull request (ssh-agent, config-file-provider, credentials-binding)?

> with the exception of the proposed Pipeline step.

I'm not sure I understand which plugin you are referring to? {{CliGitAPIImpl}}?

jglick@cloudbees.com (JIRA)

unread,
Mar 1, 2016, 11:52:05 AM3/1/16
to jenkinsc...@googlegroups.com

To clarify, […] Correct?

Correct.

Do you need assistance to test all the plugins

Well everything has automated tests, but of course extra exploratory testing would be welcome.

I'm not sure I understand which plugin you are referring to?

PR coming shortly.

cleclerc@cloudbees.com (JIRA)

unread,
Mar 3, 2016, 7:15:02 PM3/3/16
to jenkinsc...@googlegroups.com

I tested the PR for the config-file-provider. I won't have the time for the other tests (credentials-binding...)

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 9:28:02 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/WorkspaceCleanupThread.java
core/src/main/java/hudson/slaves/WorkspaceList.java
test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java
http://jenkins-ci.org/commit/jenkins/307bfc17385c14aefcb623ccc3e7144e4f01e2a3
Log:
JENKINS-27152 Introduce common API WorkspaceList.tempDir.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 9:28:03 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/WorkspaceCleanupThread.java
core/src/main/java/hudson/slaves/WorkspaceList.java
test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 9:35:04 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java
http://jenkins-ci.org/commit/durable-task-plugin/85be167e1efbd0ef2e813f197c3ab7ba5031cbca
Log:
[FIXED JENKINS-27152] Keep control files outside of the workspace.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 9:35:09 PM3/3/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon resolved as Fixed
 
Jenkins / Improvement JENKINS-27152
Change By: SCM/JIRA link daemon
Status: In Progress Resolved
Resolution: Fixed

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 9:35:12 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java
src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java
src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java
http://jenkins-ci.org/commit/durable-task-plugin/d66df29a6b0a56c0f0a221f8480e130d5bd751fb
Log:
Merge pull request #19 from jglick/temp-dir-

JENKINS-27152

JENKINS-27152 Store control files outside of the workspace

Compare: https://github.com/jenkinsci/durable-task-plugin/compare/92014f174b60...d66df29a6b0a

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:02:06 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java
http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2
Log:

JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:02:07 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

pom.xml
src/findbugs/excludesFilter.xml


src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java

src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java
http://jenkins-ci.org/commit/ssh-agent-plugin/f96025a616f9e5407befea51d76619ec5b266026
Log:
Merge pull request #11 from jglick/temp-dir-

JENKINS-27152

JENKINS-27152 Use a standardized directory for $SSH_AUTH_SOCK

Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/12f6ff0ccaee...f96025a616f9

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:03:02 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

CHANGES.md
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html
http://jenkins-ci.org/commit/workflow-plugin/8c74333321f007e829b2a9581af643b792ef47a6
Log:

JENKINS-27152 Option for pwd step to return temp directory.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:03:04 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
CHANGES.md
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html

http://jenkins-ci.org/commit/workflow-plugin/7bbcab0390b6d97dc40d4dbaf8dde34ef2c5af0c
Log:
Merge pull request #350 from jglick/temp-dir-

JENKINS-27152

JENKINS-27152 Option for pwd step to return temp directory

dogfood@java.net (JIRA)

unread,
Mar 3, 2016, 10:24:02 PM3/3/16
to jenkinsc...@googlegroups.com
dogfood commented on Improvement JENKINS-27152

Integrated in jenkins_main_trunk #4478

JENKINS-27152 Introduce common API WorkspaceList.tempDir. (Revision 307bfc17385c14aefcb623ccc3e7144e4f01e2a3)

Result = SUCCESS
jesse glick : 307bfc17385c14aefcb623ccc3e7144e4f01e2a3
Files :

  • core/src/main/java/hudson/model/WorkspaceCleanupThread.java
  • test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java
  • core/src/main/java/hudson/slaves/WorkspaceList.java

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:30:02 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
http://jenkins-ci.org/commit/docker-workflow-plugin/8832083218f687d9d04ba238c99bb1f9a048cc06
Log:

JENKINS-27152 Expect to mount workspace@tmp directory.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 10:30:03 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

pom.xml
src/main/java/org/jenkinsci/plugins/docker/workflow/AbstractEndpointStepExecution.java
src/main/java/org/jenkinsci/plugins/docker/workflow/DockerDSL.java
src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
src/test/java/org/jenkinsci/plugins/docker/workflow/DockerDSLTest.java
src/test/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStepTest.java
src/test/java/org/jenkinsci/plugins/docker/workflow/ServerEndpointStepTest.java
src/test/java/org/jenkinsci/plugins/docker/workflow/WithContainerStepTest.java
http://jenkins-ci.org/commit/docker-workflow-plugin/2312931df9030b814fed6522e931762f1a93e433
Log:
Merge pull request #33 from jglick/temp-dir-

JENKINS-27152

JENKINS-27152 Mount workspace@tmp

Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/b60bb77ccc51...2312931df903

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 11:04:03 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

src/test/java/plugins/WorkflowPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/ef45669766be39c4b1d203ef5350a47873e5e342
Log:

JENKINS-27152 Integration test for Docker Pipeline, Git, and SSH Agent plugins.

scm_issue_link@java.net (JIRA)

unread,
Mar 3, 2016, 11:04:09 PM3/3/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

pom.xml
src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer.java
src/main/java/org/jenkinsci/test/acceptance/junit/WithCredentials.java
src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/BaseStandardCredentials.java
src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/Credential.java
src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/UserPwdCredential.java
src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java
src/test/java/plugins/GitPluginTest.java
src/test/java/plugins/WorkflowPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/6681f400d9a84ded4c39e8a8c4c982c1a2196295
Log:
Merge pull request #81 from jglick/temp-dir-

JENKINS-27152

JENKINS-27152 Test of standardized temp directory with Git operations

Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/500a376b7556...6681f400d9a8

cobexer@gmail.com (JIRA)

unread,
Mar 7, 2016, 5:25:02 AM3/7/16
to jenkinsc...@googlegroups.com

Updating to 1.8 my pipeline script:
```groovy
node('server') {
dir ('.git')

{ bat 'cd .. && git clean -fdx' }

}
```
Caused the same problem differently:
warning: failed to remove .git@tmp/durable-263b366c/jenkins-log.txt
The batch file cannot be found.

Apparently the control files are NOT stored outside the workspace but only in the parent directory next to the cwd as $CWD@tmp.
Wouldn't it be simpler to store these files in $SLAVE_HOME/tmp/<job name>/...? aka a parallel hierarchy next to the workspace directory?

jglick@cloudbees.com (JIRA)

unread,
Mar 11, 2016, 11:27:03 AM3/11/16
to jenkinsc...@googlegroups.com

Christoph Obexer I do not understand the purpose of that script. Try simply

node('server') {
  bat 'git clean -fdx'
}

Wouldn't it be simpler to store these files in …

There are various future options for choice of the temporary directory but until the core API gets into a reasonable LTS version, and the plugins needing to select a temporary directory can use it, they all need to have identical copies of some simple logic, for which I chose ../$(basename)@tmp as a one-liner that works well enough in most cases. Your proposal is similar to one of the options already discussed, but it is rather more complicated than you think, since you need to consider

  • temporary slave outages which result in Slave.getRootPath being null
  • alternate workspaces like jobname@2 created by concurrent builds
  • people who specify custom workspace directories
  • Unix slave FS roots exceeding roughly 108 characters (you really do not want to know)
  • obscure Windows restrictions yet to be discovered

All fixable in one place with time, but I certainly do not want to be applying the same fix to six or seven places, especially when a mismatch between docker-workflow and one of the others will cause a Dockerized build to break.

cleclerc@cloudbees.com (JIRA)

unread,
Mar 11, 2016, 12:23:02 PM3/11/16
to jenkinsc...@googlegroups.com

Jesse Glick was is your recommendation to use this "workspace@tmp" in pipeline scripts to create temporary files?

Here is a use case with a "tmpFile" that I would like to locate in "workspace@tmp".

def tmpFile=".aws-ec2-instances-status.json" // should be located in 'workspace@tmp'
sh "aws ec2 describe-instances ...  2>&1 | tee ${tmpFile}"
def awsEc2StatusAsJson = readFile(tmpFile)
...

It may be convenient to have a pipeline step "createTempFile()".

cleclerc@cloudbees.com (JIRA)

unread,
Mar 11, 2016, 12:24:03 PM3/11/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc edited a comment on Improvement JENKINS-27152
[~jglick]  was  what  is your recommendation to use this "workspace@tmp" in pipeline scripts to create temporary files?


Here is a use case with a "{{tmpFile}}" that I would like to locate in "workspace@tmp".

{code}

def tmpFile=".aws-ec2-instances-status.json" // should be located in 'workspace@tmp'
sh "aws ec2 describe-instances ...  2>&1 | tee ${tmpFile}"
def awsEc2StatusAsJson = readFile(tmpFile)
...
{code}


It may be convenient to have a pipeline step "{{createTempFile()}}".

cobexer@gmail.com (JIRA)

unread,
Mar 15, 2016, 3:11:04 AM3/15/16
to jenkinsc...@googlegroups.com

Jesse Glick the bat 'cd .. && git clean -fdx' was required for previous versions where the control files were in the current working directory. The 1.8 update broke compatibility with that workaround because the control files moved from the CWD to the parent directory which moved them back to where they can't be. With version 1.8 the bat 'git clean -fdx' is what is needed now.

This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily (Multi-Branch Pipeline Job) - which makes this incompatibility annoying.

And thanks for the long explanation!

jglick@cloudbees.com (JIRA)

unread,
Mar 16, 2016, 2:30:04 PM3/16/16
to jenkinsc...@googlegroups.com

until the core API gets into a reasonable LTS version

Sadly according to #jenkins-meeting it just missed making it into the next line (Daniel Beck & Kohsuke Kawaguchi chose 1.651), so will have to wait for a 2.0 dep I guess.

It may be convenient to have a pipeline step "createTempFile()".

pwd tmp: true

This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily […] which makes this incompatibility annoying.

Sorry. Do not see an easy way around that.

scm_issue_link@java.net (JIRA)

unread,
Mar 29, 2016, 1:51:09 PM3/29/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/configfiles/buildwrapper/ManagedFileUtil.java
http://jenkins-ci.org/commit/config-file-provider-plugin/eb67346e042b09a535ca7e343a5e563706a19e79
Log:

JENKINS-27152 Use a temporary directory for managed files near the workspace.

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:05:04 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
http://jenkins-ci.org/commit/workflow-support-plugin/88b7a89577178ad99f718b7e08d73750378f3d98
Log:

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:12:26 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html
http://jenkins-ci.org/commit/workflow-basic-steps-plugin/ad40c7688bac0e9f3405d17eea0013800ecbe6d8
Log:

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:23:14 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:25:11 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:26:09 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:32:29 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:33:19 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java

JENKINS-27152 Option for pwd step to return temp directory.
Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
Reply all
Reply to author
Forward
0 new messages