[JIRA] [git-client] (JENKINS-21369) Git client fails to remove files/directories with "git clean -fdx"

[hannu.piki@sympa.fi (JIRA)]

Hannu Piki created JENKINS-21369 Git client fails to remove files/directories with "git clean -fdx" Issue Type: Bug Affects Versions: current Assignee: Nicolas De Loof Components: git-client Created: 14/Jan/14 8:50 AM Description: This all has worked fine previously, but latest updates for some reason there have been these kind of issues, as described below. 1. Start Jenkins service 2. Trigger ex. 20 builds 3. Wait and see if one of them fails to build 4. See if the reason is Jenkins issue, not msbuild/unit test 5. Rest of builds that starts after failed one, will fail also with same output as below 6. Wipe workspaces manually 7. Builds will most likely start working again Triggered by Gerrit: http://xxx002:8080/4195 [EnvInject] - Loading node environment variables. Building on master in workspace r:\jenkins-workspaces Fetching changes from the remote Git repository Fetching upstream changes from ssh://hudson@xxxx002:29418/xxx.git Checking out Revision 45d62ed078309607e0b6a24b842ff33c93f3b740 (feature/WI3076) Cleaning workspace Resetting working tree FATAL: Command "git clean -fdx" returned status code 1: stdout: Removing Source/xxx/zzz/Areas Removing Source/xxx/zzz/Content Removing Source/xxx/zzz/Scripts Removing Source/xxx/zzz/Views stderr: warning: failed to remove Source/xxx/zzz/Areas warning: failed to remove Source/xxx/zzz/Content warning: failed to remove Source/xxx/zzz/Scripts warning: failed to remove Source/xxx/zzz/Views hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1: stdout: Removing Source/xxx/zzz/Areas Removing Source/xxx/zzz/Content Removing Source/xxx/zzz/Scripts Removing Source/xxx/zzz/Views stderr: warning: failed to remove Source/xxx/zzz/Areas warning: failed to remove Source/xxx/zzz/Content warning: failed to remove Source/xxx/zzz/Scripts warning: failed to remove Source/xxx/zzz/Views at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1086) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1063) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:900) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:910) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:367) at hudson.plugins.git.GitAPI.clean(GitAPI.java:251) at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:28) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:890) at hudson.model.AbstractProject.checkout(AbstractProject.java:1415) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561) at hudson.model.Run.execute(Run.java:1678) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Environment: OS: Windows 2008 R2 Java: 1.6.0_26 Jenkins: 1.547 Git client: Newest (1.6.1) Other plugins: All are updated, like Git (2.0.1) Jenkins is running as Windows service, with Local system account Project: Jenkins Priority: Critical Reporter: Hannu Piki

This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Can you provide more details that would make this a repeatable bug? The stack trace shows you are using the command line git implementation. That implementation has no known problems with leaving files open. If you can provide a repeatable case, it has a much better chance of being investigated. The JGit implementation has known problems with leaving files open (see JENKINS-19994). There is also a risk that some other program is keeping the files open. There were cases in the past where a virus scanner prevented a directory from being deleted because it was keeping one or more files open from the directory.

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite resolved JENKINS-21369 as Cannot Reproduce

Git client fails to remove files/directories with "git clean -fdx"

Closing as "Cannot Reproduce" after a month of waiting for a response from the original submitter. Change By: Mark Waite (14/Mar/14 11:23 AM) Status: Open Resolved Resolution: Cannot Reproduce

[thomas.coan@sap.com (JIRA)]

Thomas COAN reopened JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Hi, I face same issue : It was working really until yesterday on a specific job (no changes on jenkins configuration for several weeks) remark : Other jobs are still working for the moment (with the same command run on their own git repo). ===================== Error with StackTrace ===================== Manually triggered by user anonymous for Gerrit: https://MY_GIT:8080/607665 Building in workspace <http://MY_JENKINS:8080/jenkins/job/AR_VA_GerritVoter/ws/> > git.exe rev-parse --is-inside-work-tree

Fetching changes from the remote Git repository

> git.exe config remote.origin.url ssh://MY_GIT:29418/sandbox/MY.git Fetching upstream changes from ssh://MY_GIT:29418/sandbox/MY.git > git.exe --version > git.exe fetch --tags --progress ssh://MY_GIT:29418/sandbox/MY.git refs/changes/65/607665/1 > git.exe rev-parse "c68be008cc694ffc7fa87e28baaf86245b696c1a^{commit}" Checking out Revision c68be008cc694ffc7fa87e28baaf86245b696c1a (master) > git.exe config core.sparsecheckout > git.exe checkout -f c68be008cc694ffc7fa87e28baaf86245b696c1a > git.exe rev-parse "FETCH_HEAD^{commit}" > git.exe rev-list 445b38d8c3bbf8d5ef30f811b2c148fb356308ce > git.exe tag -a -f -m Jenkins Build #359 jenkins-AR_VA_GerritVoter-359 Cleaning workspace > git.exe rev-parse --verify HEAD Resetting working tree > git.exe reset --hard > git.exe clean -fdx FATAL: Command "git.exe clean -fdx" returned status code 1: stdout: Removing com.sap.hilo.desktop.overlay/target/ stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/ hudson.plugins.git.GitException: Command "git.exe clean -fdx" returned status code 1: stdout: Removing com.sap.hilo.desktop.overlay/target/ stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/ at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1406) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1382) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1378) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1092) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1102) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:492) at hudson.plugins.git.GitAPI.clean(GitAPI.java:253) at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:28) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:908) at hudson.model.AbstractProject.checkout(AbstractProject.java:1254) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530) at hudson.model.Run.execute(Run.java:1732) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) ===================== === manual Check === ===================== I have tryed to run the git command manually, and it seems to be the same problem : C:\.jenkins\jobs\AR_VA_GerritVoter\workspace\gitRepo>git clean -fdx Removing com.sap.hilo.desktop.overlay/target/ warning: failed to remove com.sap.hilo.desktop.overlay/target/ Note that /target folder is in .gitignore (for more than 1 month) Change By: Thomas COAN (29/Aug/14 2:11 PM) Resolution: Cannot Reproduce Status: Resolved Reopened

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Since your problem directory is the "target" directory, I suspect that one or more programs are holding that directory, or a file in that directory open. Windows will not allow a file to be removed which is open by any process. Windows will not allow a directory to be removed which includes any open file. One simple technique to test the theory is to stop the slave agent and restart it. Another simple technique is to restart the slave computer. The Windows file removal poliicy is much more challenging than the Unix file removal policy, so another alternative is to prefer Unix based build machines instead of Windows machines.

[thomas.coan@sap.com (JIRA)]

Thomas COAN commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Thanks Mark, I have tryed to run the job after a reboot => no old process holding the folder => same problem ! in batch mode (del /F/Q [...]\target , it fails => no error, but no delete!) However I have investigated on the commit that generates the problem, and I have found a new file that generates a path greater than 255 characters in my target folder. Could it be the root cause ? I will check with the dev if he can change it and keep you in touch I cannot switch on a unix machine (not owned by myself !)

[daniel@beckweb.net (JIRA)]

Daniel Beck commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Issue report needs to show that this is a problem with Jenkins. The Sysinternals tools (e.g. Process Explorer) could help determining which process is holding a file handle.

[daniel@beckweb.net (JIRA)]

Daniel Beck commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

FWIW I've repeatedly had to remind my users to log out of the build server instead of just closing the RDC session – make sure there are no sessions with e.g. a cmd.exe in that directory open.

[daniel@beckweb.net (JIRA)]

Daniel Beck commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Sorry, didn't see the comment posted just before mine. It's likely that the long path is the problem in which case this is Not a Defect in Jenkins. A few suggestions if you're right around ~260 chars and your developer is being stubborn: Use a short Jenkins (master/slave) root directory, e.g. D:\J Change the Workspace Root Directory on the master completely (Configure System; Advanced), e.g. D:\WS if building on the master Set the hudson.model.Slave.workspaceRoot system property to something shorter than workspace if building on a slave Set the hudson.matrix.MatrixConfiguration.useShortWorkspaceName system property to true for any Multi-configuration projects Configure a custom workspace in Advanced Project Options

[thomas.coan@sap.com (JIRA)]

Thomas COAN commented on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

I have done a test after a reboot of the machine on jenkins console (without RDC) => failed In process explorer no process is handling the folder. Ok problem comes from the length of windows path > 255. Problem is solved for us => should be a jenkins restriction

[thomas.coan@sap.com (JIRA)]

Thomas COAN edited a comment on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Ok problem comes from the length of windows path > 255.

Problem is solved for us => should be a jenkins restriction

I will check your proposition but we have some restrictions as some jobs are generated by an automate (=> to do in our side for the choice of folder names)

[thomas.coan@sap.com (JIRA)]

Thomas COAN edited a comment on JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

Ok problem comes from the length of windows path > 255.

Problem is solved for us => should be a jenkins restriction

thanks Daniel,

I will check your proposition but we have some restrictions as some jobs are generated by an automate (=> to do in our side for the choice of folder names)

This message is automatically generated by JIRA.

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite resolved JENKINS-21369 as Fixed

Git client fails to remove files/directories with "git clean -fdx"

Change By: Mark Waite (29/Aug/14 3:24 PM) Status: Reopened Resolved Resolution: Fixed

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite closed JENKINS-21369 as Fixed

Git client fails to remove files/directories with "git clean -fdx"

Change By: Mark Waite (29/Aug/14 3:24 PM)

Status: Resolved Closed