[JIRA] (JENKINS-55836) Unable to delete workspace due to locked git .pack file

[chilber@agile-im.de (JIRA)]

Carsten Hilber moved an issue

Jenkins / JENKINS-55836 Unable to delete workspace due to locked git .pack file Change By: Carsten Hilber Project: Infrastructure Jenkins Key: INFRA JENKINS - 1988 55836 Workflow: classic default workflow JNJira + In-Review Component/s: git-plugin Component/s: github Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[dbeck@cloudbees.com (JIRA)]

Daniel Beck assigned an issue to Unassigned

Jenkins / JENKINS-55836 Unable to delete workspace due to locked git .pack file

Change By: Daniel Beck Assignee: Daniel Beck

Add Comment

[chilber@agile-im.de (JIRA)]

Carsten Hilber updated an issue

Jenkins / JENKINS-55836 Unable to delete workspace due to locked git .pack file

Change By: Carsten Hilber Priority: Minor Critical

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file Please provide more details to duplicate the problem. What is WS-CLEANUP and how is it used in that job? Is the Jenkins job a Pipeline job or a Freestyle job? If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job? Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace? What git related options are included in the job definition? Is there an anti-virus program running on the Windows computer? If so, is the problem still visible if virus scanning is disabled for the workspace directory? I can't duplicate the problem in my experiment. I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Please provide more details to duplicate the problem.

* What is {{WS-CLEANUP}} and how is it used in that job? * Is the Jenkins job a Pipeline job or a Freestyle job? * If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job? \ * If a Pipeline job, does the same problem exist if you use the {{deleteDir}} step? * Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace? * What git related options are included in the job definition? * Is there an anti-virus program running on the Windows computer?  If so, is the problem still visible if virus scanning is disabled for the workspace directory? I can't duplicate the problem in [my experiment|https://github.com/MarkEWaite/docker-lfs/blob/lts-with-plugins/ref/jobs/Bugs-Individual/jobs/JENKINS-55836-windows-wipe-workspace/config.xml].  I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Please [~carsten_hilber], please provide more details to duplicate the problem.   A detailed set of steps which others can use to see the problem is preferred.  If that is not possible, then please answer the following questions:

* What is {{WS-CLEANUP}} and how is it used in that job? * Is the Jenkins job a Pipeline job or a Freestyle job? * If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job? * If a Pipeline job, does the same problem exist if you use the {{deleteDir}} step? * Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace? * What git related options are included in the job definition? * Is there an anti-virus program running on the Windows computer?  If so, is the problem still visible if virus scanning is disabled for the workspace directory? I can't duplicate the problem in [my experiment|https://github.com/MarkEWaite/docker-lfs/blob/lts-with-plugins/ref/jobs/Bugs-Individual/jobs/JENKINS-55836-windows-wipe-workspace/config.xml].  I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

Add Comment

[chilber@agile-im.de (JIRA)]

Carsten Hilber commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

The Job is a freestyle job. The option I set is called 'Delete workspace before build starts', within Build Environment. It is configured to use the command line git. There are no more options given to the git task, only auth and branch is set. I additionally set the "Wipe out repository and force clone" option in git and i saw the same behavior as before. Then I set the wipe out exclusive, without the cleanup task and I got a different error message from git: Started by user Carsten Hilber Building remotely on webserv_runner_01 (webserv_runner) in workspace C:\vhvEntw\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master Wiping out workspace first. java.nio.file.FileSystemException: C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) at java.nio.file.Files.deleteIfExists(Unknown Source) at hudson.Util.tryOnceDeleteFile(Util.java:316) at hudson.Util.deleteFile(Util.java:272) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from wxp-turm-p13.ads.de/10.xx.xx.161:1697 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserResponse.retrieve(UserRequest.java:389) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.FilePath.deleteContents(FilePath.java:1283) at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:30) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1183) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1810) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused: java.io.IOException: Unable to delete 'C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts. at hudson.Util.deleteFile(Util.java:277) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.access$1800(FilePath.java:213) at hudson.FilePath$DeleteContents.invoke(FilePath.java:1289) at hudson.FilePath$DeleteContents.invoke(FilePath.java:1285) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang.Thread.run(Unknown Source) Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? For example, C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\report.xml is 3 Minuten 13 Sekunden old Starting import task... Import Cucumber features Task started...

Add Comment

[chilber@agile-im.de (JIRA)]

Carsten Hilber edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

The Job is a freestyle job. The option I set is called 'Delete workspace before build starts', within Build Environment. It is configured to use the command line git. There are no more options given to the git  task, only auth and branch is set. I additionally set the "Wipe out repository and force clone" option in git and i saw the same behavior as before. Then I set the wipe out exclusive, without the cleanup task and I got a different error message from git:

{code:java}

vhvEntw\ Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master

{code}

Add Comment

[nelsonaaronm@gmail.com (JIRA)]

Aaron Nelson commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.   Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes "/.git/*"

Add Comment

[nelsonaaronm@gmail.com (JIRA)]

Aaron Nelson edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.

Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes " * * /.git/* *"

Add Comment

[nelsonaaronm@gmail.com (JIRA)]

Aaron Nelson edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.   Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes

{code:java} * * /.git/* *{code}

Add Comment

[bart.devriendt@kbc.be (JIRA)]

Bart Devriendt commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

We are having this issue as well since we upgraded from 3.8.0 to 3.9.1.  At random times we have workspaces that cannot be cleaned anymore and the only solution seems to be to restart the Jenkins agent service.  This only occurs on our windows slaves, Linux slaves are not impacted.

Add Comment

[bart.devriendt@kbc.be (JIRA)]

Bart Devriendt commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

The fact that cleanup workspace before build is mentioned here is not really a cause.  It's a symptom of a problem that occurs earlier.  If we do a build and then manually try to wipe the workspace we also get the same error. Also it seems like if you wait a while (somewhere between half an hour and 2 hours the lock is removed and the cleanup works again.

Add Comment

[nelsonaaronm@gmail.com (JIRA)]

Aaron Nelson commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

I can confirm with Bart, after a couple of hours the lock is gone and the workspace cleanup/deleting is acting as expected.

Add Comment

[kamalcastro@gmail.com (JIRA)]

Fidal Castro Arivalagan commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

We too noticed the same issue with Windows slaves under Jenkins version 2.150.3. Workaround mentioned by Aaron Nelson wasn't working because the next git clone identifies ".git" directory as suspicious. I am able to replicate this issue with "Freestyle" job with below configurations. SCM - Git checkout Build Environment - Delete workspace before build starts Build Trigger - For every 15 mins ( H/15 * * * * ( Build - This is not important for this testing. As Bart explained, we are not able to delete the workspace/directory manually as well. Only way is to restart the Jenkins slave process. Workaround: Use "Delete workspace when build is done" in post build section apart of Build Environment. But this doesn't suit some use cases though.   Why do Jenkins process use the workspace  even after the build completes ? I see especially all these cases are throwing error with .git/objects/pack/<sha file> is in use

Add Comment

[kamalcastro@gmail.com (JIRA)]

Fidal Castro Arivalagan edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

We too noticed the same issue with Windows slaves under Jenkins version 2.150.3. Workaround mentioned by Aaron Nelson wasn't working because the next git clone identifies ".git" directory as suspicious. I am able to replicate this issue with "Freestyle" job with below configurations.

# SCM - Git checkout # Build Environment - Delete workspace before build starts # Build Trigger - For every 15 mins ( H/15 * * * * ( # Build - This is not important for this testing.

As Bart explained, we are not able to delete the workspace/directory manually as well. Only way is to restart the Jenkins slave process. Workaround: Use "Delete workspace when build is done" in post build section apart of Build Environment. But this doesn't suit some use cases though.

Why do Jenkins process use the workspace  even after the build completes ? I see especially all these cases are throwing error with .git/objects/pack/<sha file> is in use

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

I can actually reproduce the bug on Git 3.8.0 and couldn't reproduce it on 3.0.5. I hope to bisect it to later versions to narrow the changes caused it

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

I can actually reproduce the bug on Git 3.8.0 and couldn't reproduce it on 3.0.5. I hope to bisect it to later versions to narrow the changes caused it . reproduced on git plugin 3.6.4

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

I can actually reproduce the bug on Git 3.8.0 and couldn't reproduce it on 3.0.5. I hope to bisect it to later versions to narrow the changes caused it.

reproduced on git I've isolated the bug to a change that happened between the 3.0.5(no bug) and 3.1.0 release It may be a case of a conflicting change with another plugin change. If anyone else can try testing 3. 6 0 . 4 5 fixes the issue it will be helpful

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

I looked at https://github.com/jenkinsci/git-plugin/compare/git-3.0.5...git-3.1.0 which led me to https://github.com/jenkinsci/git-client-plugin/compare/git-client-2.1.0...git-client-2.3.0  Mark Waite do you think it could be related to think it could be caused by https://github.com/jenkinsci/git-client-plugin/compare/git-client-2.1.0...git-client-2.3.0#diff-4a13310f7c84a9d6e133def48c0aed5a  thanks in advance!

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Idan Bidani I don't see anything in the differences you listed that would leave a file open. The git client sequence of changes added support for git large files, but I assume you're not using git large files and thus are not executing the new code paths.

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Per Mark Waite guidance and suggestions - big thanks! I've used Kosuke's tool https://file-leak-detector.kohsuke.org/ which helped me identify the open file handlers of the Jenkins node. This the stack trace the tool provided, notice com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor  I disabled this plugin and the issue is resolved now. #17 PATH_TOWORKSPACE\GitWinBug\.git\objects\pack\pack-1db6aa468a129acaa6b8055b79680fec66793541.pack by thread:pool-1-thread-7 for JNLP4-connect connection to JENKINS_URL/IP_ADDR:60048 id=1013 on Sat Mar 16 21:44:38 EDT 2019#17 PATH_TOWORKSPACE\GitWinBug\.git\objects\pack\pack-1db6aa468a129acaa6b8055b79680fec66793541.pack by thread:pool-1-thread-7 for JNLP4-connect connection to JENKINS_URL/IP_ADDR:60048 id=1013 on Sat Mar 16 21:44:38 EDT 2019 at java.io.RandomAccessFile.<init>(Unknown Source) at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:643) at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:625) at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:295) at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:379) at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:184) at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:360) at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:259) at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:601) at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:773) at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:285) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:486) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:444) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:435) at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165) at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236) at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:890) at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:800) at com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128) at com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:105) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3087) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Unknown Source)

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani updated an issue

Jenkins / JENKINS-55836

Unable to delete workspace due to locked git .pack file

Change By: Idan Bidani Component/s: hp-application-automation-tools-plugin Component/s: git-plugin

Add Comment

[iplaman@gmail.com (JIRA)]

Idan Bidani commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file Assigned to hp-application-automation-tools-plugin. Workaround: Since I didn't need the HP plugin anymore I simply uninstalled it but for those which may still use it It could be worth trying to revert to 5.0 version(last known working version I used) For hp-application-automation-tools-plugin maintainers, it's recommended you either to place the repository manipulating calls inside try with resources, or switch (where feasible) to use calls to the git client plugin rather than calling directly to JGit.

Add Comment

[ben.ptacek@gmail.com (JIRA)]

Ben Ptacek commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

We also have a Windows node and we ran the fileleak and it is pointing to the same issue/plugin - com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128) We will report back if reverting helps.

Add Comment

[ben.ptacek@gmail.com (JIRA)]

Ben Ptacek updated an issue

Jenkins / JENKINS-55836

Unable to delete workspace due to locked git .pack file

Change By: Ben Ptacek Attachment: image-2019-03-21-15-40-39-729.png

Add Comment

[ben.ptacek@gmail.com (JIRA)]

Ben Ptacek commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file Okay, the revert could only go to 5.6.2 of  the Microfocus plugin.  The issue is when you have the Delete Workspace selected (see pic below for where this option is set).  If I unselect it, stop the slave service, manually clean the workspace, start the service and run the job, then I am able to run the job successfully thereafter.

Add Comment

[ben.ptacek@gmail.com (JIRA)]

Ben Ptacek edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Okay, the revert could only go to 5.6.2 of  the Microfocus plugin and this did not help .  The issue is when you have the Delete Workspace selected (see pic below for where this option is set).  If I unselect it, stop the slave service, manually clean the workspace, start the service and run the job, then I am able to run the job successfully thereafter. !image-2019-03-21-15-40-39-729.png!

Add Comment

[christopher.smith@sungardas.com (JIRA)]

Christopher Smith commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....? /.git/* exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job [WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE /.git/* exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE

Add Comment

[christopher.smith@sungardas.com (JIRA)]

Christopher Smith edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....?

{panel :title= } * * /.git/** exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job } *

[WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE

{panel} {panel :title= } * * /.git/** exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job } *

[WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE

{panel}

Add Comment

[christopher.smith@sungardas.com (JIRA)]

Christopher Smith edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....?

Note: The below should says */.git/** but the second asterisk isn't coming through. {panel} **/.git/** exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job*

[WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE {panel} {panel}

**/.git/** exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job*

[WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE {panel}

Add Comment

[christopher.smith@sungardas.com (JIRA)]

Christopher Smith edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....? Note: The below should says

{code:java} */.git/**   {code}   but the second asterisk isn't coming through.

{panel} **/.git/** exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job* [WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE {panel} {panel} **/.git/** exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job* [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE {panel}

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Christopher Smith you don't mention in your description if you are using the hp-automation-tools-plugin. If you are not using the hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause. If you are not using the hp-automation-tools-plugin, please submit a separate bug report with the details of your environment and your best guess at the steps that will allow others to recreate the problem.

Add Comment

[christopher.smith@sungardas.com (JIRA)]

Christopher Smith edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

***It turned out that we had used a different user than the user account the Jenkins slave service was using when troubleshooting/working with the Jenkins workspace directory. Despite both accounts being members of the Administrators group it seems everything needed to be owned/created/modified using the same account***

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront assigned an issue to Radi Berkovich

Jenkins / JENKINS-55836

Unable to delete workspace due to locked git .pack file

Change By: Daniel Gront Assignee: Radi Berkovich

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront updated an issue

Jenkins / JENKINS-55836 Unable to delete workspace due to locked git .pack file

Change By: Daniel Gront Labels: git git-plugin octane

Add Comment

[bowzoo@gmail.com (JIRA)]

Zoe Chao commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file It happens to us as well. We are using Workspace Cleanup  plugin 0.37  Jenkins 3.183.3 Freestyle

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Radi Berkovich, was it fixed in the latest version of the plugin?

Add Comment

[bowzoo@gmail.com (JIRA)]

Zohan edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

It happens to us as well.

* We are using Workspace Cleanup  plugin 0.37  * Jenkins 3.183.3 Freestyle  It keeps running the same error ``` Triggered by Benson GitLab Merge Request #181: omg/feature/gateway/IotRequestServer_v2 => master Building in workspace /var/lib/jenkins/workspace/omg_smoke_test [WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete '/var/lib/jenkins/workspace/omg_smoke_test'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete '/var/lib/jenkins/workspace/omg_smoke_test'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. Finished: FAILURE ``` Util the other user kick a new job from different repo.

Add Comment

[moneysc27@gmail.com (JIRA)]

Manish Christian commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Radi Berkovich do we have any updates on this? I can confirm that workspace is not getting deleted because of locked .pack file.

Add Comment

[radislav.berkovich@hpe.com (JIRA)]

Radi Berkovich commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Hi. I see here several directions 1.first of all, please validate that you use hp-automation-tools-plugin. if so , we did several improvements in the area,  Please try to use our last beta version [5.8.5-beta|https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/hp-application-automation-tools-plugin/5.8.5-beta/hp-application-automation-tools-plugin-5.8.5-beta.hpi.] 2.if you don't use  hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause 3.inthe beginning of the thread I saw the following : Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD Root problem of this is locks of git and its not related to  hp-automation-tools-plugin plugin.

Add Comment

[radislav.berkovich@hpe.com (JIRA)]

Radi Berkovich edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Hi. I see here several directions 1.first of all, please validate that you use hp-automation-tools-plugin. if so , we did several improvements in the area,

Please try to use our last beta version [5.8.5-beta|[https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/hp-application-automation-tools-plugin/5.8.5-beta/hp-application-automation-tools-plugin-5.8.5-beta.hpi].]

2.if you don't use  hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause

3. inthe in the beginning of the thread I saw the following : Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD

Root problem of this is locks of git and its not related to  hp-automation-tools-plugin plugin.

Add Comment

[moneysc27@gmail.com (JIRA)]

Manish Christian commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Radi Berkovich, Thank you so much!!! I can confirm that 5.8.5-beta have fixed workspace delete issue. We are on latest 5.8 version. Any plan on releasing stable version with the fix?

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Hey Manish Christian, our release process takes some time in order to ensure that everything works correctly and no bugs are found. We are hoping to release a stable version in October.

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Hey [~manishchristian], our release process takes some time in order to ensure that everything works correctly and no bugs are found. We are hoping to release a stable version in October. I am moving this issue to fixed.

Add Comment

[daniel.gront@microfocus.com (JIRA)]

Daniel Gront updated JENKINS-55836

Jenkins / JENKINS-55836

Unable to delete workspace due to locked git .pack file

Change By: Daniel Gront Status: Open Fixed but Unreleased Resolution: Fixed

Add Comment

[al.nik.kovaliov@gmail.com (JIRA)]

Aliaksandr Kavaliou commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file Hello, Can please someone tell the ETA when the fix will be released? Thanks in advance.

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[jan.paul.van.der.jagt@achmea.nl (JIRA)]

Jan Paul van der Jagt commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Is this defenitly solved in the 5.9 version?

Add Comment

[jan.paul.van.der.jagt@achmea.nl (JIRA)]

Jan Paul van der Jagt edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Is this defenitly [definitely|https://www.google.com/search?sxsrf=ACYBGNRxPXDfhRBwjouKfslftgkzGRwi3w:1572443979037&q=definitely&spell=1&sa=X&ved=0ahUKEwj7ybKmksTlAhURblAKHbjSBUkQkeECCC4oAA] *__*  solved in the 5.9 version?

Add Comment

[jan.paul.van.der.jagt@achmea.nl (JIRA)]

Jan Paul van der Jagt edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Is this [definitely|https://www.google.com/search?sxsrf=ACYBGNRxPXDfhRBwjouKfslftgkzGRwi3w:1572443979037&q=definitely&spell=1&sa=X&ved=0ahUKEwj7ybKmksTlAhURblAKHbjSBUkQkeECCC4oAA]  *__*  solved in the 5.9 version?

Add Comment

[jan.paul.van.der.jagt@achmea.nl (JIRA)]

Jan Paul van der Jagt edited a comment on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Is this [ definitely |https://www.google.com/search?sxsrf=ACYBGNRxPXDfhRBwjouKfslftgkzGRwi3w:1572443979037&q=definitely&spell=1&sa=X&ved=0ahUKEwj7ybKmksTlAhURblAKHbjSBUkQkeECCC4oAA]  solved in the 5.9 version?

Add Comment

[peter.willekens@kbc.be (JIRA)]

Peter Willekens commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

When will this fix actually be Released ? As far as I can see it was not in 5.9.

Add Comment

[radislav.berkovich@microfocus.com (JIRA)]

Radi Berkovich commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

In versions 5.9 we did several related fixes in the area. Version 5.9 was released in september.

Add Comment

[peter.willekens@kbc.be (JIRA)]

Peter Willekens commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

Can you confirm this beta fix was included in version 5.9 ? Status of this issue suggests otherwise ...

Add Comment

[radislav.berkovich@microfocus.com (JIRA)]

Radi Berkovich commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file

included, I also will change status of issue.

Add Comment

[radislav.berkovich@microfocus.com (JIRA)]

Radi Berkovich updated JENKINS-55836

Jenkins / JENKINS-55836

Unable to delete workspace due to locked git .pack file

Change By: Radi Berkovich Status: Fixed but Unreleased Closed

Add Comment

[peter.willekens@kbc.be (JIRA)]

Peter Willekens commented on JENKINS-55836

Re: Unable to delete workspace due to locked git .pack file OK, thx for the confirmation.

Add Comment