Windows agent can't delete workspace
915 views
Skip to first unread message
Sean Talts
Mar 17, 2018, 10:22:57 AM3/17/18
to Jenkins Users
Hey all,
I'm having this sudden crazy problem where my Windows agent can't delete some files in its workspace anymore, failing all builds. It's getting a java.nio.file.AccessDeniedException on some files that it created. You can see the full exceptions at the end of the log here: http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console also copied below.
I've tried so many things, including switching JVMs on master and agent, upgrading and switching to LTS, uninstalling and reinstalling a new agent service in a new working directory… Any tips or advice would be extremely appreciated; been trying to solve this all day and night since it started Thursday when the master rebooted during a job on the Windows agent.
Thanks in advance,
Sean
Full error:
java.nio.file.AccessDeniedException: C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) at java.nio.file.Files.deleteIfExists(Unknown Source) at hudson.Util.tryOnceDeleteFile(Util.java:297) at hudson.Util.deleteFile(Util.java:253) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from gelman-group-win.stat.columbia.edu/128.59.76.64:50229 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) at hudson.remoting.Channel.call(Channel.java:952) at hudson.FilePath.act(FilePath.java:998) at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.deleteRecursive(FilePath.java:1192) at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused: java.io.IOException: Unable to delete 'C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteFile(Util.java:258) at hudson.FilePath.deleteRecursive(FilePath.java:1225) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) at hudson.FilePath.deleteRecursive(FilePath.java:1216) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) at hudson.FilePath.deleteRecursive(FilePath.java:1216) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) at hudson.FilePath.deleteRecursive(FilePath.java:1216) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) at hudson.FilePath.deleteRecursive(FilePath.java:1216) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) at hudson.FilePath.deleteRecursive(FilePath.java:1216) at hudson.FilePath.access$1100(FilePath.java:208) at hudson.FilePath$13.invoke(FilePath.java:1195) at hudson.FilePath$13.invoke(FilePath.java:1192) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) at hudson.remoting.UserRequest.perform(UserRequest.java:210) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:364) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang.Thread.run(Unknown Source) Also: java.nio.file.AccessDeniedException: C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\stanc_helper.stan at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source) at java.nio.file.spi.FileSystemProvider.newOutputStream(Unknown Source) at java.nio.file.Files.newOutputStream(Unknown Source) at hudson.util.IOUtils.copy(IOUtils.java:42) at hudson.FilePath.readFromTar(FilePath.java:2363) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from gelman-group-win.stat.columbia.edu/128.59.76.64:50229 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) at hudson.remoting.Channel.call(Channel.java:952) at hudson.FilePath.act(FilePath.java:998) at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.untar(FilePath.java:540) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused: java.io.IOException: Failed to extract StanSetup.tar.gz at hudson.FilePath.readFromTar(FilePath.java:2373) at hudson.FilePath.access$400(FilePath.java:208) at hudson.FilePath$4.invoke(FilePath.java:542) at hudson.FilePath$4.invoke(FilePath.java:540) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) at hudson.remoting.UserRequest.perform(UserRequest.java:210) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:364) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang.Thread.run(Unknown Source) Caused: java.io.IOException: remote file operation failed: C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from gelman-group-win.stat.columbia.edu/128.59.76.64:50229 at hudson.FilePath.act(FilePath.java:1005) at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.untar(FilePath.java:540) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) Caused: java.io.IOException: remote file operation failed: C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from gelman-group-win.stat.columbia.edu/128.59.76.64:50229 at hudson.FilePath.act(FilePath.java:1005) at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.deleteRecursive(FilePath.java:1192) at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Christian Gagneraud
Mar 17, 2018, 6:15:35 PM3/17/18
to jenkins...@googlegroups.com
On 18 March 2018 at 03:21, Sean Talts <sean....@columbia.edu> wrote:
> Hey all,
>
> I'm having this sudden crazy problem where my Windows agent can't delete
> some files in its workspace anymore, failing all builds. It's getting a
> java.nio.file.AccessDeniedException on some files that it created. You can
> see the full exceptions at the end of the log here:
> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console also
> copied below.
>
> I've tried so many things, including switching JVMs on master and agent,
> upgrading and switching to LTS, uninstalling and reinstalling a new agent
> service in a new working directory… Any tips or advice would be extremely
> appreciated; been trying to solve this all day and night since it started
> Thursday when the master rebooted during a job on the Windows agent.
We have thins kind of errors on Windows when there are still running
> Hey all,
>
> I'm having this sudden crazy problem where my Windows agent can't delete
> some files in its workspace anymore, failing all builds. It's getting a
> java.nio.file.AccessDeniedException on some files that it created. You can
> see the full exceptions at the end of the log here:
> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console also
> copied below.
>
> I've tried so many things, including switching JVMs on master and agent,
> upgrading and switching to LTS, uninstalling and reinstalling a new agent
> service in a new working directory… Any tips or advice would be extremely
> appreciated; been trying to solve this all day and night since it started
> Thursday when the master rebooted during a job on the Windows agent.
processes that have a file descriptor opened.
You could log on to the machine and check if there are any "zombie" processes.
Chris
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/6eaa7b8d-d69b-4c5e-a818-2a94643ce96d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Benjamin Lau
Mar 18, 2018, 2:43:56 AM3/18/18
to Jenkins Users
We've used scripting around the sysinternals handle tool[1] to make
sure that everything which is touching files in the workspace is
actually terminated before trying to delete the workspace. We were
having more issues with this when trying to delete workspaces at the
ends of jobs so in many cases we just let it be and only delete the
workspace on the next run.
Cheers,
Ben
[1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTjt-yWgMg58S%3DQNV7wy9r84J4hr270rjqTU-ck0sps%2BpA%40mail.gmail.com.
sure that everything which is touching files in the workspace is
actually terminated before trying to delete the workspace. We were
having more issues with this when trying to delete workspaces at the
ends of jobs so in many cases we just let it be and only delete the
workspace on the next run.
Cheers,
Ben
[1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle
Sean Talts
Mar 18, 2018, 7:39:53 PM3/18/18
to Jenkins Users
Thank you both! I particularly like the idea of just deleting before the job instead of after. I managed to get something else working too, though I think it could be masking the problem. I now run attrib -r -s /s /d before deleting (before the run and after, just in case). Seems to be pretty quick?
A run with this: http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55
A run with this: http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55
full Jenkinsfile: https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/Jenkinsfile
Thanks again!
Mark Waite
Mar 18, 2018, 10:06:04 PM3/18/18
to jenkins...@googlegroups.com
On Sun, Mar 18, 2018 at 5:40 PM Sean Talts <sean....@columbia.edu> wrote:
Thank you both! I particularly like the idea of just deleting before the job instead of after. I managed to get something else working too, though I think it could be masking the problem. I now run attrib -r -s /s /d before deleting (before the run and after, just in case). Seems to be pretty quick?
A run with this: http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55full Jenkinsfile: https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/JenkinsfileThanks again!
If you're willing to use an "attrib" command, then you might also consider "robocopy /mir" from an empty directory to the workspace directory. I had problems deleting directories when they contained a directory structure that was too long. Using "robocopy /mir" from an empty directory to the destination directory was able to delete those too long files when the Windows "rmdir" command would not.
I preferred that solution in my case because Robocopy is a standard part of Windows 7 and Windows 10.
Mark Waite
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3fce1831-0a1f-4a66-82d1-6180b5718b98%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages