slave tries to delete / after upgrading to LTS 2.32.2

[Jifeng Zhang]

Hi,

We got a lot of following error msg from slave log after upgrading our Jenkins to 2.32.2, git-client to 2.2.1, git plugin to 3.0.1, scm api plugin to 2.0

Feb 09, 2017 7:19:42 AM hudson.Util makeWritable

INFO: Failed to chmod /

No directory specified for FileSet.

at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:479)

at org.apache.tools.ant.taskdefs.ExecuteOn.runExec(ExecuteOn.java:370)

at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)

at org.apache.tools.ant.taskdefs.Chmod.execute(Chmod.java:181)

at hudson.Util.makeWritable(Util.java:314)

at hudson.Util.tryOnceDeleteFile(Util.java:283)

at hudson.Util.tryOnceDeleteRecursive(Util.java:364)

at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383)

at hudson.Util.deleteContentsRecursive(Util.java:223)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:455)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)

at hudson.remoting.UserRequest.perform(UserRequest.java:153)

at hudson.remoting.UserRequest.perform(UserRequest.java:50)

at hudson.remoting.Request$2.run(Request.java:336)

at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

It seems like jenkins slave or git client plugin tries to delete / recursively.

And there was no job running on this slave at the time these errors were logged. 

The result is a disaster, all files owned by the jenkins user were deleted, including its home directory.

Any jenkins expert out there knows why this could happen and how to prevent it happening again? Right now it seems to happen every two days, removing everything it can under /.

Best regards,

Jifeng Zhang

[Daniel Beck]

> On 09.02.2017, at 10:34, Jifeng Zhang <zjf...@gmail.com> wrote:
>
> Any jenkins expert out there knows why this could happen and how to prevent it happening again? Right now it seems to happen every two days, removing everything it can under /.

Is there a job with a custom workspace defined as / ?

What version did you upgrade from? I assume it's _not_ 2.32.1?

[Jifeng Zhang]

Well, it _was_ an upgrade from 2.32.1. We keep our jenkins update to date with LTS and it has never been a problem. Usually a few days after the official docker image is out, we would do an update.

We have went through the jobs and couldn't find any job with custom workspace /.

[Mark Waite]

The source code at that location is trying to clear a workspace before it clones into it.

I like Daniel's suggestion to check for a job which has defined a custom workspace explicitly as the root directory.  That seems the most likely cause (to me).

You might also check to see if someone used a symbolic link in one of your jobs and made that symoblic link point to the root directory.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5944224d-ba61-474d-904e-c95d29f4cdf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

> On 09.02.2017, at 16:01, Jifeng Zhang <zjf...@gmail.com> wrote:
>
> We have went through the jobs and couldn't find any job with custom workspace /.

Maybe a custom workspace like "${whatever}/" that is effectively / when "$whatever" doesn't exist? (Does custom workspace resolve build variables? Not sure right now, but in absence of better ideas…)

[Jifeng ZHANG]

Thank you Mark and Daniel, for the ideas.

I was able to go through every job, since it is a new jenkins setup with not that many jobs. _None_ of the jobs uses custom workspace at all.

I did find out in one of the jobs, when defining git advanced behaviour for cloning in to a sub directory of current workspace, there is something like "${NAMESPACE}/${PROJECT_NAME}". But this job has not been run during times the problem happened. And even if this statement is evaluated to "/", according to what is said in the tips popup, this will be relative to the job's workspace.

I tried to correlate the time when the slave tried to delete "/" and the time jobs were running on the same slave, there seems to be no correlation between them at all.

--

Vänliga hälsningar/Best regards,
Jifeng Zhang