Dead executors

[Mihaela-Elena Velimiroviciu]

Hi,

We had many dead executors today, related to not enough space left.

Here is how the logs looked like:

Unexpected executor death
java.io.IOException: Failed to create a temporary file in /home/mozauto/mozmill-ci/jenkins-master/jobs/mozilla-aurora_update
    at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
    at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
    at hudson.util.TextFile.write(TextFile.java:118)
    at hudson.model.Job.saveNextBuildNumber(Job.java:270)
    at hudson.model.Job.assignBuildNumber(Job.java:328)
    at hudson.model.Run.<init>(Run.java:286)
    at hudson.model.AbstractBuild.<init>(AbstractBuild.java:167)
    at hudson.model.Build.<init>(Build.java:92)
    at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:34)
    at sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:186)
    at hudson.model.AbstractProject.newBuild(AbstractProject.java:995)
    at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1188)
    at hudson.model.AbstractProject.createExecutable(AbstractProject.java:143)
    at hudson.model.Executor.run(Executor.java:213)
Caused by: java.io.IOException: No space left on device
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at java.io.File.createTempFile(File.java:1989)
    at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
    ... 16 more

The jenkins master node had no free space left. After clearing the disk, we restarted the affected threads and they started running.