jenkins - multi-node setup is becoming very unstable

[dpreilan]

I am at 1.436 now. I have a master node and 3 other slave nodes. Was
working fine for at least a year, but I really don't know when things
started breaking. Maybe something after 1.410.

All my nodes share an nfs mount point, and do all their work there.

I am seeing alot of job failures now due to java/jenkins exceptions.
They always seen to be around 2 areas:
1) inject environment variables from a file
2) copying artifacts from another job

I might have seen it during the saving artifacts phase of jobs as
well.

Exceptions are java.io.channel related. I will include a stack track
when I get back to work. google bought up disk space issues and java
version issues, but I have ruled those out.

I am now running degraded because I have to delete all my slave nodes
to keep things running.

[Ferenc Kovacs]

seems like a file system(nfs) issue to me.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

[dpreilan]

The following backtrace is usually what I see.

ERROR: Failed to copy artifacts from xenv4-kernel/ARCH=ia32 with
filter: **
hudson.util.IOException2: java.lang.NoClassDefFoundError: hudson/
remoting/FastPipedOutputStream
at hudson.FilePath.copyRecursiveTo(FilePath.java:1591)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1516)
at
hudson.plugins.copyartifact.FilePathCopyMethod.copyAll(FilePathCopyMethod.java:
51)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:
226)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:
200)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild
$AbstractRunner.perform(AbstractBuild.java:694)
at hudson.model.Build$RunnerImpl.build(Build.java:178)
at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:
460)
at hudson.model.Run.run(Run.java:1404)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:
88)
at hudson.model.Executor.run(Executor.java:230)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NoClassDefFoundError: hudson/remoting/FastPipedOutputStream
at hudson.remoting.Channel$2.adapt(Channel.java:714)
at hudson.remoting.Channel$2.adapt(Channel.java:709)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1589)
... 13 more
Caused by: java.lang.NoClassDefFoundError: hudson/remoting/
FastPipedOutputStream
at hudson.remoting.Pipe.readObject(Pipe.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
at hudson.remoting.UserRequest.perform(UserRequest.java:98)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException:
hudson.remoting.FastPipedOutputStream
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 28 more
Caused by: java.util.zip.ZipException: error reading zip file
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1200(Unknown Source)
at java.util.zip.ZipFile$ZipFileInputStream.read(Unknown Source)
at java.util.zip.ZipFile$1.fill(Unknown Source)
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.util.jar.JarVerifier$VerifierStream.read(Unknown Source)
at sun.misc.Resource.getBytes(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$000(Unknown Source)
... 34 more
Build step 'Copy artifacts from another project' marked build as
failure