jenkins 1.408 server hangs
Russ Tremain
been having trouble with jenkins hanging on the web server (tomcat).
today I manage to capture activity while the server was in the hung
state.
something is preventing it from serving web requests, sometimes for
several minutes.
seems to me that jenkins is getting out of sync with the slave nodes.
I am running 0.16 ssh plugin.
at the time of this hang I could ssh in/out of the remote nodes
without problem.
It eventually recovers, but this is happening several times a day now.
Any clues? Why would slave activity effect the main service threads?
tia,
-Russ
INFO: hud590 #3518 main build action completed: SUCCESS
Jun 24, 2011 5:21:07 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel kona
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:998)
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:992)
Jun 24, 2011 5:21:07 PM hudson.remoting.Request$2 run
SEVERE: Failed to send back a reply
java.io.IOException: SSH channel is closed. (Close requested by remote)
at
com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
at
com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747)
at
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1249)
at
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1203)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1387)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at
java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1538)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at hudson.remoting.Channel.send(Channel.java:484)
at hudson.remoting.Request$2.run(Request.java:282)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Jun 24, 2011 5:21:07 PM hudson.remoting.Request$2 run
SEVERE: Failed to send back a reply
java.io.IOException: SSH channel is closed. (Close requested by remote)
at
com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
at
com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747)
at
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1249)
at
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1203)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1387)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at
java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1538)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at hudson.remoting.Channel.send(Channel.java:484)
at hudson.remoting.Request$2.run(Request.java:282)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Jun 24, 2011 5:21:07 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel.
Jun 24, 2011 5:21:17 PM hudson.model.AbstractBuild$AbstractRunner
performAllBuildSteps
WARNING: Publisher hudson.tasks.Mailer aborted due to exception
java.lang.NullPointerException
at hudson.tasks.MailSender.createFailureMail(MailSender.java:271)
at hudson.tasks.MailSender.getMail(MailSender.java:146)
at hudson.tasks.MailSender.execute(MailSender.java:94)
at hudson.tasks.Mailer.perform(Mailer.java:101)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at
hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:649)
at
hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:625)
at
hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:603)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at
hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:572)
at hudson.model.Run.run(Run.java:1386)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
Jun 24, 2011 5:22:38 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect express
Jun 24, 2011 5:22:38 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect kona
Jun 24, 2011 5:22:43 PM hudson.model.Run run
INFO: hud630 #39 main build action completed: SUCCESS
Jun 24, 2011 5:23:12 PM hudson.model.Run run
INFO: hud610 #789 main build action completed: SUCCESS
Jun 24, 2011 5:24:24 PM hudson.model.Run run
INFO: hud630promote #23 main build action completed: SUCCESS