|
For the last two weeks, I have been seeing an intermittent failure during the clearcase stage of our builds. It throws a Java exception along the lines of
FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
etc...
It always happens during the Clearcase segment of the build, but on a different step each time. Sometimes after lsview, sometimes after lshistory, sometimes after mkview ...
It happens a very low percentage of the time, and it happens on a different project each time. I have around 300 projects running between a number of matrix jobs- we are seeing one or two failures on each runthrough of this, always on different jobs and on different slaves. It started a couple weeks ago- our clearcase server crashed, and jenkins was having some trouble interacting with the server. I upgraded to 1.563 to try and solve that problem (I wound up having to manually delete all the views that had been created in the period between the backup and the crash). Once that issue had been resolved though, we started seeing this problem.
I upgraded the clearcase plugin to 1.5, and then tried downgrading Jenkins to 1.554.2. I cannot swear to it, but I thought I was on 1.558 before the first problem popped up, and had been very stable for a long time with all build failures being attributed to bad code, so suddenly having Jenkins fail some is kind of jarring.
At this point, I am not sure what I changed to cause the problem. I remember I had to upgrade slave.jar to version 2.41 from whatever it was before when I did the first upgrade to 1.563. Other then that, nothing else substantial has changed. The Jenkins is on a restricted network with no internet access- I am going to type the stack trace below, so forgive any typos.
FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
at hudson.remoting.ExportTable.get(ExportTable.java:261)
at hudson.remoting.Channel.getExportedObject(Channel.java:596)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:292
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:281)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:240)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(UNKNOWN SOURCE)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Cased by: java.lang.Exception: Object was recently deallocated
#18649 (ref.0) : hudson.Launcher$RemoteLaunchCallable$1
created at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86)
at hudson.remoting.ExportTable.export(ExportTable.java:239)
at hudson.remoting.Channel.export(Channel.java:592)
at hudson.remoting.Channel.export(Channel.java:583)
at hudson.remoting.Channel.export(Channel.java:553)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1025)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:990)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270)
... 15 more
Caused by: Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
|
Major
