jacoco zombie processes
aher...@gmail.com
On my CI server (Jenkins) I often find some abandoned process created by jacoco (used by sonar):
jenkins 18076 1 91 Oct15 ? 10:52:23 /srv/jenkins/tools/JDK/Sun_Java_SDK_1.6.0_64bits/jre/bin/java -javaagent:/tmp/jenkins/jacocoagent2829792389611507959.jar=destfile=target/jacoco.exec,excludes=*_javassist_* -jar /tmp/jenkins/surefirebooter39768686747210922.jar /tmp/jenkins/surefire7158876949801788984tmp /tmp/jenkins/surefire2676082484321574815tmp
As you can see such processes are taking all the CPU and have lost the parent process.
I put the thread dump at the end of the email if you want to have a look at it.
Is there any known bug with such problem ?
Cheers
Arnaud
root@swf06:~# /srv/jenkins/tools/JDK/Sun_Java_SDK_1.6.0_64bits/bin/jstack -F 18076
Attaching to process ID 18076, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.9-b04
Deadlock Detection:
java.lang.NullPointerException
at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426)
at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137)
at sun.jvm.hotspot.oops.Oop.isA(Oop.java:100)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Can't print deadlocks:null
Thread 18077: (state = BLOCKED)
Thread 19214: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.exoplatform.services.jcr.impl.proccess.WorkerThread.run() @bci=23, line=55 (Interpreted frame)
Thread 19213: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Interpreted frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 19136: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.commons.collections.buffer.BlockingBuffer.remove() @bci=37, line=191 (Interpreted frame)
- org.exoplatform.services.jcr.impl.core.query.lucene.IndexMerger.run() @bci=46, line=297 (Interpreted frame)
Thread 18799: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.commons.collections.buffer.BlockingBuffer.remove() @bci=37, line=191 (Interpreted frame)
- org.exoplatform.services.jcr.impl.core.query.lucene.IndexMerger.run() @bci=46, line=297 (Interpreted frame)
Thread 18694: (state = BLOCKED)
- java.util.TimerThread.mainLoop() @bci=0, line=480 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 18434: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations() @bci=32, line=504 (Interpreted frame)
- com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run() @bci=96, line=74 (Interpreted frame)
Thread 18433: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run() @bci=154, line=98 (Interpreted frame)
Thread 18238: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame)
- com.arjuna.ats.internal.arjuna.recovery.Listener.run() @bci=20, line=118 (Interpreted frame)
Thread 18218: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
- java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 18190: (state = BLOCKED)
- java.io.RandomAccessFile.writeLong(long) @bci=100, line=1008 (Interpreted frame)
- org.hsqldb.persist.LockFile.writeHeartbeat() @bci=151 (Interpreted frame)
- org.hsqldb.persist.LockFile.access$000(org.hsqldb.persist.LockFile) @bci=5 (Interpreted frame)
- org.hsqldb.persist.LockFile$HeartbeatRunner.run() @bci=17 (Interpreted frame)
- org.hsqldb.lib.HsqlTimer$TaskRunner.run() @bci=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 18181: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.exoplatform.services.jcr.impl.proccess.WorkerThread.run() @bci=23, line=55 (Interpreted frame)
Thread 18089: (state = BLOCKED)
Thread 18088: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 18087: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)
Marc R. Hoffmann
I'm not aware to such issue. From the stack trace you can see several
threads. If one of the the threads is not a deamon thread the JVM will
not terminate. What looks strange is that all threads are in BLOCKED state.
Why do you think this problem is related to JaCoCo? This might happen
with any test execution.
Best regards,
-marc