HyperThreading OpenTSDB
295 views
Skip to first unread message
Sebastien Nahelou
Jun 6, 2012, 5:29:29 AM6/6/12
to open...@googlegroups.com
Hi.
I currently do some bench about OpenTSDB and i have a question:
Is OpenTSDB optimized for multi core ?
When i tried to put 12k metrics/sec, i have the following mpstats:
11:18:06 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:18:07 AM all 25.50 0.00 0.50 0.00 0.00 0.00 0.00 0.00 74.01
11:18:07 AM 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02
11:18:07 AM 1 0.98 0.00 0.98 0.98 0.00 0.00 0.00 0.00 97.06
11:18:07 AM 2 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:18:07 AM 3 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
So my second question is how many metrics per seconde can you put (at SU) for 1 TSDMain node ?
Best regards
I currently do some bench about OpenTSDB and i have a question:
Is OpenTSDB optimized for multi core ?
When i tried to put 12k metrics/sec, i have the following mpstats:
11:18:06 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:18:07 AM all 25.50 0.00 0.50 0.00 0.00 0.00 0.00 0.00 74.01
11:18:07 AM 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02
11:18:07 AM 1 0.98 0.00 0.98 0.98 0.00 0.00 0.00 0.00 97.06
11:18:07 AM 2 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:18:07 AM 3 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
So my second question is how many metrics per seconde can you put (at SU) for 1 TSDMain node ?
Best regards
Dave Barr
Jun 6, 2012, 2:20:12 PM6/6/12
to Sebastien Nahelou, open...@googlegroups.com
Are you doing 'put''s via the socket or are you doing a tsdb import?
Is this one metric or a bunch of metrics? We've been able to easily
push 12k puts/sec per box (or steady state is around 4k per box at the
moment). Imports are faster, partly because it does some less
verification of the data (with respect to things like time ordering),
but also it can batch huge write requests.
We're generally bound by the write speed of HBase. Remember that the
metric ID is part of the row key, so a big import of one metric will
be hitting only one regionserver. To do a real perf test, make sure
you have lots of different metrics and that you have pre-split the
table so that data is going to as many regionservers as possible.
But yes, OpenTSDB uses asynchbase, which is highly threaded, When
doing big imports or when it's doing catchup after some maintenance,
I've seen tsdb happily consume all of its cores doing its work.
--Dave
Is this one metric or a bunch of metrics? We've been able to easily
push 12k puts/sec per box (or steady state is around 4k per box at the
moment). Imports are faster, partly because it does some less
verification of the data (with respect to things like time ordering),
but also it can batch huge write requests.
We're generally bound by the write speed of HBase. Remember that the
metric ID is part of the row key, so a big import of one metric will
be hitting only one regionserver. To do a real perf test, make sure
you have lots of different metrics and that you have pre-split the
table so that data is going to as many regionservers as possible.
But yes, OpenTSDB uses asynchbase, which is highly threaded, When
doing big imports or when it's doing catchup after some maintenance,
I've seen tsdb happily consume all of its cores doing its work.
--Dave
Sebastien Nahelou
Jun 7, 2012, 3:35:31 AM6/7/12
to open...@googlegroups.com, Sebastien Nahelou
Hi Dave,
Thanks for your answer.
Thanks for your answer.
Are you doing 'put''s via the socket or are you doing a tsdb import?
Is this one metric or a bunch of metrics?
It's a bunch of metrics.
I use this pluggin for tcollector for send "spam" metrics:
def main():
while True:
ts = int(time.time())
mpstats = subprocess.Popen(["mpstat -P ALL 1 1"], stdout=subprocess.PIPE, shell=True)
stdout, _ = mpstats.communicate()
if mpstats.returncode == 0:
for line in stdout.split("\n"):
fields = line.split()
if not line: continue
if fields[2].isdigit() or fields[2]== "all":
for i in range(1000):
print ("spam%d.iowait %d %s core=%s " % (i,ts, fields[6],fields[2]))
print ("spam%d.usr %d %s core=%s " % (i,ts, fields[3],fields[2]))
print ("spam%d.idle %d %s core=%s " % (i,ts,fields[11],fields[2]))
while True:
ts = int(time.time())
mpstats = subprocess.Popen(["mpstat -P ALL 1 1"], stdout=subprocess.PIPE, shell=True)
stdout, _ = mpstats.communicate()
if mpstats.returncode == 0:
for line in stdout.split("\n"):
fields = line.split()
if not line: continue
if fields[2].isdigit() or fields[2]== "all":
for i in range(1000):
print ("spam%d.iowait %d %s core=%s " % (i,ts, fields[6],fields[2]))
print ("spam%d.usr %d %s core=%s " % (i,ts, fields[3],fields[2]))
print ("spam%d.idle %d %s core=%s " % (i,ts,fields[11],fields[2]))
and i use this plugin to send see put/sec:
SLEEPTIME=1
METRIC="bench.tsdb.putMetricPerSec"
while true
do
l1=$(echo stats | nc -w 30 obench20s 4242 | grep "type=put host=obench20s")
sleep $SLEEPTIME
l2=$(echo stats | nc -w 30 obench20s 4242 | grep "type=put host=obench20s")
v1=$(echo $l1 | awk '{print $3}')
v2=$(echo $l2 | awk '{print $3}')
v3=$(echo "($v2 - $v1)/$SLEEPTIME" | bc)
date=$(date +%s)
echo $METRIC $date $v3
done
METRIC="bench.tsdb.putMetricPerSec"
while true
do
l1=$(echo stats | nc -w 30 obench20s 4242 | grep "type=put host=obench20s")
sleep $SLEEPTIME
l2=$(echo stats | nc -w 30 obench20s 4242 | grep "type=put host=obench20s")
v1=$(echo $l1 | awk '{print $3}')
v2=$(echo $l2 | awk '{print $3}')
v3=$(echo "($v2 - $v1)/$SLEEPTIME" | bc)
date=$(date +%s)
echo $METRIC $date $v3
done
So as you can see, for i in range(1000) i will have 3000 differents metrics per second with 4 differents tags (ALL, 0, 1, 2 ,3 for core). So i have 15000 differents values per second.
When i look the cpu and iowait curve, i see that only one core work at time (see attachment) (obench20s is the server wich host the region).
We're generally bound by the write speed of HBase.
The result of HBase bench is more than 9k sequential write with PerformanceEvaluation....
OpenTSDB uses asynchbase, which is highly threaded
So i don't understand why only 1 cpu works in my case :/
Le mercredi 6 juin 2012 20:20:12 UTC+2, Dave Barr a écrit :
Are you doing 'put''s via the socket or are you doing a tsdb import?
Is this one metric or a bunch of metrics? We've been able to easily
push 12k puts/sec per box (or steady state is around 4k per box at the
moment). Imports are faster, partly because it does some less
verification of the data (with respect to things like time ordering),
but also it can batch huge write requests.
We're generally bound by the write speed of HBase. Remember that the
metric ID is part of the row key, so a big import of one metric will
be hitting only one regionserver. To do a real perf test, make sure
you have lots of different metrics and that you have pre-split the
table so that data is going to as many regionservers as possible.
But yes, OpenTSDB uses asynchbase, which is highly threaded, When
doing big imports or when it's doing catchup after some maintenance,
I've seen tsdb happily consume all of its cores doing its work.
--Dave
On Wed, Jun 6, 2012 at 2:29 AM, Sebastien Nahelou
Sebastien Nahelou
Jun 7, 2012, 5:02:05 AM6/7/12
to open...@googlegroups.com, Sebastien Nahelou
I just try with LZO compression and just 5k insert (range(500). I have the same result:
That seems to be TSDMain the "problem" or my insert script. But I don't know why it continue to use only 1 core :/.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32185 root 20 0 2922m 73m 9.9m S 112.2 2.9 5:36.33 java
32185 net.opentsdb.tools.TSDMain
32185 root 20 0 2922m 73m 9.9m S 112.2 2.9 5:36.33 java
32185 net.opentsdb.tools.TSDMain
That seems to be TSDMain the "problem" or my insert script. But I don't know why it continue to use only 1 core :/.
Sebastien Nahelou
Jun 8, 2012, 9:51:27 AM6/8/12
to open...@googlegroups.com, Sebastien Nahelou
All thread seems to wait mutex....
#cat /proc/20097/status | grep Cpus_
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Cpus_allowed: ff
Cpus_allowed_list: 0-7
#jstack -J-d64 -m 20097
Attaching to process ID 20097, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.4-b02
Deadlock Detection:
No deadlocks found.
----------------- 20105 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4fe3415 _ZN7Threads10destroy_vmEv + 0x55
0x00007fe9f4cdf7eb jni_DestroyJavaVM + 0xcb
0x000000004000223f JavaMain + 0x23f
0x00000000d3a602a0 ????????
0x00000000d3a01798 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000001 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20106 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20107 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20108 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20109 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20110 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4ea6c9b _ZN7Monitor5IWaitEP6Threadl + 0x1fb
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f5031ee3 _ZN8VMThread4loopEv + 0xa3
0x00007fe9f5031ade _ZN8VMThread3runEv + 0x6e
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20111 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ec624a _ZN13ObjectMonitor4waitElbP6Thread + 0x60a
0x00007fe9f4fab2f3 _ZN18ObjectSynchronizer4waitE6HandlelP6Thread + 0x53
0x00007fe9f4d112d7 JVM_MonitorWait + 0x1e7
0x00007fe9ecabad6e * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.ref.Reference$ReferenceHandler.run() bci:46 line:116 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20112 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ec624a _ZN13ObjectMonitor4waitElbP6Thread + 0x60a
0x00007fe9f4fab2f3 _ZN18ObjectSynchronizer4waitE6HandlelP6Thread + 0x53
0x00007fe9f4d112d7 JVM_MonitorWait + 0x1e7
0x00007fe9ecabad6e * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:118 (Interpreted frame)
0x00007fe9ecaaf929 * java.lang.ref.ReferenceQueue.remove() bci:2 line:134 (Interpreted frame)
0x00007fe9ecaaf929 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20113 -----------------
0x00007fe9f59716c0 sem_wait + 0x30
0x00007fe9f4ece51c _Z19signal_thread_entryP10JavaThreadP6Thread + 0x5c
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20114 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4b6ae5f _ZN12CompileQueue3getEv + 0x15f
0x00007fe9f4b6cc91 _ZN13CompileBroker20compiler_thread_loopEv + 0x171
0x00007fe9f4fe4c99 _Z21compiler_thread_entryP10JavaThreadP6Thread + 0x9
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20115 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4b6ae5f _ZN12CompileQueue3getEv + 0x15f
0x00007fe9f4b6cc91 _ZN13CompileBroker20compiler_thread_loopEv + 0x171
0x00007fe9f4fe4c99 _Z21compiler_thread_entryP10JavaThreadP6Thread + 0x9
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20116 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4e46eb2 _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread + 0xc2
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20117 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4fdd15b _ZN13WatcherThread3runEv + 0xdb
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20121 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20122 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20123 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20124 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20128 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20138 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000d94367b8 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20140 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20150 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20164 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20168 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20248 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000d95da348 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20261 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4ed2fe5 _ZN2os5sleepEP6Threadlb + 0x145
0x00007fe9f4d267d8 JVM_Sleep + 0x338
0x00007fe9ecabad6e * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)
0x00007fe9ecbb75c8 * org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick() bci:55 line:455 (Compiled frame)
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000137cc592686 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20944 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000ffc18690 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20097 -----------------
0x00007fe9f596c04d pthread_join + 0x10d
0x00007fe9f4540700 ????????
0x00007fff7c82ff4c ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000011 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Attaching to process ID 20097, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.4-b02
Deadlock Detection:
No deadlocks found.
----------------- 20105 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4fe3415 _ZN7Threads10destroy_vmEv + 0x55
0x00007fe9f4cdf7eb jni_DestroyJavaVM + 0xcb
0x000000004000223f JavaMain + 0x23f
0x00000000d3a602a0 ????????
0x00000000d3a01798 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000001 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20106 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20107 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20108 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20109 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4c2a9b5 _ZN13GCTaskManager8get_taskEj + 0x75
0x00007fe9f4c2bcc3 _ZN12GCTaskThread3runEv + 0x123
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20110 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4ea6c9b _ZN7Monitor5IWaitEP6Threadl + 0x1fb
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f5031ee3 _ZN8VMThread4loopEv + 0xa3
0x00007fe9f5031ade _ZN8VMThread3runEv + 0x6e
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20111 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ec624a _ZN13ObjectMonitor4waitElbP6Thread + 0x60a
0x00007fe9f4fab2f3 _ZN18ObjectSynchronizer4waitE6HandlelP6Thread + 0x53
0x00007fe9f4d112d7 JVM_MonitorWait + 0x1e7
0x00007fe9ecabad6e * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.ref.Reference$ReferenceHandler.run() bci:46 line:116 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20112 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ec624a _ZN13ObjectMonitor4waitElbP6Thread + 0x60a
0x00007fe9f4fab2f3 _ZN18ObjectSynchronizer4waitE6HandlelP6Thread + 0x53
0x00007fe9f4d112d7 JVM_MonitorWait + 0x1e7
0x00007fe9ecabad6e * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:118 (Interpreted frame)
0x00007fe9ecaaf929 * java.lang.ref.ReferenceQueue.remove() bci:2 line:134 (Interpreted frame)
0x00007fe9ecaaf929 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20113 -----------------
0x00007fe9f59716c0 sem_wait + 0x30
0x00007fe9f4ece51c _Z19signal_thread_entryP10JavaThreadP6Thread + 0x5c
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20114 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4b6ae5f _ZN12CompileQueue3getEv + 0x15f
0x00007fe9f4b6cc91 _ZN13CompileBroker20compiler_thread_loopEv + 0x171
0x00007fe9f4fe4c99 _Z21compiler_thread_entryP10JavaThreadP6Thread + 0x9
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20115 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea71cd _ZN7Monitor4waitEblb + 0xed
0x00007fe9f4b6ae5f _ZN12CompileQueue3getEv + 0x15f
0x00007fe9f4b6cc91 _ZN13CompileBroker20compiler_thread_loopEv + 0x171
0x00007fe9f4fe4c99 _Z21compiler_thread_entryP10JavaThreadP6Thread + 0x9
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20116 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f4ea6b65 _ZN7Monitor5IWaitEP6Threadl + 0xc5
0x00007fe9f4ea732e _ZN7Monitor4waitEblb + 0x24e
0x00007fe9f4e46eb2 _ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread + 0xc2
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20117 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4fdd15b _ZN13WatcherThread3runEv + 0xdb
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20121 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20122 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20123 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20124 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20128 -----------------
0x00007fe9f54b62c3 __GI_epoll_wait + 0x33
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x000000000000000a with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20138 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000d94367b8 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20140 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20150 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20164 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20168 -----------------
0x00007fe9f596f3dc __pthread_cond_wait + 0xcc
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:156 (Interpreted frame)
0x00007fe9ecaafa82 * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:1987 (Interpreted frame)
0x00007fe9ecaaff5c * java.util.concurrent.ArrayBlockingQueue.take() bci:20 line:317 (Interpreted frame)
0x00007fe9ecaafe03 * java.util.concurrent.ThreadPoolExecutor.getTask() bci:78 line:947 (Interpreted frame)
0x00007fe9ecaaf929 * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:18 line:907 (Interpreted frame)
0x00007fe9ecaaff5c * java.lang.Thread.run() bci:11 line:662 (Interpreted frame)
0x00007fe9ecaaa438 <StubRoutines>
0x00007fe9f4ca4400 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1e0
0x00007fe9f4ed46d9 _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_ + 0x19
0x00007fe9f4ca3d46 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread + 0x126
0x00007fe9f4ca3df7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread + 0x47
0x00007fe9f4d379f0 _Z12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007fe9f4fde1a1 _ZN10JavaThread3runEv + 0x121
0x00007fe9f4ed5b8f _Z10java_startP6Thread + 0x13f
----------------- 20248 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000d95da348 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20261 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f4ed2fe5 _ZN2os5sleepEP6Threadlb + 0x145
0x00007fe9f4d267d8 JVM_Sleep + 0x338
0x00007fe9ecabad6e * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)
0x00007fe9ecbb75c8 * org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick() bci:55 line:455 (Compiled frame)
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000137cc592686 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20944 -----------------
0x00007fe9f596f75b __pthread_cond_timedwait + 0x13b
0x00007fe9f500aa78 Unsafe_Park + 0x188
0x00007fe9ecabad6e * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007fe9ecb41870 * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:196 (Compiled frame)
* java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:174 line:424 (Compiled frame)
0x00000000ffc18690 ????????
0xf3543af8d40ef768 ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000031 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 20097 -----------------
0x00007fe9f596c04d pthread_join + 0x10d
0x00007fe9f4540700 ????????
0x00007fff7c82ff4c ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x0000000000000011 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:478)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.sender(LinuxAMD64CFrame.java:59)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Igor Pomazanov
Jun 9, 2012, 8:30:25 PM6/9/12
to Sebastien Nahelou, open...@googlegroups.com, Sebastien Nahelou
There does not seem to be anything in OpenTSDB code itself that would cause it to be limited to using one core only. In my case it was happily consuming all cores available. You may want to look into your jvm/OS settings for explanation of this behavior.
Mutex waiting is probably just indication of waiting for response from Hbase.
Igor
Sebastien Nahelou
Jun 11, 2012, 3:59:03 AM6/11/12
to open...@googlegroups.com, Sebastien Nahelou
Hi,
If OpenTSDB wait an HBase answer, why only 1 thread is used ?
I read the jmv doc and i have not found options for limit (or unlimit) the number of thread.
I run on CentOS 6.2:
#more /proc/sys/vm/max_map_count65530#more /proc/sys/kernel/threads-max39793#ulimit -acore file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimitedpending signals (-i) 19896max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimitedopen files (-n) 1024pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0stack size (kbytes, -s) 10240cpu time (seconds, -t) unlimitedmax user processes (-u) unlimitedvirtual memory (kbytes, -v) unlimitedfile locks (-x) unlimited
It's very strange ... :-(
tsuna
Jun 18, 2012, 12:08:37 AM6/18/12
to Sebastien Nahelou, open...@googlegroups.com
On Wed, Jun 6, 2012 at 2:29 AM, Sebastien Nahelou
<sebastie...@gmail.com> wrote:
> I currently do some bench about OpenTSDB and i have a question:
> Is OpenTSDB optimized for multi core ?
Yes OpenTSBD is multi-threaded and can use all available hardware threads.
> I currently do some bench about OpenTSDB and i have a question:
> Is OpenTSDB optimized for multi core ?
Note also that this has nothing to do with "HyperThreading", which is
something different.
What you're seeing is because each socket connection is handled only
by one thread at a time. You have only one connection, and thus you
can only max out one core at a time. This "limitation" comes from the
underlying library (Netty) but is actually useful so that all upstream
evens (events coming from the network, such as socket reads) can avoid
taking several locks, since Netty guarantees that all upstream events
for a particular connection are always handled by a particular IO
thread.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Sebastien Nahelou
Jun 18, 2012, 4:43:39 AM6/18/12
to open...@googlegroups.com
Thank you very much for this precision
Reply all
Reply to author
Forward
0 new messages