get millions of vertex query hang
fanc...@gmail.com
I want to dump all vertexes whose type is "personClass", There about 3 million vertexes should return.
I define schema using:
final PropertyKey type=mgmt.makePropertyKey("type").cardinality(Cardinality.SET).dataType(String.class).make();
mgmt.buildIndex("type", Vertex.class).addKey(type).buildCompositeIndex();
My query is:
for (Object v : graph.query().has("type", "personClass").vertices()) {
}
after iterate over 810,000 vertexs, it hangs.
I checked that hbase is ok and gc is normal.
The client has enough memory either
I use jstack and found it hang on:
"main" prio=5 tid=0x00007f9e98812000 nid=0x1903 waiting on condition [0x000000010c4ff000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
- locked <0x00000007716cddd8> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:780)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:789)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:155)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getSlice(HBaseKeyColumnValueStore.java:75)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:65)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache$2.call(ExpirationKCVSCache.java:91)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache$2.call(ExpirationKCVSCache.java:87)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4791)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
- locked <0x00000007716cdec8> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4788)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache.getSlice(ExpirationKCVSCache.java:87)
at com.thinkaurelius.titan.diskstorage.BackendTransaction$5.call(BackendTransaction.java:383)
at com.thinkaurelius.titan.diskstorage.BackendTransaction$5.call(BackendTransaction.java:380)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:428)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:380)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.query(IndexSerializer.java:495)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$8$6$1.call(StandardTitanTx.java:1114)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$8$6$1.call(StandardTitanTx.java:1111)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4791)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
- locked <0x00000007716cdf88> (a com.google.common.cache.LocalCache$StrongAccessEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4788)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$8$6.call(StandardTitanTx.java:1111)
at com.thinkaurelius.titan.graphdb.query.QueryUtil.processIntersectingRetrievals(QueryUtil.java:213)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$8.execute(StandardTitanTx.java:1125)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$8.execute(StandardTitanTx.java:994)
at com.thinkaurelius.titan.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:195)
at com.thinkaurelius.titan.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:61)
at com.thinkaurelius.titan.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:40)
at com.thinkaurelius.titan.graphdb.query.ResultSetIterator.next(ResultSetIterator.java:53)
at com.thinkaurelius.titan.graphdb.query.ResultSetIterator.next(ResultSetIterator.java:14)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:687)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
jmap output
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
using thread-local object allocation.
Parallel GC with 4 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 2684354560 (2560.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 298319872 (284.5MB)
used = 296097664 (282.3807373046875MB)
free = 2222208 (2.1192626953125MB)
99.25509219848418% used
From Space:
capacity = 298319872 (284.5MB)
used = 231227648 (220.515869140625MB)
free = 67092224 (63.984130859375MB)
77.50997157842707% used
To Space:
capacity = 298319872 (284.5MB)
used = 0 (0.0MB)
free = 298319872 (284.5MB)
0.0% used
PS Old Generation
capacity = 1075838976 (1026.0MB)
used = 564392616 (538.2467422485352MB)
free = 511446360 (487.75325775146484MB)
52.46069612558822% used
PS Perm Generation
capacity = 35651584 (34.0MB)
used = 24191120 (23.070449829101562MB)
free = 11460464 (10.929550170898438MB)
67.8542642032399% used
fanc...@gmail.com
Caused by: java.io.IOException: Call to mobvoi-knowledge-graph-0/192.168.10.121:60020 failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1484)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1456)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1656)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1714)
does titan has some problem when query result is very large? In my previous post, when vertex has very large number of edges. iterate is also slow.
Matthias Broecheler
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/a56a4369-6afb-4337-961c-25bb98398793%40googlegroups.com.--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
