Memory Problem with stardog 3

[tj.sch...@gmail.com]

Hi

I just installed Stardog 3 yesterday I have severe issues using it - with version 2.2.4, I do not have this problem.

I am working on text search queries (http://docs.stardog.com/#_search_in_sparql).

Memory related exceptions are thrown. Here are some examples from my stardog.log:

[WARNING io.netty.util.internal.logging.Slf4JLogger.warn - Mar 26, 2015 09:45:33.129] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at io.netty.util.internal.MpscLinkedQueue.offer(MpscLinkedQueue.java:126)
    at io.netty.util.internal.MpscLinkedQueue.add(MpscLinkedQueue.java:227)
    at io.netty.util.concurrent.SingleThreadEventExecutor.fetchFromDelayedQueue(SingleThreadEventExecutor.java:270)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:369)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    at java.lang.Thread.run(Thread.java:745)

[SEVERE com.complexible.common.protocols.server.rpc.ServerHandler.exceptionCaught - Mar 26, 2015 09:45:33.130] exceptionCaughtServerHandler
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.complexible.stardog.plan.eval.operator.util.SingleKeySink.getKey(SingleKeySink.java:34)
    at com.complexible.stardog.plan.eval.operator.util.SolutionKeySinks.keyFor(SolutionKeySinks.java:27)
    at com.complexible.stardog.plan.eval.operator.util.MemoryHashJoinTable.add(MemoryHashJoinTable.java:81)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.buildHashTable(HashJoinOp.java:268)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._init(HashJoinOp.java:240)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:142)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:404)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToSkip(AbstractSkippingIteration.java:139)
    at com.complexible.common.collect.AbstractSkippingIteration.skipTo(AbstractSkippingIteration.java:196)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.skipRight(MergeJoinOp.java:381)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._nextMove(MergeJoinOp.java:271)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:239)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._nextLeft(HashJoinOp.java:224)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:151)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.nextLeft(LoopJoin.java:142)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.computeNext(LoopJoin.java:175)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.computeNext(LoopJoin.java:23)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:230)

[SEVERE com.complexible.common.protocols.server.rpc.ServerHandler.exceptionCaught - Mar 26, 2015 09:55:29.313] exceptionCaughtServerHandler
java.lang.OutOfMemoryError: Java heap space
    at com.complexible.stardog.plan.eval.operator.impl.ArraySolution.<init>(ArraySolution.java:75)
    at com.complexible.stardog.plan.eval.operator.impl.ArraySolution.copy(ArraySolution.java:206)
    at com.complexible.stardog.plan.eval.operator.impl.ArraySolution.copy(ArraySolution.java:22)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.buildHashTable(HashJoinOp.java:264)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._init(HashJoinOp.java:240)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:142)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:404)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToSkip(AbstractSkippingIteration.java:139)
    at com.complexible.common.collect.AbstractSkippingIteration.skipTo(AbstractSkippingIteration.java:196)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.skipRight(MergeJoinOp.java:381)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._nextMove(MergeJoinOp.java:271)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:239)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._nextLeft(HashJoinOp.java:224)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:151)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.nextLeft(LoopJoin.java:142)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.computeNext(LoopJoin.java:175)
    at com.complexible.stardog.plan.eval.operator.impl.LoopJoin.computeNext(LoopJoin.java:23)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:230)

 

[SEVERE com.complexible.common.protocols.server.rpc.ServerHandler.exceptionCaught - Mar 26, 2015 10:10:43.521] exceptionCaughtServerHandler
java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:658)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
    at sun.nio.ch.IOUtil.read(IOUtil.java:195)
    at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:700)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:685)
    at com.complexible.common.io.disk.ChannelFile.read(ChannelFile.java:67)
    at com.complexible.common.io.impl.SinglePage.read(SinglePage.java:52)
    at com.complexible.stardog.index.disk.btree.impl.ArrayTreePage.<init>(ArrayTreePage.java:37)
    at com.complexible.stardog.index.disk.btree.impl.ArrayTreePage.<init>(ArrayTreePage.java:27)
    at com.complexible.stardog.index.disk.btree.impl.ArrayTreePage$Factory.create(ArrayTreePage.java:113)
    at com.complexible.stardog.index.disk.btree.impl.TreePageManagerImpl.getPage(TreePageManagerImpl.java:70)
    at com.complexible.stardog.index.disk.btree.impl.TreePageManagerImpl.getPage(TreePageManagerImpl.java:113)
    at com.complexible.stardog.index.disk.btree.impl.BPlusTreeImpl.getPage(BPlusTreeImpl.java:208)
    at com.complexible.stardog.index.disk.btree.impl.TreePageIterator.moveToPage(TreePageIterator.java:215)
    at com.complexible.stardog.index.disk.btree.impl.TreePageIterator.skipTo(TreePageIterator.java:182)
    at com.complexible.stardog.plan.eval.operator.impl.IndexScan$InternalScan.skipTo(IndexScan.java:515)
    at com.complexible.stardog.plan.eval.operator.impl.IndexScan.computeSkipTo(IndexScan.java:399)
    at com.complexible.stardog.plan.eval.operator.impl.IndexScan.computeSkipTo(IndexScan.java:41)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToSkip(AbstractSkippingIteration.java:139)
    at com.complexible.common.collect.AbstractSkippingIteration.skipTo(AbstractSkippingIteration.java:196)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.skipLeft(MergeJoinOp.java:354)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._nextMove(MergeJoinOp.java:276)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:239)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.buildHashTable(HashJoinOp.java:263)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._init(HashJoinOp.java:240)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:142)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:230)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.SortOp.init(SortOp.java:96)
    at com.complexible.stardog.plan.eval.operator.impl.SortOp.computeNext(SortOp.java:75)
    at com.complexible.stardog.plan.eval.operator.impl.SortOp.computeNext(SortOp.java:27)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._init(MergeJoinOp.java:320)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:404)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeSkipTo(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToSkip(AbstractSkippingIteration.java:139)
    at com.complexible.common.collect.AbstractSkippingIteration.skipTo(AbstractSkippingIteration.java:196)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.skipRight(MergeJoinOp.java:381)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp._nextMove(MergeJoinOp.java:271)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:239)
    at com.complexible.stardog.plan.eval.operator.impl.MergeJoinOp.computeNext(MergeJoinOp.java:34)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp._nextLeft(HashJoinOp.java:224)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:151)
    at com.complexible.stardog.plan.eval.operator.impl.HashJoinOp.computeNext(HashJoinOp.java:30)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.SingleProjectionOp.computeNext(SingleProjectionOp.java:107)
    at com.complexible.stardog.plan.eval.operator.impl.SingleProjectionOp.computeNext(SingleProjectionOp.java:36)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.GroupifySortedOp.computeNext(GroupifySortedOp.java:98)
    at com.complexible.stardog.plan.eval.operator.impl.GroupifySortedOp.computeNext(GroupifySortedOp.java:32)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.DistinctOp.computeNext(DistinctOp.java:53)
    at com.complexible.stardog.plan.eval.operator.impl.DistinctOp.computeNext(DistinctOp.java:21)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.impl.SliceOp._hasNext(SliceOp.java:87)
    at com.complexible.stardog.plan.eval.operator.impl.SliceOp.computeNext(SliceOp.java:95)
    at com.complexible.stardog.plan.eval.operator.impl.SliceOp.computeNext(SliceOp.java:26)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:92)
    at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:27)
    at com.complexible.common.collect.AbstractSkippingIteration.tryToComputeNext(AbstractSkippingIteration.java:125)
    at com.complexible.common.collect.AbstractSkippingIteration.hasNext(AbstractSkippingIteration.java:110)
    at com.complexible.common.iterations.TransformException.hasNext(TransformException.java:87)
    at com.complexible.stardog.plan.eval.operator.util.OpBasedBindingSetIteration.computeNext(OpBasedBindingSetIteration.java:110)
    at com.complexible.stardog.plan.eval.operator.util.OpBasedBindingSetIteration.computeNext(OpBasedBindingSetIteration.java:33)
    at com.complexible.common.iterations.AbstractIteration.tryToComputeNext(AbstractIteration.java:106)
    at com.complexible.common.iterations.AbstractIteration.hasNext(AbstractIteration.java:93)
    at com.complexible.common.rdf.query.IterationAsTupleQueryResult.hasNext(IterationAsTupleQueryResult.java:74)
    at com.complexible.stardog.protocols.server.ConnectionFinishQueryResult.hasNext(ConnectionFinishQueryResult.java:75)
    at org.openrdf.query.QueryResults.report(QueryResults.java:155)
    at org.openrdf.query.resultio.QueryResultIO.write(QueryResultIO.java:634)
    at com.complexible.stardog.protocols.http.server.io.ResponseBodyEncoders$12.encode(ResponseBodyEncoders.java:435)
    at com.complexible.stardog.protocols.http.server.io.ResponseBodyEncoders$2.encode(ResponseBodyEncoders.java:99)
    at com.complexible.stardog.protocols.http.annex.QueryPanelEncoder.encode(QueryPanelEncoder.java:112)
    at com.complexible.stardog.protocols.http.server.HttpMessageEncoder.write(HttpMessageEncoder.java:157)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
    at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    at java.lang.Thread.run(Thread.java:745)

regards,

Tobias Schweizer

Digital Humanities Lab
University of Basel
Switzerland

[Kendall Clark]

Hi,

Can you share data and what text searches yr doing? How much RAM are you giving JVM & Stardog on startup?

Cheers,
Kendall

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[tj.sch...@gmail.com]

I use the default settings I guess (I did not change any JVM settings for stardog).

There is nothing special about my queries, I am doing as described on your page:  http://docs.stardog.com/#_search_in_sparql

I am using Ubuntu 14.04 and

java version "1.7.0_75"
OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~trusty1)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)

when creating a db in stardog, I use the following options:

index.named.graphs = true
index.literals.canonical = true

reasoning.schema.graphs = tag:stardog:api:context:all
reasoning.punning.enabled = false

icv.enabled = false

search.enabled = true

transactions.durable = true

However, I do not have this problem with Stardog 2.2.4.

regards,

Tobias Schweizer

[Michael Grove]

Tobias,

Can you please send the query & data so we can try to reproduce the memory usage issue locally?

Thanks.

Mike