Skip to first unread message
s.ar...@geophy.com
Sep 27, 2016, 7:51:28 AM9/27/16
to Stardog
Hi all, any way to load large dataset with low memory requirement.
The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory available.
Here is the configuration:
stardog-4.1.3$ ./bin/stardog-admin server status
Backup Storage Directory : .backup
CPU Load : 31.6200 %
Connection Timeout : 1h
Export Storage Directory : .exports
Leading Wildcard Search : false
Memory Heap : 1.6G (Allocated: 7.7G Max: 7.7G)
Memory Non-Heap : 97M (Allocated: 100M Max: 8.0G)
Named Graph Security : false
Platform Arch : amd64
Platform OS : Linux 4.4.0-38-generic, Java 1.8.0_91
Query All Graphs : false
Query Timeout : 5m
Search Default Limit : 100
Stardog Home : /home/samur/tools/stardog-4.1.3/bin/..
Stardog Version : 4.1.3
Strict Parsing : true
Uptime : 1 day 7 hours 5 minutes 39 seconds
Watchdog Enabled : true
Watchdog Port : 5833
Watchdog Remote Access : true
I am getting this error:
------------------------------------------------------------------------------------------------------------------------------------------
INFO 2016-09-22 16:47:54,490 [Stardog.Executor-357] com.complexible.stardog.StardogKernel:printInternal(314): Parsing triples: 8% complete in 00:39:25 (283.0M triples - 119.6K triples/sec)
WARN 2016-09-22 16:48:06,120 [Stardog.Executor-356] com.complexible.common.rdf.rio.RDFStreamProcessor:setException(557): Error during loading /home/samur/tools/d2rq-0.8.1/data.ttl.gz: java.lang.OutOfMemoryError: Direct buffer memory
java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory
at com.complexible.common.rdf.rio.RDFStreamProcessor$Consumer.work(RDFStreamProcessor.java:991) [stardog-utils-rdf-4.1.3.jar:?]
at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:737) [stardog-utils-rdf-4.1.3.jar:?]
at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:726) [stardog-utils-rdf-4.1.3.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[?:1.8.0_91]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_91]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_91]
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) ~[?:1.8.0_91]
at sun.nio.ch.IOUtil.write(IOUtil.java:58) ~[?:1.8.0_91]
at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778) ~[?:1.8.0_91]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764) ~[?:1.8.0_91]
at com.complexible.stardog.dht.impl.PagedDiskHashTable$1.write(PagedDiskHashTable.java:168) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.OffHeapCompactPage.write(OffHeapCompactPage.java:257) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.writePage(PagedDiskHashTable.java:969) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.inactivatePage(PagedDiskHashTable.java:979) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.HashPageCache.evictBatch(HashPageCache.java:197) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.createPage(PagedDiskHashTable.java:1118) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.addOverflowPage(PagedDiskHashTable.java:1196) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.SequentialHashPage.putMaybeFits(SequentialHashPage.java:84) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.SequentialHashPage.put(SequentialHashPage.java:75) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.SequentialHashPage.put(SequentialHashPage.java:56) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.putIntoPage(PagedDiskHashTable.java:940) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable.access$1400(PagedDiskHashTable.java:79) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable$Updater.put(PagedDiskHashTable.java:1325) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.impl.PagedDiskHashTable$Updater.put(PagedDiskHashTable.java:1314) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.dictionary.HashDictionary.getOrAddUncached(HashDictionary.java:561) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.dictionary.HashDictionary.getOrAddID(HashDictionary.java:526) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.dht.dictionary.HashDictionary.add(HashDictionary.java:497) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.index.dictionary.DelegatingMappingDictionary.add(DelegatingMappingDictionary.java:44) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.index.dictionary.InliningMappingDictionary.add(InliningMappingDictionary.java:73) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:70) ~[stardog-4.1.3.jar:?]
at com.complexible.common.rdf.rio.RDFStreamProcessor$Consumer.work(RDFStreamProcessor.java:978) ~[stardog-utils-rdf-4.1.3.jar:?]
... 6 more
Any way to fix it?
Michael Grove
Sep 27, 2016, 9:00:36 AM9/27/16
to stardog
On Tue, Sep 27, 2016 at 7:51 AM, <s.ar...@geophy.com> wrote:
> Hi all, any way to load large dataset with low memory requirement.
>
> The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory
> available.
You can learn more about capacity planning for Stardog in the documentation [1].
> Hi all, any way to load large dataset with low memory requirement.
>
> The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory
> available.
>
> Here is the configuration:
>
> stardog-4.1.3$ ./bin/stardog-admin server status
> Backup Storage Directory : .backup
> CPU Load : 31.6200 %
> Connection Timeout : 1h
> Export Storage Directory : .exports
> Leading Wildcard Search : false
> Memory Heap : 1.6G (Allocated: 7.7G Max: 7.7G)
> Memory Non-Heap : 97M (Allocated: 100M Max: 8.0G)
> Named Graph Security : false
> Platform Arch : amd64
> Platform OS : Linux 4.4.0-38-generic, Java 1.8.0_91
> Query All Graphs : false
> Query Timeout : 5m
> Search Default Limit : 100
> Stardog Home : /home/samur/tools/stardog-4.1.3/bin/..
> Stardog Version : 4.1.3
> Strict Parsing : true
> Uptime : 1 day 7 hours 5 minutes 39 seconds
> Watchdog Enabled : true
> Watchdog Port : 5833
> Watchdog Remote Access : true
direct memory set to 8G there's probably some sort of resource
contention issues you're going to face; there has to be memory
available for your OS to run.
Cheers,
Mike
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com
> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
s.ar...@geophy.com
Sep 27, 2016, 9:05:57 AM9/27/16
to Stardog
On Tuesday, 27 September 2016 15:00:36 UTC+2, Michael Grove wrote:
On Tue, Sep 27, 2016 at 7:51 AM, <s.ar...@geophy.com> wrote:
> Hi all, any way to load large dataset with low memory requirement.
>
> The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory
> available.
You can learn more about capacity planning for Stardog in the documentation [1].
Could you please send the actual link to the documentation?
I have 64 GB memory.
I allocated 8GB for heap / direct memory as you see in the output of the command stardog-admin server status in the beginning of the message.
Michael Grove
Sep 27, 2016, 9:15:09 AM9/27/16
to stardog
On Tue, Sep 27, 2016 at 9:05 AM, <s.ar...@geophy.com> wrote:
>
>
> On Tuesday, 27 September 2016 15:00:36 UTC+2, Michael Grove wrote:
>>
>> On Tue, Sep 27, 2016 at 7:51 AM, <s.ar...@geophy.com> wrote:
>> > Hi all, any way to load large dataset with low memory requirement.
>> >
>> > The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory
>> > available.
>>
>> You can learn more about capacity planning for Stardog in the
>> documentation [1].
>
>
> Could you please send the actual link to the documentation?
Sure [1].
>
>
> On Tuesday, 27 September 2016 15:00:36 UTC+2, Michael Grove wrote:
>>
>> On Tue, Sep 27, 2016 at 7:51 AM, <s.ar...@geophy.com> wrote:
>> > Hi all, any way to load large dataset with low memory requirement.
>> >
>> > The dataset I have is 14GB compress (ttl.gz). I only have 8GB memory
>> > available.
>>
>> You can learn more about capacity planning for Stardog in the
>> documentation [1].
>
>
> Could you please send the actual link to the documentation?
In the future, you can just go to the documentation; the table of
contents is along the right side of the page and lists every topic
covered in the docs. We try to make it as easy as we can to find the
information you're looking for.
averages we post in the capacity planning section. Since you have a
lot more memory available, I recommend just allocating more memory to
Stardog until you can load the dataset. All datasets will have
different overhead, and enabling the secondary indexes like full-text
or geospatial increases memory overhead of writes. We can only provide
a basic starting point and you have to tailor that to what you need
based on your use case.
Cheers,
Mike
[1] http://docs.stardog.com/#_capacity_planning
Evren Sirin
Sep 27, 2016, 9:30:33 AM9/27/16
to Stardog
As you can see in the capacity planning section the direct memory
becomes more important as you get to the billion scale and the
required direct memory increases non-linearly. You probably have 3 to
5 billion triples in your file so I'd recommend at least 16GB direct
memory. You do not need to increase the heap size that much though
smaller heap means more GC activity and slower loading time.
Best,
Evren
becomes more important as you get to the billion scale and the
required direct memory increases non-linearly. You probably have 3 to
5 billion triples in your file so I'd recommend at least 16GB direct
memory. You do not need to increase the heap size that much though
smaller heap means more GC activity and slower loading time.
Best,
Evren
Reply all
Reply to author
Forward
0 new messages