heap space allocation error. 1.0 w/ new sdfs.jar

[Paul]

What are the memory boundries for different size allocations? If I
create a "8tb" volume and start to copy a 2.5gig file, it gets about
130megs in before the copy process locks. opendedup produces the
errors below. If I make the vol 9TB, it gets 5.5megs before the same
hting happens.

Should open dedup really care how much space I tell it there is if the
actual physical back end is always still the same size? In this case,
the real volume size is 100gb.

I'm playing around to see what different settings give me.

Running on an ubuntu 10.10 desktop system with 8gigs of ram.

root@ubisoft:~# cp /home/paul/Desktop/GRMCNENXVOL_EN_DVD.iso /media/
sdfs1/Exception in thread "Thread-24" java.lang.OutOfMemoryError: Java
heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
org.opendedup.collections.ByteArrayLongMap.setUp(ByteArrayLongMap.java:
111)
at
org.opendedup.collections.ByteArrayLongMap.<init>(ByteArrayLongMap.java:
40)
at
org.opendedup.collections.CSByteArrayLongMap.getMap(CSByteArrayLongMap.java:
127)
at
org.opendedup.collections.CSByteArrayLongMap.containsKey(CSByteArrayLongMap.java:
335)
at org.opendedup.sdfs.filestore.HashStore.hashExists(HashStore.java:
152)
at
org.opendedup.sdfs.servers.HashChunkService.hashExists(HashChunkService.java:
70)
at
org.opendedup.sdfs.servers.HCServiceProxy.writeChunk(HCServiceProxy.java:
96)
at
org.opendedup.sdfs.io.SparseDedupFile.writeCache(SparseDedupFile.java:
217)
at
org.opendedup.sdfs.io.WritableCacheBuffer.close(WritableCacheBuffer.java:
301)
at org.opendedup.util.PoolThread.run(PoolThread.java:28)
Exception in thread "Thread-20" java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
org.opendedup.collections.ByteArrayLongMap.setUp(ByteArrayLongMap.java:
111)
at
org.opendedup.collections.ByteArrayLongMap.<init>(ByteArrayLongMap.java:
40)
at
org.opendedup.collections.CSByteArrayLongMap.getMap(CSByteArrayLongMap.java:
127)
at
org.opendedup.collections.CSByteArrayLongMap.containsKey(CSByteArrayLongMap.java:
335)
at org.opendedup.sdfs.filestore.HashStore.hashExists(HashStore.java:
152)
at
org.opendedup.sdfs.servers.HashChunkService.hashExists(HashChunkService.java:
70)
at
org.opendedup.sdfs.servers.HCServiceProxy.writeChunk(HCServiceProxy.java:
96)
at
org.opendedup.sdfs.io.SparseDedupFile.writeCache(SparseDedupFile.java:
217)
at
org.opendedup.sdfs.io.WritableCacheBuffer.close(WritableCacheBuffer.java:
301)
at org.opendedup.util.PoolThread.run(PoolThread.java:28)
Exception in thread "Thread-15" java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
org.opendedup.collections.ByteArrayLongMap.setUp(ByteArrayLongMap.java:
111)
at
org.opendedup.collections.ByteArrayLongMap.<init>(ByteArrayLongMap.java:
40)
at
org.opendedup.collections.CSByteArrayLongMap.getMap(CSByteArrayLongMap.java:
127)
at
org.opendedup.collections.CSByteArrayLongMap.containsKey(CSByteArrayLongMap.java:
335)
at org.opendedup.sdfs.filestore.HashStore.hashExists(HashStore.java:
152)
at
org.opendedup.sdfs.servers.HashChunkService.hashExists(HashChunkService.java:
70)
at
org.opendedup.sdfs.servers.HCServiceProxy.writeChunk(HCServiceProxy.java:
96)
at
org.opendedup.sdfs.io.SparseDedupFile.writeCache(SparseDedupFile.java:
217)
at
org.opendedup.sdfs.io.WritableCacheBuffer.close(WritableCacheBuffer.java:
301)
at org.opendedup.util.PoolThread.run(PoolThread.java:28)
Exception in thread "Thread-23" java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
org.opendedup.collections.ByteArrayLongMap.setUp(ByteArrayLongMap.java:
111)
at
org.opendedup.collections.ByteArrayLongMap.<init>(ByteArrayLongMap.java:
40)
at
org.opendedup.collections.CSByteArrayLongMap.getMap(CSByteArrayLongMap.java:
127)
at
org.opendedup.collections.CSByteArrayLongMap.containsKey(CSByteArrayLongMap.java:
335)
at org.opendedup.sdfs.filestore.HashStore.hashExists(HashStore.java:
152)
at
org.opendedup.sdfs.servers.HashChunkService.hashExists(HashChunkService.java:
70)
at
org.opendedup.sdfs.servers.HCServiceProxy.writeChunk(HCServiceProxy.java:
96)
at
org.opendedup.sdfs.io.SparseDedupFile.writeCache(SparseDedupFile.java:
217)
at
org.opendedup.sdfs.io.WritableCacheBuffer.close(WritableCacheBuffer.java:
301)
at org.opendedup.util.PoolThread.run(PoolThread.java:28)
Exception in thread "Thread-21" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-19" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-22" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-25" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-16" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-17" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-26" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-18" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap
space
cp: writing `/media/sdfs1/GRMCNENXVOL_EN_DVD.iso': Bad address
root@ubisoft:~# Exception in thread "Thread-11"
java.lang.OutOfMemoryError: Java heap space




root@ubisoft:~# cat /etc/sdfs/sdfs1-volume-cfg.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<subsystem-config version="1.0.1">
<locations dedup-db-store="/media/VMsdfs/ddb" io-log="/media/VMsdfs/
io.SDFSLogger.getLog()"/>
<io chunk-size="128" claim-hash-schedule="0 0 0/2 * * ?" dedup-
files="true" file-read-cache="5" max-file-inactive="900" max-file-
write-buffers="32" max-open-files="1024" meta-file-cache="1024" multi-
read-timeout="1000" safe-close="true" safe-sync="false" system-read-
cache="1000" write-threads="12"/>
<permissions default-file="0644" default-folder="0755" default-
group="0" default-owner="0"/>
<volume capacity="8TB" current-size="0" maximum-percentage-full="-1.0"
path="/media/VMsdfs/files"/>
<local-chunkstore allocation-size="8796093022208" chunk-gc-schedule="0
0 0/4 * * ?" chunk-store="/media/VMsdfs/chunkstore/chunks" chunk-store-
dirty-timeout="1000" chunk-store-read-cache="5" enabled="true"
encrypt="false" encryption-key="Zgp1yU4mBySRNwm2-pTketWJV76wlbWYGH9"
eviction-age="6" hash-db-store="/media/VMsdfs/chunkstore/hdb" pre-
allocate="false" read-ahead-pages="1"/>
</subsystem-config>

[Sam Silverberg]

Paul,

SDFS allocates memory for the unique hash table once data is written to the volume for the first time. You should change the allocation size to something more reasonable. The chunkstore is where unique data is stored and can usually be much lower than the volume sized. allocation-size="8796093022208" allocates about 2GB of ram for the Hash table. Since the hash table only is used for unique blocks you should set this to something more appropriate for your data set. As an example, if you plan on getting 10 to 1 deduplication rates set the allocation-size to 8TB/10 or 879609302220. 

The mount.sdfs script, by default, only allocates 2GB for the java heap. You will want to set this to 3GB to get your sdfs mount, as currently configured to work. 

Change the -Xmx2g -Xms2g to -Xmx3g -Xms3g

 

Hope this helps

[Paul]

"set this to something more appropriate for your data set. As an
example, if
> you plan on getting 10 to 1 deduplication rates set the allocation-size to
> 8TB/10 or 879609302220."

That's just it. I set it to 8TB and it died at 135megs of a copy. So
what I have listed in my original post, if I understand you correctly,
is exactly what you are suggesting I should set.

I have it set to 2TB now and was able to create a 50gig .raw vol for a
kvm.

On Nov 12, 1:19 pm, Sam Silverberg <sam.silverb...@gmail.com> wrote:
> Paul,
>
> SDFS allocates memory for the unique hash table once data is written to the
> volume for the first time. You should change the allocation size to
> something more reasonable. The chunkstore is where unique data is stored and
> can usually be much lower than the volume
> sized. allocation-size="8796093022208" allocates about 2GB of ram for the
> Hash table. Since the hash table only is used for unique blocks you should
> set this to something more appropriate for your data set. As an example, if
> you plan on getting 10 to 1 deduplication rates set the allocation-size to
> 8TB/10 or 879609302220.
>
> The mount.sdfs script, by default, only allocates 2GB for the java heap. You
> will want to set this to 3GB to get your sdfs mount, as currently configured
> to work.
>
> Change the -Xmx2g -Xms2g to -Xmx3g -Xms3g
>
> Hope this helps
>

[Paul]

I guess what I don't understand is, if the amount of data to be
deduplicated, in this case a 2.5gb iso, is the same, why does it seem
to take more memory if a volume is set to a 'virtual' size of 9tb,
rather than something more reasonable. The actual data isn't any
different either way, no?

[Sam Silverberg]

The memory required is not related to the volume size but rather the unique data that will be copied to the volume. The unique data lookup table, that is filled with the unique data finger prints, is allocated into memory when the SDFS volume is mounted. The table's size, and thus the memory required, is related to the allocation-size configuration attribute and not the volume size.

As an example, if you set the allocation size to 1TB or allocation-size="107374182400" then the unique data lookup table memory will be around 270 megabytes based on the following calculation:

memory size = ([allocation-size]/[chunk-size in bytes])*32

or

memory size = (107374182400/131072)*32

32 is the size of the unique data record stored in memory.

121072 is 128k in bytes

[Paul]

Definitely. Thanks!

On Nov 13, 11:38 am, Sam Silverberg <sam.silverb...@gmail.com> wrote:
> The memory required is not related to the volume size but rather the unique
> data that will be copied to the volume. The unique data lookup table, that
> is filled with the unique data finger prints, is allocated into memory when
> the SDFS volume is mounted. The table's size, and thus the memory required,
> is related to the allocation-size configuration attribute and not the volume
> size.
>
> As an example, if you set the allocation size to 1TB or
> allocation-size="107374182400" then the unique data lookup table memory will
> be around 270 megabytes based on the following calculation:
>
> memory size = ([allocation-size]/[chunk-size in bytes])*32
>
> or
>
> memory size = (107374182400/131072)*32
>
> 32 is the size of the unique data record stored in memory.
> 121072 is 128k in bytes
>

> ...
>
> read more »