Out of direct buffer memory (again)

[prange]

I upgraded to the latest version available (0.3.0) of BabuDB, and got
this exception again:



INFO | jvm 1 | 2009/12/09 11:50:37 | [Configuration Updater]
DEBUG org.apache.felix.configadmin - Scheduling task Fire
ConfigurationEvent: pid=com.texi.tls2.datawarehouse
INFO | jvm 1 | 2009/12/09 11:50:37 | [Configuration Updater]
DEBUG org.apache.felix.configadmin - Running task Update:
pid=org.ops4j.pax.logging
INFO | jvm 1 | 2009/12/09 11:50:39 | 8192: poolSize
= 1 numRequests = 13685 creates = 2 deletes
= 0
INFO | jvm 1 | 2009/12/09 11:50:39 | 65536: poolSize
= 0 numRequests = 0 creates = 0 deletes
= 0
INFO | jvm 1 | 2009/12/09 11:50:39 | 524288: poolSize
= 0 numRequests = 0 creates = 0 deletes
= 0
INFO | jvm 1 | 2009/12/09 11:50:39 | 2097152: poolSize
= 0 numRequests = 0 creates = 0 deletes
= 0
INFO | jvm 1 | 2009/12/09 11:50:39 | unpooled (> 2097152)
numRequests = creates = 1 deletes = 0
INFO | jvm 1 | 2009/12/09 11:50:39 | Exception in thread
"hz.executor-_hzInstance_0-thread-4" java.lang.OutOfMemoryError:
Direct buffer memory
INFO | jvm 1 | 2009/12/09 11:50:39 | at
java.nio.Bits.reserveMemory(Bits.java:633)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.include.common.buffer.BufferPool.getNewBuffer
(BufferPool.java:163)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.include.common.buffer.BufferPool.allocate(BufferPool.java:
90)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.log.DiskLogFile.next(DiskLogFile.java:75)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.log.DiskLogIterator.findNextEntry
(DiskLogIterator.java:150)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.log.DiskLogIterator.next(DiskLogIterator.java:106)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.BabuDB.replayLogs(BabuDB.java:450)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.BabuDB.<init>(BabuDB.java:175)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
org.xtreemfs.babudb.BabuDBFactory.createBabuDB(BabuDBFactory.java:31)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
com.texi.tls2.persistence.store.babudb.BabuDbStore.createDatabase
(BabuDbStore.java:183)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
com.texi.tls2.persistence.store.babudb.BabuDbStore.getDatabase
(BabuDbStore.java:164)
INFO | jvm 1 | 2009/12/09 11:50:39 | at
com.texi.tls2.persistence.store.babudb.BabuDbStore.get
(BabuDbStore.java:111)


I (still) use Ubuntu 9.04 and java 1.6.0_16

It happens when i load existing databases...

[Jan Stender]

Hi Atle,

Do you have an idea what the approximate size of the largest chunks of
data is that you use as keys or values? Do you bundle up multiple large
key-value pairs in a single insert group?

According to the stack trace, the system runs out of memory when trying
to replay the database log on startup. The only possible reason for this
I can think of is that the log contains a huge log entry that does not
fit in memory.

The buffer stats indicate that at least one large buffer (> 2M) was
allocated before:

INFO | jvm 1 | 2009/12/09 11:50:39 | unpooled (> 2097152)
numRequests = creates = 1 deletes = 0

Have you tried to increase the memory made available to you application
(e.g. via 'java -Xmx2G ...')? Another possible workaround might be to
reduce the size of your records/insert groups at application level.

Unless the error turns out to come from a bug in BabuDB, I don't see a
general solution at BabuDB level.

Best regards,
Jan

[Atle Prange]

Hi Jan.

Increasing the heap size will not work since it is the direct buffer
memory that is exhausted. But if one entry is over 2M there is a bug in
my application. Thanks for the tip, ill report back any progress i have.

-atle