commit and concurrency

[Swapandeep Singh]

Hi,

I am currently playing around with Map DB in my current project and trying to tweak its performance. It started without reading much of documentation as it seemed very intuitive. However it isnt working as I had hoped for and I think it would help if I had a better understading of how things are working. 

I am using MapDB as a cache so that I can store the incoming data outside of heap. Occasionally the cached data will be pushed to a remote server.

Setup

1. MapDB version : 2.0_Snapshot

2. Incoming data : ~5M events spread across around ~100K 

In current setup, a single MapDB is created at the JVM boot time. The events are processed in parallel (~15 threads). I had a couple of iterations on how the model this but here is the rough flow. An instance of MapDB is shared across all threads

1. At construction time, create a new instance of DB

db = DBMaker

        .fileDB(mapDBFile.toFile())
        .fileMmapEnable()
        .fileMmapEnableIfSupported()
        .fileMmapPreclearDisable()
        .fileMmapCleanerHackEnable()
        .make()

2. At runtime add data to this db

      map = db.treeMap(event identifier)

      map.write(data)

      db.commit()

However, this leads to thread locks and leads me to believe I am doing something wrong. My questions are

1. What is the recommended  setup to use MapDB when used as a throwaway cache so that data is off the heap

2. should I do db.commit() after every write? Because if I dont, I see GC kicking in and taking a lot of space / time in Object.finalize (which I assume is because without commit, data is not being flushed and stored in heap)

3. What is the downside of using a single DB and writing the data into a single tree map vs writing data into a single db but 100K tree maps.

4. Do i need to do sharding in top to MapDB (i.e. write my data into a group of DBs, rather than a single DB on the box)

Any help would be greatly appreciated.

Thanks

Swapandeep Singh

Relevant thread dumps infornation

"processingThreadPool-544b2fe0-e416-470c-a723-c5ca09daed5f" #40 daemon prio=5 os_prio=0 tid=0x00007f7be8020800 nid=0x18dba waiting for monitor entry [0x00007f7c14d8a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.mapdb.DB.treeMap(DB.java:1445)
        - waiting to lock <0x00007f7dde6c6368> (a org.mapdb.DB)

"processingThreadPool-3f9cf667-ed55-4302-a4ee-09aaacf2c935" #41 daemon prio=5 os_prio=0 tid=0x00007f7be8022800 nid=0x18dbb waiting on condition [0x00007f7c14c89000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f7dde8b5098> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at org.mapdb.Store.get(Store.java:242)
        at org.mapdb.DB$4.run(DB.java:206)
        at org.mapdb.DB$4.run(DB.java:200)
        at org.mapdb.SerializerPojo.serializeUnknownObject(SerializerPojo.java:472)
        at org.mapdb.SerializerBase.serialize(SerializerBase.java:1000)
        at org.mapdb.SerializerBase.serialize(SerializerBase.java:946)
        at org.mapdb.Serializer.valueArraySerialize(Serializer.java:2235)
        at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:824)
        at org.mapdb.BTreeMap$NodeSerializer.serialize(BTreeMap.java:760)
        at org.mapdb.Store.serialize(Store.java:304)
        at org.mapdb.StoreCached.flushWriteCacheSegment(StoreCached.java:453)
        at org.mapdb.StoreCached.flushWriteCache(StoreCached.java:429)
        at org.mapdb.StoreWAL.commit(StoreWAL.java:613)
        at org.mapdb.DB.commit(DB.java:2638)
        - locked <0x00007f7dde6c6368> (a org.mapdb.DB)
        at com.aws.billing.usagefacts.cache.ShardedMapDBCache.put(ShardedMapDBCache.java:107)

[Jan Kotek]

Hi,

MapDB 2.X has bugs which cause deadlock. Please use lattest 3.x

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.