MapDB not inserting more than 268.4 Million entries to IndexTreeList

[Tarun Kumar]

Hi, I am trying to insert 1B entries to IndexTreeList. It ingests just 268400000 entries and stops the ingestion. Exception is:

java.lang.AssertionError

        at org.mapdb.IndexTreeListJava.treePut(IndexTreeListJava.java:331)

        at org.mapdb.IndexTreeLongLongMap.put(IndexTreeLongLongMap.kt:77)

        at org.mapdb.IndexTreeList.add(IndexTreeList.kt:25)

My implementation is:

val db = DBMaker.memoryDirectDB().allocateStartSize(50 * 1024L * 1024 * 1024).make()  // allocate 50G

val list = db.indexTreeList("list", Serializer.STRING).create()

(1 to 500000000).foreach { n => {
  list.add("" + n)

}} 

I am using 120G swap + 30G memory machine and "-XX:MaxDirectMemorySize=100000m" JVM argument. This is my memory snapshot:

             total       used       free     shared    buffers     cached

Mem:         30663      30429        234          0          0         35

-/+ buffers/cache:      30393        270

Swap:       122879      27179      95700

Any idea what is going on?

[alla...@utoronto.ca]

I tried a similar thing and did manage to get a billion things into mapdb.  However, I could not do it in memory nor using memory mapped access.  The only thing that worked to handle that much data was file channel.  You just run out of memory and swap eventually.