JVM crash using large database and mmap enabled files

[Steve Ash]

I have a mapdb database that is 68GB. I open the database with:

DBMaker.newFileDB(dbFile)

                .cacheSize(50 * 1024 * 1024)

                .commitFileSyncDisable()

                .mmapFileEnableIfSupported();

When I try to open it and do things with it- I get malloc failures in the JVM and the jvm crashes.  When I look at the he_err_pid log it shows the memory mapped virtual address space for the mapdb .p file.  So somehow this is allocating virtual address space that is encroaching on some of the virtual address space that the JVM is trying to commit?

Here's one of the failures (they aren't allows the exact same every time -- but they do happen every time):

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1adec22000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.

# An error report file with more information is saved as:

The he_err_pid file is 8mb. I have attached a .zip of it if you're interested.  The interesting bit is the VAS mapped to the mmap .p file.  Starts:

7fa89b600000-7fa89b700000 r--s fe8c00000 fc:00 9574574                   /home/steve/idea/argo.match/argo.match.tools/match-syngen/names-db.dat.p

many many thousands of lines later

7fb8aff00000-7fb8b0000000 r--s 00300000 fc:00 9574574                    /home/steve/idea/argo.match/argo.match.tools/match-syngen/names-db.dat.p

So I realize that 68gb is big -- but its virtual address space.  This is 64 bit.  So should have plenty ;)  I guess this is a jvm bug that you can do a mmap with a VAS size so large that it prevents the normal thread stack de/allocation.  But none the less seems like mapdb should work around this if nothing else.  Maybe split up the .p files? Or only map in sections at a time? I don't know.

Thoughts?

[Steve Ash]

And of course now I can't open the file at all :(

Caught: java.lang.AssertionError: data were not fully read, check your serializer 

java.lang.AssertionError: data were not fully read, check your serializer 

at org.mapdb.Store.deserialize(Store.java:299)

at org.mapdb.StoreDirect.get2(StoreDirect.java:475)

at org.mapdb.StoreWAL.get2(StoreWAL.java:368)

at org.mapdb.StoreWAL.get(StoreWAL.java:352)

at org.mapdb.Caches$HashTable.get(Caches.java:245)

at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)

at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1081)

at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1096)

at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1096)

at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1096)

at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1046)

at org.mapdb.HTreeMap$HashIterator.advance(HTreeMap.java:1033)

at org.mapdb.HTreeMap$HashIterator.moveToNext(HTreeMap.java:992)

at org.mapdb.HTreeMap$ValueIterator.next(HTreeMap.java:1127)

at com.argodata.match.syngen.spider.NameDictPrinter.run(NameDictPrinter.groovy:19)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

I didn't change the serializers or anything...

[Jan Kotek]

Hi Steve,

 

I have seen similar issue before, although it was not JVM crash just OOEM exception. It was caused by not enough free space on EXT4 fs. I deleted files, but those were still linked (and not released) by other running process.

 

MapDB maps storage files in 1MB chunks, for 64GB it opens 64K mmap pointers. Prior 1.0.5 there was also leak which did not released pointers correctly, causing some problems on Windows.

 

Mmap files are sort of black magic for me. I encountered handful of JVM bugs which I had to workaround. I think allocating way too many pointers might be triggering another bug, so I am reworking this section for 2.0.

 

I hope data loss is not critical. ".commitFileSyncDisable()" pretty much eliminates WAL durability. But if store was closed correctly before reopening than data were flushed correctly, and there could be a problem in MapDB.

 

There are some assertions and checksums in MapDB to prevent loading incorrect data. In theory we could disable those and attempt to rescue some data from HTreeMap.

 

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Ash]

How are you getting around this long standing limitation:

http://bugs.java.com/view_bug.do?bug_id=6893654

You received this message because you are subscribed to a topic in the Google Groups "MapDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mapdb/PIfjkv8mDZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mapdb+un...@googlegroups.com.

[Jan Kotek]

I was not aware of this. I just assume that UUID is thread safe.

 

Inconsistent hash could in theory explain JVM crash. If MapDB reads wrong file offset, we could end with very high file pointer, and that could crash JVM.

 

There is workaround. Ignore UUID.hashCode and UUID.equals methods. Use custom hasher:

 

Map hashMap = db.createHashMap()

.hasher(new UUIDHasher).keySerializer(Serializer.UUID).makeOrGet()

 

Original link does not work, here is link to google cache:

https://webcache.googleusercontent.com/search?q=cache:DUfO-GKxwlEJ:bugs.java.com/view_bug.do%3Fbug_id%3D6611830+&cd=1&hl=en&ct=clnk

 

I raised new issue to investigate this for 2.0

https://github.com/jankotek/MapDB/issues/387

 

 

Jan

[Jan Kotek]

This seems to be only problem with Java6, it was rewritten in java 7

 

Jan

[Steve Ash]

Ah ok. I was actually referring to the problem of not being able to reliably unmap files.  There was a workaround on that page showing how to call the cleaner method reflectively

[Jan Kotek]

MapDB already uses reflection to unmap files when they are closed.

 

I think that is only problem on Windows (files are locked until unmapped).

MapDB does not shrink (unmap) files unless compaction is involved.

 

Perhaps it could cause problem if you open/close storage frequently.

 

Jan

[Kapil Ghodawat]

I am facing similar issue of jvm crash. Is this problem fixed in 2.0 version (I am still using 1.0.8)
I have multiple mapDB maps each is a BTreeMap and I have multiple DBs though I am opening them all at once which causes jvm to crash (on 64-bit Ubuntu)
I believe there is some sort of limit on maximum number of memory-mapping on OS beyond which linux kills the process (OOM Killer or something)
http://www.linuxforums.org/forum/kernel/131557-what-maximum-number-mmaped-areas.html

But is there any way mapDB handles this if I don't want to turn off memory mapping for my maps?

[Jan Kotek]

I opened an issue for that, will be addressed in 3.1

https://github.com/jankotek/mapdb/issues/723