JVM configuration: OOM error

[Saurabh Raje]

Hi,

Can anyone suggest some default configuration options / chronicle sizing options for JVM to avoid the following OOM error (during mmap).

Exception in thread "Thread-X" java.io.IOException: Map failed
    at net.openhft.chronicle.VanillaChronicle$AbstractVanillaExcerpt.index(VanillaChronicle.java:412)
    at net.openhft.chronicle.VanillaChronicle$AbstractVanillaExcerpt.nextIndex(VanillaChronicle.java:426)
    at net.openhft.chronicle.tools.WrappedExcerpt.nextIndex(WrappedExcerpt.java:63)
    ... 2 more
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:892)
    at net.openhft.lang.io.VanillaMappedFile.map(VanillaMappedFile.java:86)
    at net.openhft.lang.io.VanillaMappedFile.readWriteBytes(VanillaMappedFile.java:145)
    at net.openhft.lang.io.VanillaMappedCache.put(VanillaMappedCache.java:84)
    at net.openhft.chronicle.VanillaDataCache.dataFor(VanillaDataCache.java:70)
    at net.openhft.chronicle.VanillaChronicle$AbstractVanillaExcerpt.index(VanillaChronicle.java:388)
    ... 4 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:889)
    ... 9 more

 

[Peter Lawrey]

Hello Saurabh,

This is an OS error as you are exceeding the resources allowed by your machine..  It is more common on Windows as you will be limited by the size of the hardware you have.  On Linux you should be able to create maps larger than main memory.

What Operating System, size of memory do you have and what configuration for Chronicle Map are you using?

Regards,

   Peter,

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Saurabh Raje]

Hi Peter,

I'm using default configuration for chronicle queue on Linux.

Investigated the issue a bit further. Essentially what is happening is that there are multiple clients who are:

• creating a chronicle queue remote appender
• creating an excerpt
• writing the excerpt to queue

Due to this multiple 64MB files are being generated on the chronicle queue source server. These files (mostly) contain a single ~1KB message & rest zero bytes. I somehow think all these files have been mmap'ed resulting in OOM.

Regards,

Saurabh

[Peter Lawrey]

Hello Saurabh,

  You shouldn't be getting more than one set of files per remote client.  How many clients are being created and how many files are you getting?

Regards,

   Peter.

[Saurabh Raje]

Hi Peter,

Sorry for the late reply.

I'm getting one file every time I create a new appender. Unfortunately the earlier client code used to open an appender, write a record (~.5KB) and close the appender. Due to this multiple 64MB files were being created. Also the developer's laptop used to go OOM when Chronicle Queue attempted to mmap the 6th data block. Currently the way we have fixed this is by making the cacheCapacity (==dataCacheCapacity & ==indexCacheCapacity) configurable.

Regards,

Saurabh

[Peter Lawrey]

Hello Saurabh,
This happens when you create a new entry in a new thread each time. This is very inefficient. We suggest you use a small thread pool and reuse your threads or use indexed chronicle and hold a lock when writing.

We are planning a queue which won't have this issue however you will still want to control how your threads are created.

Regards,
Peter.