java.lang.IllegalArgumentException: Entry is too large

[Martin Petzold]

Could not find any information about this. What does this Exception actually mean:

Caused by: java.lang.IllegalArgumentException: Entry is too large: requires 161 chucks, 129 is maximum.
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCode(CompiledMapQueryContext.java:1760)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCodeGuarded(CompiledMapQueryContext.java:120)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.alloc(CompiledMapQueryContext.java:3006)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.initEntryAndKey(CompiledMapQueryContext.java:3436)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.putEntry(CompiledMapQueryContext.java:3891)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.doInsert(CompiledMapQueryContext.java:4080)
    at net.openhft.chronicle.map.MapEntryOperations.insert(MapEntryOperations.java:157)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.insert(CompiledMapQueryContext.java:4051)
    at net.openhft.chronicle.map.MapMethods.put(MapMethods.java:89)
    at net.openhft.chronicle.map.VanillaChronicleMap.put(VanillaChronicleMap.java:567)

Thanks! 

[Martin Petzold]

Additionally information: I have serialized objects as values and the size varies. I assumed the value in a ChronicleMap can generally be any size?

I have currently set averageValueSize to 256, no idea how to determine a suitable value for serialized objects of arbitrary size. Maybe I would be able to guess a maximum size, but I would like the Map to work in any case.

Am 19.11.16 um 17:47 schrieb Martin Petzold:

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/wASWElvC3zM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Martin Petzold (CEO and Co-Founder)

TAVLA SCE mbH i.G. c/o GATEWAY - Universität zu Köln
Ägidiusstr. 14
50937 Cologne
Germany

Phone: +49 (0)221 / 470 - 1128
Mobile: +49 (0)179 / 9220154
E-Mail: martin....@tavla.de
Web: www.tavla.de

[Roman Leventov]

Please read the docs:

 - http://static.javadoc.io/net.openhft/chronicle-map/3.10.1/net/openhft/chronicle/map/ChronicleMapBuilder.html#averageValue-V-

 - http://static.javadoc.io/net.openhft/chronicle-map/3.10.1/net/openhft/chronicle/map/ChronicleMapBuilder.html#averageValueSize-double-

 - https://github.com/openhft/chronicle-map#key-and-value-types

TLDR: prefer averageValue() over averageValueSize() (and averageKey() over averageKeySize()).

Also note that if your values and keys are just Serializable, Chronicle Map will end up serializing the class representation (accoring to serialization spec), that is huge, along with *every* key or value that you store. Implementing custom serializers (https://github.com/openhft/chronicle-map#custom-serializers) or even just java.io.Externalizable could be a huge win.

Also, could you please post the whole ChronicleMapBuilder configuration that you use?

On 19 November 2016 at 14:40, Martin Petzold <martin....@tavla.de> wrote:

Additionally information: I have serialized objects as values and the size varies. I assumed the value in a ChronicleMap can generally be any size?

I have currently set averageValueSize to 256, no idea how to determine a suitable value for serialized objects of arbitrary size. Maybe I would be able to guess a maximum size, but I would like the Map to work in any case.

Am 19.11.16 um 17:47 schrieb Martin Petzold:

Could not find any information about this. What does this Exception actually mean:

Caused by: java.lang.IllegalArgumentException: Entry is too large: requires 161 chucks, 129 is maximum.
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCode(CompiledMapQueryContext.java:1760)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCodeGuarded(CompiledMapQueryContext.java:120)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.alloc(CompiledMapQueryContext.java:3006)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.initEntryAndKey(CompiledMapQueryContext.java:3436)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.putEntry(CompiledMapQueryContext.java:3891)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.doInsert(CompiledMapQueryContext.java:4080)
    at net.openhft.chronicle.map.MapEntryOperations.insert(MapEntryOperations.java:157)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext.insert(CompiledMapQueryContext.java:4051)
    at net.openhft.chronicle.map.MapMethods.put(MapMethods.java:89)
    at net.openhft.chronicle.map.VanillaChronicleMap.put(VanillaChronicleMap.java:567)

Thanks! 
--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/wASWElvC3zM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Martin Petzold (CEO and Co-Founder)

TAVLA SCE mbH i.G. c/o GATEWAY - Universität zu Köln
Ägidiusstr. 14
50937 Cologne
Germany

Phone: +49 (0)221 / 470 - 1128
Mobile: +49 (0)179 / 9220154
E-Mail: martin....@tavla.de
Web: www.tavla.de

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Martin Petzold]

Thanks, I will consider a custom serializer or java.io.Externalizable to improve performance later. In my case I don't have a very large amount of data, but need key-value structure, data persistence and very high performance on random data access. However, my values are neither fixed nor estimable in size, so I need to know if there is an upper limit for value size in ChronicleMap, which seems to be the case?

Example: A value is a String, mostly between 1 and 100 characters. But it can happen that it is 500 or 100.000 characters.

My use case is actually, that I have a JSON-like data structure in Java which should be persisted. I have implemented this on top of java.util.List and java.util.Map, which are now both backed by a ChronicleMap instance. Therefore, I have "String" (JSON object) respectively "Integer" (JSON array) as keys and "String" (no upper limit!), "Number", "Boolean" as values. All value types are implemented as serializable Java objects. I will switch to java.io.Externalizable to improve performance later.

ChronicleMap configuration:

ConcurrentHashMap<String,Serializable> map = ChronicleMapBuilder.of(String.class, Serializable.class).averageKeySize(64).averageValueSize(256).entries(25).createPersistedTo(...);

The error occurred when storing a large String, currently I can't tell the exact size. Once I set averageValueSize to 1024 it worked.

Am 20.11.16 um 01:28 schrieb Roman Leventov:

To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roman Leventov]

This makes sense now. You specified that your values are going to be 256 bytes on average, and there will be just 25 of them. So Chronicle Map decided to create segments with about 256 * 25 bytes in the space, available for entry allocation. But then you tried to insert a value which is bigger than the whole allocation space. Chronicle Map is not able to handle this situation now.

Workaround: just configure more builder.entries(), or use low-level configurations for better control: builder.actualChunkSize(), .actualChunksPerSegmentTier(), .actualSegments().

If your values are JSON values, rather than using Externalizable you should much better use make custom serializers and delegate to existing JSON serializer/deserializer such as https://github.com/OpenHFT/Chronicle-Wire, which works relies on the same Chronicle-Bytes library as Chronicle Map. This will be much more efficient.

[Martin Petzold]

Ok, thanks. I will make sure that a value is never bigger than the whole allocation space. However, does ChronicleMap also resize dynamically in case there are more entries than defined initially?

Am 21.11.16 um 04:27 schrieb Roman Leventov:

To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.