IllegalStateException while adding values into Chronicle Map

[Paolo Di Tommaso]

Hi, 

I'm giving a try to ChronicleMap. I'm using it to store something about 3.5 million entries. 

The map is declared like this: 

Map<Long,byte[]> data = ChronicleMapBuilder.of(Long.class, byte[].class).file(file).create();

While inserting the items in the map I'm getting the following exception after around 1.7 million entries have been added:

Caught: java.lang.IllegalStateException: VanillaShortShortMultiMap is full

java.lang.IllegalStateException: VanillaShortShortMultiMap is full

at net.openhft.collections.VanillaShortShortMultiMap.nextPos(VanillaShortShortMultiMap.java:226)

at net.openhft.collections.AbstractVanillaSharedHashMap$Segment.put(VanillaSharedHashMap.java:834)

at net.openhft.collections.AbstractVanillaSharedHashMap.put0(VanillaSharedHashMap.java:348)

at net.openhft.collections.AbstractVanillaSharedHashMap.put(VanillaSharedHashMap.java:330)

What's wrong with it ?

Best,

Paolo

[Peter Lawrey]

What is wrong is that the map doesn't resize automatically.  It is assumed you will make the virtual size of the map larger than you need and it will handle this reasonably efficiently. With the default settings you will run out of space between 1 and 2 million entries.

I suggest you set the .entries(4000000) and .entrySize(??) to the size of the key+value you will use.;

BTW I would consider avoiding using byte[] if you can as it is likely to be more efficient if you use an actual data type. (Assuming your true type is not an array of bytes)

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paolo Di Tommaso]

I see. I was confused while reading in the README: ".. is that it does not have to be resized, unlike the ConcurrentHashMap". I realised now that there you are referring the heap, but it still requires to allocate a memory mapped file for all the entries. 

Setting .entries(4000000)  it worked like a charm. In my simple test loading and sorting 3.5 million items it get a speed-up of about 15-20% when compare to LevelDB (Java implementation). That's very good!

The main problem is that in my use case the number of entries and their size can vary widely. So I can't know these numbers in advance. I thinking to solve this by using multiple multiple chronicle map instances. For example starting with one that allocates 100MB, when it is full it continues adding entries into a new created map, and so on.

About avoiding using byte[] I'm sure that would bring further benefits, but the problem here is that I may need to store objects that do not implement the Serializable nor Externalizable interfaces e.g. java.nio.file.Path and this can't allow neither to implement a Chronicle custom marshaller. Thus, the solution I've found is to serialise that objects using Kryo and store them as byte arrays.

Two more quick things: 

1) is there any plan for chronicle map project to target Java 7? (I saw that SharedHashMap does, but I would avoid to use a deprecated project).    

2) between the chronicle map compile/runtime dependencies there's org.testng:testng:6.8.8. Is this library really required at runtime or it is supposed for testing purpose?   

Thanks,

Paolo

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/6UurY1qscS8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Rob Austin]

Paolo

Thanks for your feedback and questions, I’ll look at improving the git readme, we don’t want to be confusing or misleading. 

>1) is there any plan for chronicle map project to target Java 7? (I saw that SharedHashMap does, but I would avoid to use a deprecated project).    

Currently chronicle map works under java 7, the original idea was that it would be java 8 only, however as of yet we have not added any code that only works in java 8, accept for some code in the test cases, so it currently should work under java 7

>between the chronicle map compile/runtime dependencies there's org.testng:testng:6.8.8. Is this library really required at runtime or it is supposed for testing purpose?   

It should only be for testing,  ( so it looks like the pom.xml may have an error by not including <scope>test</scope> ), I’ll take a look

Rob

[Paolo Di Tommaso]

Hi Rob, 

On Wed, Oct 1, 2014 at 3:24 PM, Rob Austin <robaus...@gmail.com> wrote:

Paolo

Thanks for your feedback and questions, I’ll look at improving the git readme, we don’t want to be confusing or misleading. 

I'm sure about that. I appreciate the high quality of your project. 

 

>1) is there any plan for chronicle map project to target Java 7? (I saw that SharedHashMap does, but I would avoid to use a deprecated project).    

Currently chronicle map works under java 7, the original idea was that it would be java 8 only, however as of yet we have not added any code that only works in java 8, accept for some code in the test cases, so it currently should work under java 7

Unfortunately when launching with Java 7 I get this exception

Caught: java.lang.UnsupportedClassVersionError: net/openhft/chronicle/map/ChronicleMap : Unsupported major.minor version 52.0

java.lang.UnsupportedClassVersionError: net/openhft/chronicle/map/ChronicleMap : Unsupported major.minor version 52.0

I think the problem is simply that your are compiling with Java 8 without specifying the target jvm bytecode. 

 

>between the chronicle map compile/runtime dependencies there's org.testng:testng:6.8.8. Is this library really required at runtime or it is supposed for testing purpose?   

It should only be for testing,  ( so it looks like the pom.xml may have an error by not including <scope>test</scope> ), I’ll take a look

OK, good. 

Cheers,
Paolo

 

[Rob Austin]

Paolo

Unfortunately when launching with Java 7 I get this exception

Caught: java.lang.UnsupportedClassVersionError: net/openhft/chronicle/map/ChronicleMap : Unsupported major.minor version 52.0

java.lang.UnsupportedClassVersionError: net/openhft/chronicle/map/ChronicleMap : Unsupported major.minor version 52.0

I think the problem is simply that your are compiling with Java 8 without specifying the target jvm bytecode. 

Gosh - you maybe right that the target release was incorrect. Its possible that mixing of the source and test version is not advisable - see the maven pom.xml below, in this xml you’ll see its sets the tests to 1.8 and source to 1.6 . Given that we only had one java8 test I can comment it out, this way we can defiantly target java 1.6 in the next release.

Rob

  <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-compiler-plugin</artifactId>

                <configuration>

                    <compilerArgument>-Xlint:deprecation</compilerArgument>

                    <source>1.6</source>

                    <target>1.6</target>

                    <testSource>1.8</testSource>

                    <testTarget>1.8</testTarget>

                    <encoding>UTF-8</encoding>

                </configuration>

            </plugin>

[Paolo Di Tommaso]

ah, Maven pain.. ;)

Thanks,

Paolo

--

[Peter Lawrey]

The difference is that it pre - allocates virtual space, however only the pages you use get turned into real memory or disk space. E.g if you add one entry to a 1 TB store and do "du map.file" you will see only a few MB are used to start with.

[Peter Lawrey]

If you can't serialized the data, a byte [] is fine.  You should pick the maximum sizes you need and the Map should do it's best to optimise your usage.

Chronicle Map works on Java 7 but the documentation needed to be updated.

There shouldn't be any testing libraries needed.

[Paolo Di Tommaso]

Yes, but in any case you need to have the allocated space free on your disk, otherwise you will get "no space left on device" error if you use a entries number too big. 

This is way I'm think to use multiple chronicle map instances. 

Cheers,

Paolo

[Rob Austin]

I think on Linux ( when using Chronicle ), the file size is only the size of your used space, not your allocated space.

[Paolo Di Tommaso]

Well, I've double checked this on Mac OSX, and just creating a Chronicle map and putting an item into it, I get a 500MB data file. Setting .entries to 4000000 and adding an entry the file was 2GB. 

This means OSX and Linux manage the memory mapped file allocation a different way (?)

p

[Rob Austin]

>This means OSX and Linux manage the memory mapped file allocation a different way.

Yes, It looks that way.

[Paolo Di Tommaso]

OK, I didn't know this. 

Thanks,

Paolo

[Rob Austin]

Peter - can you confirm this as you use Ubuntu for development. 

You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Peter Lawrey]

If you use

ls -l file

It shows you the extents. If you use

du file

It shows you how much is actually used. Can you confirm you used du?

You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Paolo Di Tommaso]

Yes, I can confirm this on the Mac as well. 

du file: shows the real use space 

ls file: shows the extend (allocated) file size

  

But this do change the problem that I cannot pre-allocate a map as big as possible. When I try to create a map, setting .entries to Long.MAX_VALUE, I get immediately a "no left space on device" exception. 

I do not think this a chronicle issue, but it is related to how the OS check the space available on the file system. 

 

p

[Peter Lawrey]

On Ubuntu I can create a 100 TB map on my laptop.  Both top and ls -l say the process virtual size / file size is 100 TB, however the resident memory and du say the size is 71 MB after adding 10000 entries.

At present, we are assuming clients who are heavy user's of these libraries are using Linux (Centos usually) and it behaves the same.

On Windows it appears to be worse than Mac OSX as it reserves the same amount of memory eagerly, not just disk space.

--

[Paolo Di Tommaso]

I see. Thanks to clarify this. 

Best,

Paolo

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/6UurY1qscS8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Paolo Di Tommaso]

FYI, upgrading to Chronicle map 2.0.1a I got a speed up of 1.6 times! My sort benchmark pass from 84 second to 52.5. 

Impressive!

Best,
Paolo

[Rob Austin]

Thats great news, are you using TCP Replication or just shared memory across processes ?

[Peter Lawrey]

Is that compared with an older version of Chronicle?

[Paolo Di Tommaso]

Yep, before I was using version 1.0.2

p

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/6UurY1qscS8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Rob Austin]

1.0.2 was released on 21/08/2014 

[Paolo Di Tommaso]

It is single process benchmark. So no replication.

I've noticed where there are no more space available to store entries it throws a generic IllegalStateException

Caught: java.lang.IllegalStateException: Cannot write 74 only -12 remaining

java.lang.IllegalStateException: Cannot write 74 only -12 remaining

at net.openhft.lang.io.AbstractBytes.checkWrite(AbstractBytes.java:759)

at net.openhft.lang.io.AbstractBytes.write(AbstractBytes.java:753)

 

May I suggest to raise a more specific exception so that the calling app can handle this condition ? 

Best, p 

[Rob Austin]

Thats a good idea, so I have created the following JIRA - https://higherfrequencytrading.atlassian.net/browse/HCOLL-166

[Rob Austin]

I assume that you do know that the maximum number of entries is configurable  ( so you can increase the default if you wish ).

but yes you raise a good point in your comment below :

[Paolo Di Tommaso]

Yes, but the problem is that I can't predict the maximum amount of entries the user will store in the map.

So I thinking to roll in a new chronicle map instance when there's no more space available in the current one. 

p

--
You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/6UurY1qscS8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicl...@googlegroups.com.

[Rob Austin]

Sounds a good solution for now, sometime in the future ( maybe next year, we will support maps that can scale the number of entries dynamically, but this is not a very simple problem to solve ), in the meantime if you come up with a good work around ( perhaps in a utility class we would be very happy to look at a pull request )

Another option you have ( if you are using linux )  is to make the maximum number of entries stupidly large, as this won’t take up any disk space or memory until the entry is written to the map, this way, it’s unlikely you will face this problem, see for more about this :

https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#size-of-space-reserved-on-disk

https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#entries

Rob

[Peter Lawrey]

We have optimised situations where you either don't use;

- all the entries you have allowed for.  This works best on Unix where the disk space and memory used reflected the number of actual entries, not the number you allowed for.

- all the space you allow for each entry.  This helps if you have entries which are multiple cache lines (128 bytes +), only the lines you touch sit in you CPU cache and if you have multiple pages (8+ Kbytes) only the pages you touch use memory or disk.  The CPU cache usage matters as it can be 10000x smaller than main memory.

On Unix, you can create one million, one MB entries and it creates a file which looks like 1 TB (ls -l map).  In top it uses 1 TB of *virtual* memory, however if you add 1000 x 1 KB entries it will uses about 8 KB per entry (two pages) or about 8 MB of disk of main memory.  You can see the size actually used with du.