How can I modify or update a record if MapDB requires immutable keys / values?

[Derek Leblanc]

Is the solution to use Map<Key,List<Values>> and store my values that will be modified in the List? Is what i'm asking even possible?

I am attempting to use MapDB to maintain a NASDAQ order book as quickly as possible which contains hundreds of millions of orders with a unique integer order_id and a few associated values (2 Strings, 2 Integer, 1 Float)

part of the problem is these orders have to be modified as the program chugs along so I have to get an order by it's ID and modify it. 

MapDB seems quite capable and I really hope it can handle my problem.

Thanks for the help :)

[Jan Kotek]

Hi,

 

you modify record by copying it and inserting new value. So in this case you may use list, but have to make new instance after each insert.

 

What you really want is called Multimap. There is example how MapDB handles it. I would also recommend to use `BTreeKeySerializer.TUPLE2` to minimise space used by this set.

 

https://github.com/jankotek/MapDB/blob/master/src/test/java/examples/MultiMap.java

 

Regards,

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Jan Kotek

 

Hope this helps.

[David Windrem]

Why do the serialized keys/values have to be immutable?  Does MapDB keep references to these objects in cache after they are serialized onto the storage medium?
Do the objects have to actually be immutable (not capable of change) or just in practice not be changed while they are in storage?
If I have a custom class that is mutable, do I have to copy its data into something like an immutable POJO and store the POJO into MapDB?
We have a relatively large data structure consisting of a large number of objects of different classes that we are currently serializing onto flash memory, starting at the root object.
The data structure when serialized (using java serialization) occupies approximately 2 megabytes in flash (file) storage.
We would like to break this up by incrementally serializing smaller pieces to (a) help solve threading  problems (because the entire data structure cannot be locked during serialization as the application is soft realtime) and to improve performance (persisting parts instead of the whole thing when updates are made).  I'm trying to figure out if MapDB is a possible solution for us. However, our data structures are mostly mutable.

[Jan Kotek]

Hi,

 

Keys and values such as java.lang.String, Integer, Long... and so on are immutable. To update immutable value you simply insert new version using Map.put(key,value)

 

> Does MapDB keep references to these objects in cache after they are serialized onto the storage medium?

 

Yes, reference is kept to some of them (instance cache). It is to minimize serialization/deserilization overhead. There are several types of cache (hash table, MRU, hard/soft ref...)

 

> or just in practice not be changed while they are in storage?

Strict immutability is not required, as soon as you do not modify instances which are already stored. There is instance cache, background writer... so result of such modifications are hard to predict. I generally advice total immutability just because it is easer to grasp.

 

>If I have a custom class that is mutable, do I have to copy its data into something like an immutable POJO and store the POJO into MapDB?

 

Depends on your datamodel

 

> We have a relatively large data structure consisting of a large number of objects of different classes that we are currently serializing onto flash memory, starting at the root object.
The data structure when serialized (using java serialization) occupies approximately 2 megabytes in flash (file) storage.

 

If 2MB is total size of data mdoel than MapDB is probably overkill.

 

If 2MB is size of single key/value than you need to broke it down to smaller pieces. That 2MB chunk would be serialized/deserialized way to often and performance would be just terrible. Perhaps you could put this 2MB chunk each into separate map?

 

> I'm trying to figure out if MapDB is a possible solution for us. However, our data structures are mostly mutable.

 

MapDB is probably good fit. In fact it is more like 'alternative memory model' (to heap) rather than database. You would have to replace references in your model by 'recids' and use 'Engine' interface to retrieve/update them.

Contact me with better description of your datamodel and I would be happy to help.

 

Other alternative is to use 'real' object database such as DB4O.

 

Regards,

Jan