Idea about introducing MapMap (map of maps) similar to MultiMap

[mongonix]

Hi,

I've thought about the following idea over the weekend:

MultiMap is a very useful data-structures in many cases, but its set of values for each key is represented as a collection (e.g. list or set).
For some use-cases though, it could be interesting to have a very similar data structure, which uses usual java.util.Maps for values. Let's  call such values "value-maps" to distinguish them from the usual HZ maps. In all other respects, its overall design should be similar to MultiMap. It means, among other things that a value-map belonging to a given key is not distributed and is handled as a usual value by HZ, just like it is done for a MultiMap values collection belonging to the same key.

The methods for manipulating entries of a MapMap would be of course Map specific, e.g:
Map<K2,V> myMap = ...;

// K1 is the class for usual HZ keys, K2 is the class used for the keys of a value-map, V is the type used for values of a value-map
MapMap<K1, K2, V> mapMap = hzInstance.getMapMap(mapMapName);

// Add A->B to the map associated with key1
mapMap.put("key1", "A", "B");

// Remove an entry from a value-map
mapMap.remove("key1", "A");

// Add a whole map in one go
mapMap.putAll("key1", myMap);

// Get the (copy of) value-map. Direct changes on a value-map are not propagated!
// Always use MapMap operations to update value-maps.
Map<K2, V> value = mapMap.get("key1");

Similar to MultiMap, those updates operations could be optimized, i.e. HZ would not serialize the whole value-map on every update, but only changed entries.

What do think about it? Does it make sense?
How difficult would it be to implement such a data structure?
Is it correct to assume that it is very similar to MultiMaps and a lot of code can be reused? Or do you think it would require a very different implementation due to some subtle details?

Thanks,
  Leo

[Fuad Malikov]

Yes, we thought about it and it could be a good candidate to store session data in distributed fashion. And the updates would be very less costly. But I don't like the name MapMap. Can we have any better name?

-Fuad

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/72_MqPQwj7MJ.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[mongonix]

Hi Fuad,

On Monday, May 7, 2012 12:02:04 PM UTC+2, Fuad Malikov wrote:

Yes, we thought about it and it could be a good candidate to store session data in distributed fashion. And the updates would be very less costly.

Yes, absolutely!

Question: Is it already on the roadmap? If so, when is this feature planned?
If not, should I file an issue on github?
 

But I don't like the name MapMap. Can we have any better name?

Sure. I'm also not happy about the name ;-( I introduced MapMap just a temporary name and tried to follow the MultiMap naming convention. (actually, MultiMap would be a very nice name for MapMap, but it is used for lists already ;-)
And in Infinispan, they call a similar thing AtomicHashMap, but I'm not sure that it is much better and reflects the semantics.

So, may be something like ValueMap, because this map is a value in a distributed map? Later, if someone wants to introduce similar approach for e.g. sets and so on, one could use ValueSet, ValueList, ValueXYZ, etc. Anyway, I don't have any strong opinion about a name for such a thing. Suggestions are welcome!

-Leo
 

-Fuad

[Peter Veentjer]

Although not spectacular, what about the ManyMap.

> --
> You received this message because you are subscribed to the Google Groups
> "Hazelcast" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/hazelcast/-/1OUfRNkT1R4J.

[Tim Peierls]

Guava calls it Table.

[mongonix]

Hi,

I created a new issue on github for this feature:
https://github.com/hazelcast/hazelcast/issues/147

All name proposals suggested so far are listed there.

-Leo

[Nils Kilden-Pedersen]

On Mon, May 7, 2012 at 11:02 AM, Fuad Malikov <fu...@hazelcast.com> wrote:

Yes, we thought about it and it could be a good candidate to store session data in distributed fashion. And the updates would be very less costly. But I don't like the name MapMap. Can we have any better name?

NestedMap?

[Hugo]

TableMap?
DictionaryMap?
KeyValueMap?
ColumnFamily?

On May 8, 2:33 pm, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

> On Mon, May 7, 2012 at 11:02 AM, Fuad Malikov <f...@hazelcast.com> wrote:
> > Yes, we thought about it and it could be a good candidate to store session
> > data in distributed fashion. And the updates would be very less costly. But
> > I don't like the name MapMap. Can we have any better name?
>
> NestedMap?
>
>
>
>
>
> > -Fuad
>

> >http://groups.google.com/group/hazelcast?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Fuad Malikov]

Answers are inlined:

On Mon, May 7, 2012 at 1:38 PM, mongonix <romi...@gmail.com> wrote:

Hi Fuad,

On Monday, May 7, 2012 12:02:04 PM UTC+2, Fuad Malikov wrote:

Yes, we thought about it and it could be a good candidate to store session data in distributed fashion. And the updates would be very less costly.

Yes, absolutely!

Question: Is it already on the roadmap? If so, when is this feature planned?
If not, should I file an issue on github?

Thanks for the issue. No ETA yet. In the past we had people from community with strong ninja skills that implemented Semaphore, CountdownLatch and etc:) 

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/1OUfRNkT1R4J.

[Fuad Malikov]

I started a poll on linkedin. It only allows 5 choices per poll and we had 8 choices right now. I had to start 2 polls with 4 choices on each. If you have some new ideas, add them as a comment. 

http://www.linkedin.com/groups/1-How-should-we-name-2991377.S.113927868?qid=edcfdc49-d49d-4f4f-95bb-13ab25061098&trk=group_most_recent_rich-0-b-ttl&goback=%2Egmr_2991377

http://www.linkedin.com/groups/2-How-should-we-name-2991377.S.113929589?qid=edcfdc49-d49d-4f4f-95bb-13ab25061098&trk=group_most_recent_rich-0-b-ttl&goback=%2Egmr_2991377

-Fuad

[Fuad Malikov]

Based on the poll results, the winner is NestedMap. This was Nils's suggestion, thank you. There were some other suggestions like MultiKeyMap, which I couldn't include into poll. Is there any ninja coder volunteering for the development of NestedMap?:)

-Fuad

[mongonix]

Hi,

On Wednesday, May 16, 2012 8:05:17 AM UTC+2, Fuad Malikov wrote:

Based on the poll results, the winner is NestedMap. This was Nils's suggestion, thank you. There were some other suggestions like MultiKeyMap, which I couldn't include into poll. Is there any ninja coder volunteering for the development of NestedMap?:)

I could give it a try.

Could you give me some hints where to look at and roughly what pieces of the Hazelcast code-base (which classes, which commands, what should be extended, etc) should be touched when a new distributed data-structure like this is added to the library? This would help me to achieve the results faster, because I will spend less time on digging out this knowledge by analyzing Hazelcast's code.

Thanks,
   Leo

[Fuad Malikov]

Hi Leo,

You basically should analyze MultiMap and you can start by looking at classes:

MultiMapProxyImpl

ConcurrentMapManager

CMap

PutMultiOperationHandler

Thanks,

-fuad

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/okY4ZvzLjOUJ.

[mongonix]

Hi,

On Tuesday, May 22, 2012 3:17:32 PM UTC+2, Fuad Malikov wrote:

Hi Leo,

You basically should analyze MultiMap and you can start by looking at classes:

MultiMapProxyImpl

ConcurrentMapManager

CMap

PutMultiOperationHandler

I started looking into the code to see what needs to be adapted and added. The initial analysis shows that handling of MultiMaps as it is implemented now is rather complex and spread over quite some places (including Record, transactions, requests, etc), touching a lot of rather sensitive places. There is a lot of places, where there is a hard-coded explicit check if current data structure is a MultiMap and a special logic to handle it. Plus, the code assumes that it know how MultiMaps are represented internally (e.g. sets or lists) and uses this knowledge (as opposed e.g. to treating MultiMaps as black-boxes and delegating all the logic to a small set of MultiMap specific classes). IMHO, based on my still rather limited knowledge of HZ codebase, the current implementation does not make it very easy to add a new data structure if it is different enough from existing ones in terms of APIs (e.g. each operation takes more than one key, etc).

With this in mind, I see the following potential approaches:

1) Copy/paste all MultiMap related code and introduce corresponding NestedMap related code. It will probably work, but will introduce a lot of very similar code.
In addition, I have to add at very many places methods with NestedMap suffix/prefix, as it is done for MultiMaps currently, and this will pollute the code even more. Many of those methods with NestedMap suffix would allow for (key1, key2, value) parameters, similar to the way how it is done for MultiMap with (key, value) parameters (e.g. MProxy should be extended and so on). So, it will work, but it will not look like a nice and easily maintainable code. It would also require quite some effort. And should one want to add a new data structure in the future, the situation may become even worse.

2) An alternative could be to treat in the most places in code the last two elements of the (K1, K2, V) tuple, i.e. (K2, V),  as a compound-value and sort of a black-box, whose semantic is not known to majority of the code. Such a compound-value is treated almost as an atomic value and can be e.g. internally sent/received during get/put operations. In this sense, it is not different to the way how usual simple map values are treated by Hazelcast. Only the endpoints of the chain exchanging these atoms need to know how to interpret this value when trying to insert it into the map. I.e. instead of inserting it into the map in a usual way like it is done e.g. for integers, it would split it into K2 and V parts and insert it into the local value-map belonging to the key K1. When one needs to send (K1, K2, V) for a put operation, then the opposite will be done. (K2, V) will be combined into one new value NV, which will be serialized and a usual map update operation will be sent for key K1 and value NV. The other side will receive it in a usual way and only when it will be about to update the entry for K1, it will check that the target entry is not a usual value, but a Map and therefore NV needs to be split into parts and processed accordingly.
I think this approach may cover also other data structures and e.g. MultiMap can be expressed in its terms as well. More over, it would allow for many further custom data-structures that are also delta-aware similar to MultiMap and NestedMap.

I hope I managed to explain my ideas in more or less understandable way ;-) And I'm very eager to hear your opinions on these issues, before a lot of time is spent on either of the proposed solutions.

Thanks,
  Leo

[mongonix]

Hi,

To avoid pollution of this group with too many implementation details, I copied/moved the implementation discussion to the corresponding issue on GitHub:
https://github.com/hazelcast/hazelcast/issues/147

Hope to see some responses soon, so that I can proceed with the implementation.

Thanks,
  Leo

[mongonix]

Hi Talip, Hi Fuad,

Ping!
Don't want to annoy you, but some feedback on this topic would be really nice. Without it I cannot proceed with the implementation. We need to decide which direction should be taken.

Thanks,
   Leo

[Fuad Malikov]

Hi Leo, 

Sorry for that. Both of us are traveling. I just returned and will have a look at it.

Regards,

Fuad Malikov

Sent from Mobile

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/N0P1WSvY8eIJ.

[mongonix]

The usual monthly ping :-)

Any hope to get an answer regarding how to proceed?

Thanks,

   Leo

On Tuesday, May 29, 2012 10:47:17 AM UTC+2, Fuad Malikov wrote:

Hi Leo, 

Sorry for that. Both of us are traveling. I just returned and will have a look at it.

Regards,

Fuad Malikov

Sent from Mobile

To unsubscribe from this group, send email to hazelcast+unsubscribe@googlegroups.com.

[Fuad Malikov]

Sorry Leo,

I have replied at issue 147. 
Fuad Malikov
Co-founder & Managing Partner
Hazelcast | Open source in-memory data grid
Mobile: +90.538.378.9777 | @fuadm

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/a_tAAyRqHSoJ.

To post to this group, send email to haze...@googlegroups.com.

To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.