MapStore and missing entries

[Thomas Hallgren]

While experimenting with a MapStore implementation I noticed that an
attempt to lookup a non existent key results in a MapStore lookup. I
would like to avoid that. All keys are loaded on start-up so it should
be quite safe to assume that if the key is missing from the in-memory
map, then it is really missing. There's no need to check with the
MapStore.

Any thoughts on that?

[Tim Peierls]

If you're loading all keys on startup and the key set doesn't change thereafter, it's probably simpler to avoid the use of MapLoader/MapStore entirely. Just use the IMap as a cache for reads, and write values through to both the IMap and the backing store.

--tim

[Thomas Hallgren]

> If you're loading all keys on startup and the key set doesn't change
> thereafter, it's probably simpler to avoid the use of MapLoader/MapStore
> entirely. Just use the IMap as a cache for reads, and write values through
> to both the IMap and the backing store.
>

The key set does change. But all changes are reflected in the in-
memory cluster first and then, asynchronously pushed to the persistent
store.

The only reasons I can see for consulting the persistent store is if
some eviction policy is in effect (in my case there's no such policy),
or if the persistent store was modified by some external source (god
forbid). In all other cases I fail to see a reason to check for the
presence of a value that isn't present in the in-memory cluster. So
why does that happen?

- thomas

[Tim Peierls]

On Mon, Oct 17, 2011 at 6:12 PM, Thomas Hallgren <iro...@gmail.com> wrote:

The only reasons I can see for consulting the persistent store is if

some eviction policy is in effect (in my case there's no such policy),
or if the persistent store was modified by some external source (god
forbid). In all other cases I fail to see a reason to check for the
presence of a value that isn't present in the in-memory cluster. So
why does that happen?

Because in general the full key set is not loaded into the cluster. Your case is special, and you don't need MapStore/MapLoader at all.

--tim

[Thomas Hallgren]

> Because in general the full key set is *not* loaded into the cluster. Your

> case is special, and you don't need MapStore/MapLoader at all.
>

I think I do. I want to use it as a disk based backup so that I can
restart the cluster as a whole without loosing any data. But I don't
want it to slow down normal processing.

Another question. Let's assume that I have a 100 seconds write delay
for this storage. I delete using key 'x' from the map. I then load 'x'
from the map again. At that time the delete of 'x' hasn't yet been
flushed.
Will the missing 'x' cause a MapStore lookup? I hope not, because that
would be wrong.

[Talip Ozturk]

> I think I do. I want to use it as a disk based backup so that I can
> restart the cluster as a whole without loosing any data. But I don't
> want it to slow down normal processing.

How about always returning Null for MapLoader.load(key)? Yes Hazelcast
will still call your load(key) but
at least it will return fast.

> Another question. Let's assume that I have a 100 seconds write delay
> for this storage. I delete using key 'x' from the map. I then load 'x'
> from the map again. At that time the delete of 'x' hasn't yet been
> flushed.
> Will the missing 'x' cause a MapStore lookup? I hope not, because that
> would be wrong.

Yes it will lookup. Currently Hazelcast cannot distinguish 'eviction'
and 'deletion' so it will try to load regardless. I think this part
can be improved.

-talip

[Tim Peierls]

On Mon, Oct 17, 2011 at 6:46 PM, Thomas Hallgren <iro...@gmail.com> wrote:

> Because in general the full key set is *not* loaded into the cluster. Your

> case is special, and you don't need MapStore/MapLoader at all.

I think I do. I want to use it as a disk based backup so that I can
restart the cluster as a whole without loosing any data. But I don't
want it to slow down normal processing.

You can invert the relationship between the MapStore/Loader logic and the IMap without changing the performance characteristics. Instead of IMap.get triggering MapLoader.load, you can roll your own ConcurrentMap decorator for your IMap that loads the initial key set, serves get requests directly, and writes puts through to the backing store.

I started out using MapLoader/MapStore heavily, even to the point of making a feature request (MapStoreFactory), so I could configure MapStore instances from information encoded in the map name. Lately, though, it has been simpler for me to manage the backing store directly myself rather than indirectly through MapStore/Loader.

I found Guava's ForwardingConcurrentMap invaluable in writing my decorator class, btw.

--tim