Hazelcast Enterprise 3.4 Eager Loading of Maps Not Happening

[Richard]

We are testing an application developed using Hazelcast Enterprise 3.3.3 which was eager loading Maps which implemented MapStore/MapLoader. Once we started to use 3.4 we have noticed that the maps are not eager loaded despite that part of the configuration of the maps remaining the same. Using the Hazelcast Management Centre also shows the maps have not eager loaded as does placing breakpoints in debuggers.

The map configuration is ok as once application logic invokes getMap(), the map will load its data but we need the eager loading to work. Has anyone else seen this problem ?

Richard

[Ahmet Mircik]

Hi Richard, 

This should not be happen. 

According to the tests here, https://github.com/hazelcast/hazelcast/blob/v3.4/hazelcast/src/test/java/com/hazelcast/map/mapstore/MapStoreTest.java #testInitialLoadModeEager it is working as expected.

Can you send us a simple reproducer test for your case? 

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/4c80a814-752d-44df-be5a-ac6dc9f96c58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard]

Hi Ahmet,

Thanks for your reply. I will give some more details on how we are experiencing this problem.

1. We create a Hazelcast instance as a Spring bean via java configuration and it gets created successfully.
2. We have another bean which has a @PostConstruct method that uses the Hazelcast Instance bean to iterate over its configuration and for each IMap which is configured for EAGER loading calls getMap(). We do this so that we force the eager loading of data before all beans are created in the Spring context and before our application starts processing data.

This worked fine under 3.3.3 but with 3.4 when we call getMap(), in the code in 2. above, for the first time on which ever IMap is first to be eager loaded the call will block and never return and no call is ever made into the loadAllKeys() methods of the underlying MapStore class.

We have found that by commenting out this code in 2. and then later inspecting a map contents, the first call to getMap() will result in eager loading, at this time the Spring context has been fully created.

[Ahmet Mircik]

Hi Richard,

For Hazelcast versions < 3.4 getMap was also blocking in LAZY mode;

It was like this:
LAZY --> blocks getMap till the end of all keys load,
EAGER --> blocks getMap till the end of key + value pairs load

But now for the versions >= 3.4:
LAZY --> not blocks getMap,
EAGER ---> blocks getMap till the end of all key + value pairs load (this is same with old behavior).

Only difference should be this.
Hope this helps you.

PS: If you share here a reproducer test code with your spring configurations, it may also make things more clear. 

​

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/123b1b7b-2e98-4e51-9a55-3f9e3f3159e7%40googlegroups.com.

[Richard]

Hi Ahmet,

Thanks for the info on the changes in LAZY loading, its useful to know the change as I was not aware of it.

Here is a stack trace of the thread which is blocked waiting for getMap() to return, it may be of some use.


"admin-thread-pool-25448(5)" daemon prio=6 tid=0x0000000022ab4000 nid=0xd64 waiting on condition [0x000000001adab000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.hazelcast.map.impl.proxy.MapProxySupport.waitUntilLoaded(MapProxySupport.java:593)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.waitUntilLoaded(MapProxyImpl.java:72)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initializeMapStoreLoad(MapProxySupport.java:157)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initialize(MapProxySupport.java:149)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.initialize(MapProxyImpl.java:72)
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.createProxy(ProxyServiceImpl.java:339)
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.getOrCreateProxy(ProxyServiceImpl.java:310)
    at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:131)
    at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:403)
    at com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:191)
    at com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:71)

[Richard]

Hi Ahmet,

I have a theory about what the issue is with eager loading a Map whilst Spring is initializing. In 3.x this worked fine but stopped working in 3.4 This may be because the loadAllKeys() method of a MapStore is now executed on its own thread where as before only the data loading of values in loadAll() was taking place on a new thread. In our case we use SpringData to load the keys via a Repository but I think the Spring bean dependency tracking is now failing or Spring cant hand out the transaction manager bean as another thread is currently initializing a bean in PostConstruct as mentioned in an earlier thread.

I also note discussions on this topic here https://github.com/hazelcast/hazelcast/issues/3222

Can you confirm if loadAllKeys() now runs on its own thread as I suspect and this si new as of 3.4 ? To be honest if this is the cause its a real pain as making sure all Hazelcast EAGER maps are loaded before the Spring context completes is very useful for us.

[Ahmet Mircik]

Yes loadAllKeys() call is offloaded to a different thread. But in any case, getMap should be a blocking operation for EAGER maps and this offloading should not affect your code.

Are you sure that your maps are using EAGER loading? Because i also tested with Spring and it worked fine.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d2975163-4d13-463d-b401-833e1bf4182f%40googlegroups.com.

[Richard]

Hi Ahmet,

Thanks for the confirmation that the loadAllKeys() method is now running on its own thread, new functionality added in 3.4 This will stop users such as us from EAGER loading the maps as Spring is initializing.

As for your question 'Are you sure that your maps are using EAGER loading?' yes I am absolutely sure we have correctly configured our maps to EAGER load. The problem we are encountering is down the fact that the MapStore loadAllKeys() method now runs on a different thread than in version prior to 3.4 and this is a problem when EAGER loading during the Spring context initialization of beans.

[Ahmet Mircik]

Hi Richard,

It would be very helpful if you can share your configurations, code-snippets or a reproducer test; this will help in making things clear, because according to my tests getMap with EAGER loading enabled is still a blocking operation. Here is a thread dump from my tests.

"main" #1 prio=5 os_prio=31 tid=0x00007fd0ea006000 nid=0x1303 waiting on condition [0x00000001042fe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.hazelcast.map.impl.proxy.MapProxySupport.waitUntilLoaded(MapProxySupport.java:607)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.waitUntilLoaded(MapProxyImpl.java:71)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initializeMapStoreLoad(MapProxySupport.java:160)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initialize(MapProxySupport.java:152)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.initialize(MapProxyImpl.java:71)
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.createProxy(ProxyServiceImpl.java:339)
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.getOrCreateProxy(ProxyServiceImpl.java:310)
    at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:131)
    at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:403)
    at com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:191)
    at com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:71)
    at com.hazelcast.spring.TestFullApplicationContext.testMapConfig(TestFullApplicationContext.java:195)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)

​

On Fri, Jan 9, 2015 at 1:43 PM, Richard <sam0...@googlemail.com> wrote:

Hi Ahmet,

Thanks for the confirmation that the loadAllKeys() method is now running on its own thread, new functionality added in 3.4 This will stop users such as us from EAGER loading the maps as Spring is initializing.

As for your question 'Are you sure that your maps are using EAGER loading?' yes I am absolutely sure we have correctly configured our maps to EAGER load. The problem we are encountering is down the fact that the MapStore loadAllKeys() method now runs on a different thread than in version prior to 3.4 and this is a problem when EAGER loading during the Spring context initialization of beans.

On Thursday, 8 January 2015 20:08:55 UTC, Ahmet Mircik wrote:

Yes loadAllKeys() call is offloaded to a different thread. But in any case, getMap should be a blocking operation for EAGER maps and this offloading should not affect your code.

Are you sure that your maps are using EAGER loading? Because i also tested with Spring and it worked fine.

On Thu, Jan 8, 2015 at 7:40 PM, Richard <sam0...@googlemail.com> wrote:

Hi Ahmet,

I have a theory about what the issue is with eager loading a Map whilst Spring is initializing. In 3.x this worked fine but stopped working in 3.4 This may be because the loadAllKeys() method of a MapStore is now executed on its own thread where as before only the data loading of values in loadAll() was taking place on a new thread. In our case we use SpringData to load the keys via a Repository but I think the Spring bean dependency tracking is now failing or Spring cant hand out the transaction manager bean as another thread is currently initializing a bean in PostConstruct as mentioned in an earlier thread.

I also note discussions on this topic here https://github.com/hazelcast/hazelcast/issues/3222

Can you confirm if loadAllKeys() now runs on its own thread as I suspect and this si new as of 3.4 ? To be honest if this is the cause its a real pain as making sure all Hazelcast EAGER maps are loaded before the Spring context completes is very useful for us.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d2975163-4d13-463d-b401-833e1bf4182f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/6b83f4c5-30f0-4395-b26a-4292a1d360d4%40googlegroups.com.