How could I delay/hold the partition to a new cluster member that joining until I received a notification?

[Daniel Paniagua]

Hello, 

I have a clustered java application (Spring + Hibernate) that uses a Hazecast (hazelcast-spring, v3.4) as a 2nd level cache, and also uses a Hazelcast Mapstore that persist the changes to the DB using Hibernate (adapted from this example). 

So my application has to initialize a Hazelcast instance (that joins a existing cluster if present) as part of its loading process. 

This woks well with only 1 member in the cluster. So the Mapstore gets new data and this is being persisted normally to the database.  

However, when a new application joins the cluster (and in turns joins the hazelcast cluster), Hazelcast tries to partition the data immediately to the new member before the new application has initializes its database own sessionfactory completely, this causes a null-pointer exception (see below) since the session factory (on the 2nd instance) depends on hazelcast for the 2nd level cache and hazelcast (on the 1st instance) depends on the session factory to partition the data of the Mapstore.

So I need a way to avoid the partition process from happening until the new application is fully initialized and only then let Hazelcast know so it can start the partitioning to the new member. 

I checked the hazelcast advanced configuration properties but couldn't find anything that could control this. In any case, I will prefer something that gives me more control, rather than some time-based delay (e.g. a new listener/event or something similar if available) 

Any ideas?

I hope question is clear and thanks in advance for any assistance...

P.S:

Error when the new application member joins the cluster: 

hz.hazelCastInstance.partition-operation.thread-1] 18Sep15-10:10:29 ERROR(TokenMapStore.java:96)-Failed to load
java.lang.NullPointerException
	at xxx.hazelcast.map.hibernate.HazelcastMapStoreHibernateDao.getCurrentSession(HazelcastMapStoreHibernateDao.java:104)

...

at com.hazelcast.map.impl.MapStoreWrapper.load(MapStoreWrapper.java:110)
	at com.hazelcast.map.impl.mapstore.writebehind.WriteBehindStore.load(WriteBehindStore.java:137)
	at com.hazelcast.map.impl.mapstore.writebehind.WriteBehindStore.load(WriteBehindStore.java:24)
	at com.hazelcast.map.impl.DefaultRecordStore.loadRecordOrNull(DefaultRecordStore.java:313)
	at com.hazelcast.map.impl.DefaultRecordStore.get(DefaultRecordStore.java:580)
	at com.hazelcast.map.impl.operation.GetOperation.run(GetOperation.java:43)
	at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:733)
	at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:709)
	at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:683)
	at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:627)
	at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1500(BasicOperationService.java:614)
	at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:566)
	at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)
	at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)
	at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)
...

Sample of my sectionFactory configuration:

<bean id="sessionFactory"
      class="org.springframework.orm.hibernate4.LocalSessionFactoryBean" depends-on="hazelCastInstance">
    <property name="dataSource" ref="dataSource" />
    <property name="mappingLocations" ref="defaultHibernateMappingLocationList"/>
    <property name="hibernateProperties">
        <props>
            <prop key="org.hibernate.FlushMode">COMMIT</prop>
            <prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto}</prop>
            <prop key="hibernate.dialect">${hibernate.dialect}</prop>
            <prop key="hibernate.show_sql">${hibernate.show_sql}</prop>
            <prop key="hibernate.generate_statistics">${hibernate.generate_statistics}</prop>
            <prop key="hibernate.cache.use_second_level_cache">${hibernate.cache.use_second_level_cache}</prop>
            <prop key="hibernate.cache.use_minimal_puts">true</prop>
            <prop key="hibernate.cache.region.factory_class">com.hazelcast.hibernate.HazelcastLocalCacheRegionFactory</prop>
            <prop key="hibernate.cache.use_structured_entries">${hibernate.cache.use_structured_entries}</prop>
            <prop key="hibernate.cache.hazelcast.instance_name">hazelCastInstance</prop>
            <prop key="hibernate.default_batch_fetch_size">${hibernate.default_batch_fetch_size}</prop>
            <prop key="hibernate.order_inserts">${hibernate.order_inserts}</prop>
            <prop key="hibernate.order_updates">${hibernate.order_updates}</prop>
            <prop key="hibernate.jdbc.batch_size">${hibernate.jdbc.batch_size}</prop>
        </props>
    </property>
</bean>

[Enes Akar]

I could not understand your scenario exactly but following system property may help:

hazelcast.initial.min.cluster.size

0

int

Initial expected cluster size to wait before node to start completely.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/c0d0f685-b0a1-4fc8-b0a6-eeca7b6da696%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[İbrahim Gürses]

Hi Daniel,

I could not understand your scenario but exception you get might be related to the issue here : https://github.com/hazelcast/hazelcast/issues/4456

Can you please give a name to your session factory in every application in your cluster?  I should prevent hibernate to serialize session factory during repartitioning.

<prop key="hibernate.session_factory_name">yourFactoryName</prop>

If it does not help, you can look at example here, which shows how to use hibernate + spring with map loaders.

https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/spring-hibernate-2ndlevel-cache

[Daniel Paniagua]

Hi, 

Thank you for responses. 

I tried setting the hibernate.session_factory_name and hazelcast.initial.min.cluster.size settings and on my preliminary tests appears to be working. 

One minor issue though, by setting session_factory_name attribute, hibernate bind the session factory to the JNDI context (see doc below), this causes some error messages during the (junit) test execution  since there is no JNDI context defined (its outside a application server container.) so I had to define a "mock" for that case. 

Ibrahim, have you experience this? if so how did you deal with it?

Regards, 

P.s: 

Hibernate properties documentation:

hibernate.session_factory_name,

If set, the factory attempts to bind this name to itself in the JNDI context. This name is also used to support cross JVM Session (de)serialization.

Again to reiterate, my use case was: 

An instance of a java application (that can be in cluster mode), uses hazelcast as 2nd level cache therefore depends on a hazelcast instance to initialize.

The app also uses the a hibernate-backed MapStore that depends on the Session factory for persistence, hence the circular dependency: 

The hazelcast (hibernate-backed) Maptore depends on the session factory and the session factory depends on hazelcast for the 2nd level cache. 

So, when a 2nd app instance starts, the hazelcast starts partition the data to a the 2nd app whose session factory hasn't not completely initialized yet.

[İbrahim Gürses]

Hi Daniel, 

I think you get `javax.naming.NoInitialContextException` in your unit tests, because there is no initial context defined for JNDI. I could not find any alternative solution different than your solution.  Also, I could understand your problem more clearly.  I think creation of session factory takes too much time in your case, do you have a big entity object graph defined in hibernate config xml. I could not reproduce your case with our sample projects, because they contain one or two entities and session factory is created before hazelcast instance starts repartitioning the data.  Also how do you inject your session factory inside your MapStore? See sample project for MapLoader with hibernate : https://github.com/hazelcast/hazelcast-code-samples/tree/master/distributed-map/mapstore/src/main/java/hibernate 

Best 

İbrahim.

On Tuesday, September 22, 2015 at 2:45:51 AM UTC+3, Daniel Paniagua wrote:

[Daniel Paniagua]

Hi, 
I have around 80 entities in total.
Regarding the injection of the session, I use a utility class: "HazelcastSessionFactoryUtil" very similar to the sample you referred; that gets a session (if not present) from sessionFactory defined in the context.
See the sessionFactory bean definition on my initial email.

Thanks, 

Daniel

[se...@ikentoo.com]

Hi,

Could you explain exactly what is implied by "node to start completely" ? What does the node do or not do while its not "started completely" ?

Thanks,

Serge

[İbrahim Gürses]

Hi Serge,

By setting two properties below, you can delay node start, meaning you will be blocked when calling Hazelcast.newHazelcastIntance() until requirements in those properties are met.

hazelcast.initial.min.cluster.size

0

int

Initial expected cluster size to wait before node to start completely.

hazelcast.initial.wait.seconds

0

int

Initial time in seconds to wait before node to start completely.

On Tuesday, September 22, 2015 at 2:45:51 AM UTC+3, Daniel Paniagua wrote:

[se...@ikentoo.com]

Hi Ibrahim,

Thanks this will not help us in this case since it completely blocks :-(

We have an issue when using Spring Framework where we inject dependencies into our MapStore implementation but it seems that Hazelcast starts to use this MapStore instance before it has been fully wired (during partition operation at node startup)!

i.e. the dependecies have not yet been injected but the instance exists.

Thanks,

Serge

[se...@ikentoo.com]

Ok so continueing on with this issue. 

Is it possible to delay that start of the repartition process? It seems like you can do it for node removal using 

hazelcast.migration.min.delay.on.member.removed.seconds

But what about when a node is added, how can you delay the start of the repartition process ?

Thanks,

Serge

[Jaromir Hamala]

Hi Serge,

I haven't tested it, but this seems like what you are looking for: https://github.com/hazelcast/hazelcast/blob/9815def4bac321ee2cc1c197dbd37d64396ce844/hazelcast/src/main/java/com/hazelcast/instance/GroupProperty.java#L251

Cheers,

Jaromir

[Sozonoff Serge]

Thanks Jaromir,

I have tested this parameter already and sadly it seems to block the returning of Hazelcast.newHazelcastInstance() call until the join has actually taken place.
Hence it blocks the startup of the App

Bottom line is that if the repartition process happens before the Spring Application has completely initialised then our MapStore implementation, which uses Spring injected dependencies, fails.
We would need a way to

a. delay the repartition process and so give time for the App to complete startup
b. disable auto repartitioning and programatically kick of the process ourselves from code

Serge

[Jaromir Hamala]

I see. I am afraid there is no way to achieve that at this moment. May I kindly ask you to create a new issue at https://github.com/hazelcast/hazelcast/issues/new and describe your scenario.

Many thanks!

Jaromir

[se...@ikentoo.com]

Done : https://github.com/hazelcast/hazelcast/issues/6922