need to create two separate clusters in hazelcast.xml configuration

Bill Rodriguez

unread,

Apr 19, 2018, 10:23:34 AM4/19/18

to Hazelcast

I am running hazelcast 3.8.4

We have two applications that need separate clusters. How do I setup the hazelcast,xml file to accomplish this?

Both clusters will be using different port numbers. I cannot find an example of hazelcast.xml to do this?

thanks

Bill

Lead Engineer at Landis&Gyr

bill.ro...@landisgyr.com

Alparslan Avcı

unread,

Apr 19, 2018, 10:39:50 AM4/19/18

to Hazelcast

Hello Bill,

You can create different clusters by using a different `group` configuration. Please check the link below to see how you can configure it:

http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#creating-cluster-groups

Regards,

Alparslan

Bill Rodriguez

unread,

Apr 19, 2018, 11:07:07 AM4/19/18

to Hazelcast

Alparslan,

Thanks for the reply. I am still a bit puzzled.

Can I have two group tags in the hazelcast.xml file and create a separate instance of the each group programmatically?

ex:

hazelcast.xml:

<hazelcast> <group> <name>production1</name> <password>prod-pass</password> </group> ...

<group> <name>production2</name> <password>prod-pass</password> </group> ...

java code:

Then instantiate the instances as follows:

onfig configProd = new Config(); configProd.getGroupConfig().setName( "production1" ).setPassword( "prod-pass" );

HazelcastInstance h1 = Hazelcast.newHazelcastInstance( configProd ); Config configDev = new Config(); configDev.getGroupConfig().setName( "production2" ).setPassword( "prod-pass" );

HazelcastInstance h2 = Hazelcast.newHazelcastInstance( configDev );

Alparslan Avcı

unread,

Apr 19, 2018, 2:29:22 PM4/19/18

to Hazelcast

Hi again Bill,

You can't have two group configuration in the same hazelcast.xml file.

Each Hazelcast cluster instance needs a configuration (either declaratively or programmatically). If you create an instance using hazelcast.xml file, it will use all the configuration elements in the file including the group configuration. If you create another instance based on the same hazelcast.xml file, it will again use the same configuration elements in the file. So, both instances will have the same group config. If you need two different clusters, each Hazelcast instance should be based on different XML configuration files, which contain different group configurations.

The same applies for the programmatic configuration. Thus when you are creating a Hazelcast instance like your example, you can give a Config instance and configure its group property. If you provide different configurations for each instance, then you will have two instances within two different clusters.

Hope this helps.

Alparslan

Bill Rodriguez

unread,

Apr 19, 2018, 3:23:17 PM4/19/18

to Hazelcast

Again thank you for your response. That helps alot-- now

we have tried to configure two separate groups using two separate xml files for two different applications running on same JVM-- as follows:

We added the following two entries into our domain.xml --such that two separate xml files where defined:

hazelcast.xml contained:

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.8.xsd"

xmlns="http://www.hazelcast.com/schema/config"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<group>

<name>dev0</name>

</group>

<instance-name>power-center<instance-name>

<management-center enabled="false">http://localhost:8080/mancenter</management-center>

......

hazelcast-multiscada.xml contained:

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.8.xsd"

xmlns="http://www.hazelcast.com/schema/config"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<group>

<name>dev1</name>

</group>

<instance-name>power-center<instance-name>

<management-center enabled="false">http://localhost:8080/mancenter</management-center>

......

Application one created an instance of hazelcast as follows:

Config config;

config = new FileSystemXmlConfig(System.getProperty("hazelcast.config"));

logger.info("Calling MS get hazelcast instance");

newHazelcastInstance = Hazelcast.newHazelcastInstance(config);

-------------

Application two created an instance as follows:

Config config;

config = new FileSystemXmlConfig(System.getProperty("hazelcast-multiscada.config"));

newHazelcastInstance = Hazelcast.newHazelcastInstance(config);

Application one started up fine and loaded the correct hazelcast.xml file. Application two read hazelcast-multiscada,xml, but then when the instance of hazecast was created

it applied hazelcast.xml and we got a port conflict since those ports had already been taken by application one.

Here are logs of second application :

-service-servlet': startup date [Thu Apr 19 13:14:04 UTC 2018]; root of context hierarchy

04/19/2018 13:14:08,624 INFO [org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver] (ServerService Thread Pool -- 115) Detected @ExceptionHandler methods in globalExceptionHandler

04/19/2018 13:14:08,694 INFO [com.landisgyr.multiscada.applicationContext.SpringApplicationContextFactory] (ServerService Thread Pool -- 115) SpringApplicationContext:store Inside Application Context

04/19/2018 13:14:08,700 INFO [com.landisgyr.multiscada.bootstrap.SchedulerBootStrap] (ServerService Thread Pool -- 115) Inside start scheduler

04/19/2018 13:14:08,721 INFO [com.hazelcast.config.FileSystemXmlConfig] (ServerService Thread Pool -- 115) Configuring Hazelcast from '/jboss/uat01/master/EAP-6.4.0/domain/configuration/hazelcast-multiscada.xml'.

04/19/2018 13:14:08,774 INFO [com.landisgyr.multiscada.scheduler.HazelCastMSScheduler] (ServerService Thread Pool -- 115) Calling MS get hazelcast instance

04/19/2018 13:14:08,802 INFO [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev1] [3.8.4] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.7.168.24, 10.7.168.23]

04/19/2018 13:14:08,802 INFO [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev1] [3.8.4] Prefer IPv4 stack is true.

04/19/2018 13:14:08,803 INFO [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev1] [3.8.4] Picked [10.7.168.23]:47478, using socket ServerSocket[addr=/0.0.0.0,localport=47478], bind any local is true

04/19/2018 13:14:08,811 INFO [com.hazelcast.system] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Hazelcast 3.8.4 (20170809 - 297a77e) starting at [10.7.168.23]:47478

04/19/2018 13:14:08,811 INFO [com.hazelcast.system] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Configured Hazelcast Serialization version : 1

04/19/2018 13:14:08,959 INFO [com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Backpressure is disabled

04/19/2018 13:14:09,304 INFO [com.hazelcast.instance.Node] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Creating TcpIpJoiner

04/19/2018 13:14:09,424 INFO [com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Starting 4 partition threads

04/19/2018 13:14:09,427 INFO [com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Starting 3 generic threads (1 dedicated for priority tasks)

04/19/2018 13:14:09,432 INFO [com.hazelcast.core.LifecycleService] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] [10.7.168.23]:47478 is STARTING

04/19/2018 13:14:09,449 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Connecting to /10.7.168.24:47478, timeout: 0, bind-any: true

04/19/2018 13:14:09,450 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Could not connect to: /10.7.168.24:47478. Reason: SocketException[Connection refused to address /10.7.168.24:47478]

04/19/2018 13:14:09,450 INFO [com.hazelcast.cluster.impl.TcpIpJoiner] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] [10.7.168.24]:47478 is added to the blacklist.

04/19/2018 13:14:09,452 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Connecting to /10.7.168.24:47479, timeout: 0, bind-any: true

04/19/2018 13:14:09,452 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-2) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Connecting to /10.7.168.23:47479, timeout: 0, bind-any: true

04/19/2018 13:14:09,452 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Could not connect to: /10.7.168.24:47479. Reason: SocketException[Connection refused to address /10.7.168.24:47479]

04/19/2018 13:14:09,452 INFO [com.hazelcast.cluster.impl.TcpIpJoiner] (hz._hzInstance_1_hazelcast_dev1.cached.thread-3) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] [10.7.168.24]:47479 is added to the blacklist.

04/19/2018 13:14:09,453 INFO [com.hazelcast.nio.tcp.InitConnectionTask] (hz._hzInstance_1_hazelcast_dev1.cached.thread-2) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Could not connect to: /10.7.168.23:47479. Reason: SocketException[Connection refused to address /10.7.168.23:47479]

04/19/2018 13:14:09,453 INFO [com.hazelcast.cluster.impl.TcpIpJoiner] (hz._hzInstance_1_hazelcast_dev1.cached.thread-2) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] [10.7.168.23]:47479 is added to the blacklist.

04/19/2018 13:14:10,453 INFO [com.hazelcast.system] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] Cluster version set to 3.8

04/19/2018 13:14:10,454 INFO [com.hazelcast.cluster.impl.TcpIpJoiner] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4]

Members [1] {

Member [10.7.168.23]:47478 - bbd08af8-4388-474a-b620-eb9b1d1c10c8 this

}

04/19/2018 13:14:10,483 INFO [com.hazelcast.core.LifecycleService] (ServerService Thread Pool -- 115) [10.7.168.23]:47478 [hazelcast_dev1] [3.8.4] [10.7.168.23]:47478 is STARTED

04/19/2018 13:14:10,504 INFO [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 115) Using default implementation for ThreadExecutor

04/19/2018 13:14:10,518 INFO [org.quartz.core.SchedulerSignalerImpl] (ServerService Thread Pool -- 115) Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl

04/19/2018 13:14:10,519 INFO [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 115) Quartz Scheduler v.2.2.1 created.

04/19/2018 13:14:10,520 WARN [com.bikeemotion.quartz.jobstore.hazelcast.HazelcastJobStore] (ServerService Thread Pool -- 115) Starting new local hazelcast client since not hazelcast instance setted before starting scheduler.

04/19/2018 13:14:10,521 INFO [com.hazelcast.config.XmlConfigLocator] (ServerService Thread Pool -- 115) Loading configuration /jboss/uat01/master/EAP-6.4.0/domain/configuration/hazelcast.xml from System property 'hazelcast.config'

04/19/2018 13:14:10,521 INFO [com.hazelcast.config.XmlConfigLocator] (ServerService Thread Pool -- 115) Using configuration file at /jboss/uat01/master/EAP-6.4.0/domain/configuration/hazelcast.xml

04/19/2018 13:14:10,555 INFO [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev0] [3.8.4] Interfaces is enabled, trying to pick one address matching to one of: [10.7.168.23]

04/19/2018 13:14:10,555 INFO [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev0] [3.8.4] Prefer IPv4 stack is true.

04/19/2018 13:14:10,556 SEVERE [com.hazelcast.instance.DefaultAddressPicker] (ServerService Thread Pool -- 115) [LOCAL] [hazelcast_dev0] [3.8.4] ServerSocket bind has failed. Hazelcast cannot start! config-port: 47476, latest-port: 47478: com.hazelcast.core.HazelcastException: ServerSocket bind has failed. Hazelcast cannot start! config-port: 47476, latest-port: 47478

at com.hazelcast.instance.DefaultAddressPicker.getPublicAddressByPortSearch(DefaultAddressPicker.java:146) [hazelcast-all-3.8.4.jar:3.8.4]

at com.hazelcast.instance.DefaultAddressPicker.pickAddress(DefaultAddressPicker.java:75) [hazelcast-all-3.8.4.jar:3.8.4]

at com.hazelcast.instance.Node.<init>(Node.java:183) [hazelcast-all-3.8.4.jar:3.8.4]

at com.hazelcast.instance.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:159) [hazelcast-all-3.8.4.jar:3.8.4]

We are really stuck and have spent lots of cycles getting both of the apps to run separate clusters-- PLS ADVICE what course of action we need

It is very much appreciated..

Bill

Bill Rodriguez

unread,

Apr 20, 2018, 1:56:01 PM4/20/18

to Hazelcast

Alparslan,

Any ideas concerning my last post?

I am stuck..

Thanks very much,

Bill

On Thursday, April 19, 2018 at 2:29:22 PM UTC-4, Alparslan Avcı wrote:

Alparslan Avcı

unread,

Apr 23, 2018, 6:10:47 AM4/23/18

to Hazelcast

Bill,

Can you please share the full configurations for the both clusters?

Thanks,

Alparslan

Bill Rodriguez

unread,

Apr 23, 2018, 11:23:37 AM4/23/18

to Hazelcast

Alparslan,

I figure it out-- When I was creating the instance of Hazelcast here:

newHazelcastInstance = Hazelcast.newHazelcastInstance(config);

I was never setting the instance to the client... Once I did the below-- then both applications loaded the correct config file and created the clusters correctly using specific

ports defined in the config--

HazelcastJobStore.setHazelcastClient(newHazelcastInstance);

it now works!!

Thanks again very much for triggering me to look for other avenues in the config..