Hazelcast nodes don't join the cluster and seem to timeout after 3 minutes

Dean Schulze

unread,

Aug 16, 2016, 9:24:46 PM8/16/16

to Hazelcast

I've got my first Hazelcast implementation working on the network at work. All machines have 10.xxx.yyy.zzz IP addresses and they are all Ubuntu 14.04 systems with one Windows 10 system.

When I run the same application on my network at home the nodes don't join a cluster. The two machines have 192.168.0.x IP addresses. They are Ubuntu 14 and Ubuntu 16 systems. TCP socket communication works as does UDP datagrams. They both pause for just over 3 minutes (maybe a default timeout ?) and then proceed without ever joining a cluster. The log output is below.

I've checked that MULTICAST is running in ifconfig and is in /cat/boot-config-....

Where do I look to find what the problem is? Thanks.

Ubuntu 14.04

Aug 16, 2016 6:42:30 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast-default.xml' from classpath.
Aug 16, 2016 6:42:30 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6.4] Prefer IPv4 stack is true.
Aug 16, 2016 6:42:30 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6.4] Picked Address[192.168.0.3]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Aug 16, 2016 6:42:30 AM com.hazelcast.system
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Hazelcast 3.6.4 (20160701 - 5b94d9f) starting at Address[192.168.0.3]:5701
Aug 16, 2016 6:42:30 AM com.hazelcast.system
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
Aug 16, 2016 6:42:30 AM com.hazelcast.system
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Configured Hazelcast Serialization version : 1
Aug 16, 2016 6:42:31 AM com.hazelcast.spi.OperationService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Backpressure is disabled
Aug 16, 2016 6:42:31 AM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Starting with 4 generic operation threads and 8 partition operation threads.
Aug 16, 2016 6:42:31 AM com.hazelcast.instance.Node
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Creating MulticastJoiner
Aug 16, 2016 6:42:31 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Address[192.168.0.3]:5701 is STARTING
Aug 16, 2016 6:42:31 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
INFO: [192.168.0.3]:5701 [dev] [3.6.4] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Aug 16, 2016 6:42:34 AM com.hazelcast.cluster.impl.MulticastJoiner
INFO: [192.168.0.3]:5701 [dev] [3.6.4] 


Members [1] {
	Member [192.168.0.3]:5701 this
}

Aug 16, 2016 6:42:34 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Address[192.168.0.3]:5701 is STARTED
Aug 16, 2016 6:42:34 AM com.hazelcast.partition.InternalPartitionService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Initializing cluster partition table arrangement...
main 2016-08-16 06:42:34,618 INFO  [GridBarrier:40] - Setting this client to owner status.
main 2016-08-16 06:42:34,627 INFO  [GridBarrier:50] - Waiting for 1 nodes.
main 2016-08-16 06:45:35,165 INFO  [GridBarrier:54] - Releasing Owner
Aug 16, 2016 6:45:35 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Address[192.168.0.3]:5701 is SHUTTING_DOWN
Aug 16, 2016 6:45:35 AM com.hazelcast.instance.Node
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Shutting down multicast service...
Aug 16, 2016 6:45:35 AM com.hazelcast.instance.Node
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Shutting down connection manager...
Aug 16, 2016 6:45:35 AM com.hazelcast.instance.Node
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Shutting down node engine...
Aug 16, 2016 6:45:35 AM com.hazelcast.instance.NodeExtension
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Destroying node NodeExtension.
Aug 16, 2016 6:45:35 AM com.hazelcast.instance.Node
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Hazelcast Shutdown is completed in 17 ms.
Aug 16, 2016 6:45:35 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.3]:5701 [dev] [3.6.4] Address[192.168.0.3]:5701 is SHUTDOWN



Ubuntu 16.04

Aug 16, 2016 6:42:24 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast-default.xml' from classpath.
Aug 16, 2016 6:42:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6.4] Prefer IPv4 stack is true.
Aug 16, 2016 6:42:25 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.6.4] Picked Address[192.168.0.30]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Aug 16, 2016 6:42:25 AM com.hazelcast.system
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Hazelcast 3.6.4 (20160701 - 5b94d9f) starting at Address[192.168.0.30]:5701
Aug 16, 2016 6:42:25 AM com.hazelcast.system
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
Aug 16, 2016 6:42:25 AM com.hazelcast.system
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Configured Hazelcast Serialization version : 1
Aug 16, 2016 6:42:25 AM com.hazelcast.spi.OperationService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Backpressure is disabled
Aug 16, 2016 6:42:25 AM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Starting with 2 generic operation threads and 4 partition operation threads.
Aug 16, 2016 6:42:25 AM com.hazelcast.instance.Node
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Creating MulticastJoiner
Aug 16, 2016 6:42:25 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Address[192.168.0.30]:5701 is STARTING
Aug 16, 2016 6:42:25 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
INFO: [192.168.0.30]:5701 [dev] [3.6.4] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Aug 16, 2016 6:42:27 AM com.hazelcast.cluster.impl.MulticastJoiner
INFO: [192.168.0.30]:5701 [dev] [3.6.4] 


Members [1] {
	Member [192.168.0.30]:5701 this
}

Aug 16, 2016 6:42:27 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Address[192.168.0.30]:5701 is STARTED
Aug 16, 2016 6:42:27 AM com.hazelcast.partition.InternalPartitionService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Initializing cluster partition table arrangement...
main 2016-08-16 06:42:27,896 INFO  [GridBarrier:40] - Setting this client to owner status.
main 2016-08-16 06:42:27,903 INFO  [GridBarrier:50] - Waiting for 1 nodes.
main 2016-08-16 06:45:28,355 INFO  [GridBarrier:54] - Releasing Owner
Aug 16, 2016 6:45:28 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Address[192.168.0.30]:5701 is SHUTTING_DOWN
Aug 16, 2016 6:45:28 AM com.hazelcast.instance.Node
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Shutting down multicast service...
Aug 16, 2016 6:45:28 AM com.hazelcast.instance.Node
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Shutting down connection manager...
Aug 16, 2016 6:45:28 AM com.hazelcast.instance.Node
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Shutting down node engine...
Aug 16, 2016 6:45:28 AM com.hazelcast.instance.NodeExtension
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Destroying node NodeExtension.
Aug 16, 2016 6:45:28 AM com.hazelcast.instance.Node
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Hazelcast Shutdown is completed in 13 ms.
Aug 16, 2016 6:45:28 AM com.hazelcast.core.LifecycleService
INFO: [192.168.0.30]:5701 [dev] [3.6.4] Address[192.168.0.30]:5701 is SHUTDOWN

Nick Pratt

unread,

Aug 16, 2016, 9:44:22 PM8/16/16

to haze...@googlegroups.com

Maybe check the firewall on the 2 boxes?

N

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/785b7791-f9c3-4810-a4e8-c4d46b7466ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dean Schulze

unread,

Aug 18, 2016, 1:55:57 AM8/18/16

to Hazelcast

Both systems have firewall status inactive.

Anything else I should check?

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

Noctarius

unread,

Aug 18, 2016, 2:18:13 AM8/18/16

to Christoph Engelbert - Hazelcast

Hey,

Is multicast possible in your network? Sometimes networks prevent multicast from working, e.g. using firewalls or managed switches blocking the actual traffic.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2b5b1c4c-5ce4-4fad-8df0-6904180e4ebe%40googlegroups.com.

Dean Schulze

unread,

Aug 18, 2016, 4:54:49 PM8/18/16

to Hazelcast

UDP datagrams work over my network and switch. (My switch is a low end gigabit switch.) I assumed that implied that multicast would work too. Is that a bad assumption?

Dean Schulze

unread,

Aug 18, 2016, 8:10:45 PM8/18/16

to Hazelcast

I did some checking on my switch and maybe it is the problem. I have a TP-Link TL-SG108 which has something called IGMP Snooping which apparently cannot be turned off:

Do not use TP-link TL-SG105/SG108 switches http://www.tp-link.us/products/details/?model=TL-SG105 these models support IGMP snooping! This function can not be disabled in these models.

and

IGMP snooping allows a switch to only forward multicast traffic to the links that have solicited them. Essentially, IGMP snooping is a layer 2 optimization for the layer 3 IGMP. IGMP snooping takes place internally on switches and is not a protocol feature. Snooping is therefore especially useful for bandwidth-intensive IP multicast applications such as IPTV.

I've never heard of IGMP snooping before. Could this be the problem?

On Thursday, August 18, 2016 at 12:18:13 AM UTC-6, Christoph Engelbert wrote:

Dean Schulze

unread,

Aug 22, 2016, 11:00:23 PM8/22/16

to Hazelcast

I found the problem. One of the node computers had both wired and wireless connections active. Once I shutdown the wireless connection both nodes would join the cluster. (Both nodes has wired and wireless connections active. It only takes one with both network connections active to cause Hazelcast to fail.)

I know very little about networking. Why would having two network connections cause this problem? Everything else worked.

Reply all

Reply to author

Forward