EC2 Auto discovery not running

Manoj Khotele

unread,

Feb 20, 2016, 10:45:07 AM2/20/16

to Hazelcast

I am running one after another 2 hazelcast instances in AWS EC2, with same hazelcast.xml file.

The second node does not join the first one.

I don't observe any delay before both of these nodes form their own cluster.

What's wrong with following hazelcast.xml?

<hazelcast xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
http://www.hazelcast.com/schema/config/hazelcast-config-3.6.xsd"
xmlns="http://www.hazelcast.com/schema/config">
<group>
<name>hz-test-group</name>
<password>hz-test-pwd</password>
</group>
<properties>
<property name="hazelcast.discovery.enabled">true</property>
</properties>
<network>
<join>
<multicast enabled="false"/>
<tcp-ip connection-timeout-seconds="30" enabled="false"/>
<aws enabled="true">
<access-key>my-access-key</access-key>
<secret-key>my-secret-key</secret-key>
<region>ap-southeast-1a</region>
<host-header>ec2.amazonaws.com</host-header>
<security-group-name>hazelcast-sg</security-group-name>
<tag-key>type</tag-key>
<tag-value>hz-nodes</tag-value>
</aws>
</join>
</network>
</hazelcast>

The logs are as follows:

########################################
# RUN_JAVA=
# JAVA_OPTS=
# starting now....
########################################
Process id for hazelcast instance is written to location: /opt/hazelcast/hazelcast_instance.pid
Feb 20, 2016 3:36:36 PM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Feb 20, 2016 3:36:37 PM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [hz-test-group] [3.6] Prefer IPv4 stack is true.
Feb 20, 2016 3:36:37 PM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [hz-test-group] [3.6] Picked Address[172.17.0.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Feb 20, 2016 3:36:37 PM com.hazelcast.spi.OperationService
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Backpressure is disabled
Feb 20, 2016 3:36:37 PM com.hazelcast.spi.impl.operationexecutor.classic.Classic OperationExecutor
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Starting with 2 generic operation threads and 2 partition operation threads.
Feb 20, 2016 3:36:37 PM com.hazelcast.system
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Hazelcast 3.6 (20160125 - 13fea63) starting at Address[172.17.0.2]:5701
Feb 20, 2016 3:36:37 PM com.hazelcast.system
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
Feb 20, 2016 3:36:37 PM com.hazelcast.system
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Configured Hazelcast Serialization version : 1
Feb 20, 2016 3:36:37 PM com.hazelcast.instance.Node
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Activating Discovery SPI Joiner
Feb 20, 2016 3:36:37 PM com.hazelcast.core.LifecycleService
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Address[172.17.0.2]:5701 is STARTING
Feb 20, 2016 3:36:37 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreading Model
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
Feb 20, 2016 3:36:37 PM com.hazelcast.cluster.impl.DiscoveryJoiner
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6]

Members [1] {
Member [172.17.0.2]:5701 this
}

Feb 20, 2016 3:36:38 PM com.hazelcast.core.LifecycleService
INFO: [172.17.0.2]:5701 [hz-test-group] [3.6] Address[172.17.0.2]:5701 is STARTED

Best Regards,

Manoj

Peter Veentjer

unread,

Feb 21, 2016, 8:28:32 AM2/21/16

to haze...@googlegroups.com

Have you configured the security group correctly? If not, then probably you have a firewall issue. See

the text below how to detect if there is a problem.

.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/f328b938-45c4-4312-844b-2db23bb39c00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Manoj Khotele

unread,

Feb 21, 2016, 9:57:17 AM2/21/16

to Hazelcast

I verified that it's not a firewall issue.

On following the steps as suggested, this is what I got on second machine:

[ec2-user@ip-172-31-24-188 ~]$ iperf -c 54.254.198.177 -d -p 5701
------------------------------------------------------------
Server listening on TCP port 5701
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 54.254.198.177, TCP port 5701
TCP window size: 780 KByte (default)
------------------------------------------------------------
[ 3] local 172.31.24.188 port 33821 connected with 54.254.198.177 port 5701
[ 5] local 172.31.24.188 port 5701 connected with 54.254.198.177 port 54623
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 484 MBytes 405 Mbits/sec
[ 5] 0.0-10.2 sec 140 MBytes 115 Mbits/sec

Thanks Peter, for suggesting the way to check that firewall is not the cause.

Then what else could stop these machines from discovering each other.

Peter Veentjer

unread,

Feb 21, 2016, 4:03:57 PM2/21/16

to haze...@googlegroups.com

Can you switch logging to finest.

There should be very detailed logging regarding the discovery process that should give you a clear understanding from retrieving the instances from ec2 and why nodes are being rejected.

Another quick hack to see if you can get it up and running is temporarily switch to TCP/IP-Discovery and add the (private) ip addresses to the config, and see if the members connect correctly.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/c2d544d5-64d3-4932-81ab-522e7bc6c202%40googlegroups.com.

Manoj Khotele

unread,

Feb 22, 2016, 9:21:17 AM2/22/16

to Hazelcast

Thanks Peter for quick response.

This may look very silly, ut how can I switch logging to finest?

I am following https://hub.docker.com/r/hazelcast/hazelcast/ to run hazelcast via docker image. How can I extend the base docker image to configure logging?

I tried adding this propertyin hazelcast.xml - <property name="hazelcast.logging.type">log4j2</property> and including log4j.xml. But this gives exception "java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager". I can understand this error.

My question is - is there a quick way to keep all the default settings, but just switch logging level?

By the way, I haven't tried second hack suggested by you. I am gonna try that.

Manoj Khotele

unread,

Feb 23, 2016, 5:52:17 AM2/23/16

to Hazelcast

I tried forming a cluster using tcp.

In this case also, nodes are not joining.

Te logs are:

WARNING: [172.31.21.100]:5701 [dev] [3.6] hz._hzInstance_1_dev.IO.thread-in-1 Closing socket to endpoint Address[54.254.203.176]:5701, Cause:java.io.EOFException: Remote socket closed!

Feb 23, 2016 10:29:19 AM com.hazelcast.nio.tcp.InitConnectionTask

INFO: [172.31.21.100]:5701 [dev] [3.6] Connecting to /54.254.203.176:5701, timeout: 0, bind-any: true

Feb 23, 2016 10:29:19 AM com.hazelcast.nio.tcp.TcpIpConnectionManager

INFO: [172.31.21.100]:5701 [dev] [3.6] Established socket connection between /172.31.21.100:51072 and /54.254.203.176:5701

Feb 23, 2016 10:29:19 AM com.hazelcast.nio.tcp.TcpIpConnection

INFO: [172.31.21.100]:5701 [dev] [3.6] Connection [Address[54.254.203.176]:5701] lost. Reason: java.io.EOFException[Remote socket closed!]

Feb 23, 2016 10:29:19 AM com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketReader

WARNING: [172.31.21.100]:5701 [dev] [3.6] hz._hzInstance_1_dev.IO.thread-in-2 Closing socket to endpoint Address[54.254.203.176]:5701, Cause:java.io.EOFException: Remote socket closed!