Hazelcast client unable to connect to Hazelcast cluster

[Rishav Shandilya]

Hi Team,

We have a Hazelcast client running inside AWS lambda can't connect to a Hazelcast cluster deployed on some EC2 instances itself.

Getting timeout exception:

Trying to connect to [*.*.***.**]:5702 as owner member

We are setting cluster group name and a node public address in ClientConfig. Although our same client app can connect to the cluster if we run client app locally.

Could you please let me know what could be the possible reason for this?

[ihsan demir]

Can you please share your exact client config and the detailed client logs? And server side logs if available. Please also share your client and server versions.

Thanks.

[me...@hazelcast.com]

Also, the address you mentioned in the logs (Trying to connect to [*.*.***.**]:5702 as owner member), is this the private or the public address of the cluster member?

[Rishav Shandilya]

Please find below our hazelcast-client.xml:

<hazelcast-client>
        <group>
                <name>test123</name>
                # This is our cluster group name that my hazelcast client should connect to.
        </group>
        <network>
                <cluster-members>
                        <address>public ip of one of the nodes</address>
                </cluster-members>
                <outbound-ports>
                        <ports>34600</ports>
                        <ports>34700-34710</ports>
                </outbound-ports>
                <smart-routing>true</smart-routing>
                <redo-operation>true</redo-operation>
                <connection-timeout>60000</connection-timeout>
                <connection-attempt-period>3000</connection-attempt-period>
                <connection-attempt-limit>1</connection-attempt-limit>
                <socket-options>
                        <tcp-no-delay>false</tcp-no-delay>
                        <keep-alive>true</keep-alive>
                        <reuse-address>true</reuse-address>
                        <linger-seconds>3</linger-seconds>
                        <buffer-size>128</buffer-size>
                </socket-options>
                <aws enabled="false" connection-timeout-seconds="11">
                        <inside-aws>true</inside-aws>
                        <access-key>********</access-key>
                        <secret-key>************</secret-key>
                        <region>ap-south-1</region>
                        <!-- <host-header>ec2.amazonaws.com</host-header> -->
                        <security-group-name>************</security-group-name>
                        <!-- <tag-key>type</tag-key> <tag-value>hz-nodes</tag-value> -->
                </aws>
                <gcp enabled="false">
                        <zones>us-east1-b,us-east1-c</zones>
                </gcp>
                <azure enabled="false">
                        <client-id>CLIENT_ID</client-id>
                        <client-secret>CLIENT_SECRET</client-secret>
                        <tenant-id>TENANT_ID</tenant-id>
                        <subscription-id>SUB_ID</subscription-id>
                        <cluster-id>HZLCAST001</cluster-id>
                        <group-name>GROUP-NAME</group-name>
                </azure>
                <kubernetes enabled="false">
                        <namespace>MY-KUBERNETES-NAMESPACE</namespace>
                        <service-name>MY-SERVICE-NAME</service-name>
                        <service-label-name>MY-SERVICE-LABEL-NAME</service-label-name>
                        <service-label-value>MY-SERVICE-LABEL-VALUE</service-label-value>
                </kubernetes>
                <eureka enabled="false">
                        <self-registration>true</self-registration>
                        <namespace>hazelcast</namespace>
                </eureka>
                <icmp-ping enabled="false">
                        <timeout-milliseconds>2000</timeout-milliseconds>
                        <interval-milliseconds>3000</interval-milliseconds>
                        <ttl>100</ttl>
                        <max-attempts>5</max-attempts>
                        <echo-fail-fast-on-startup>false</echo-fail-fast-on-startup>
                </icmp-ping>
                <hazelcast-cloud enabled="false">
                        <discovery-token>EXAMPLE_TOKEN</discovery-token>
                </hazelcast-cloud>
        </network>
</hazelcast-client>

- We are using hazelcast 3.12.4 and hazelcast-client 3.12.4

Please find below Hazelcast client logs. We have the below logs only for now. Will share more logs if needed.

We don't have any server logs created for this issue since I think hz client is unable to connect to cluster as of now.

Feb 11, 2022 1:08:22 PM com.hazelcast.client.HazelcastClient
INFO: CLIENT_NAME [test123] [3.12.4] A non-empty group password is configured for the Hazelcast client. Starting with Hazelcast version 3.11, clients with the same group name, but with different group passwords (that do not use authentication) will be accepted to a cluster. The group password configuration will be removed completely in a future release.
Feb 11, 2022 1:08:24 PM com.hazelcast.client.spi.ClientInvocationService
INFO: CLIENT_NAME [test123] [3.12.4] Running with 2 response threads, dynamic=false
Feb 11, 2022 1:08:25 PM com.hazelcast.core.LifecycleService
INFO: CLIENT_NAME [test123] [3.12.4] HazelcastClient 3.12.4 (20191030 - eab1290) is STARTING
Feb 11, 2022 1:08:25 PM com.hazelcast.core.LifecycleService
INFO: CLIENT_NAME [test123] [3.12.4] HazelcastClient 3.12.4 (20191030 - eab1290) is STARTED
Feb 11, 2022 1:08:25 PM com.hazelcast.client.connection.nio.ClusterConnectorService
INFO: CLIENT_NAME [test123] [3.12.4] Trying to connect to cluster with name: test123
Feb 11, 2022 1:08:25 PM com.hazelcast.client.connection.nio.ClusterConnectorService
INFO: CLIENT_NAME [test123] [3.12.4] Trying to connect to [node-public-ip]:5701 as owner member

Let me know if you need more info

[me...@hazelcast.com]

I would like to try one of the following to solve this issue:

- Upgrade the server&client version to 4.2+ (ideally, to the latest version, 5.0.2) and try again. Since 4.2, Hazelcast servers and clients have a feature called external smart client discovery to make the user experience around such scenarios much better. With these versions, all you had to provide is one of the public IPs of your cluster members (as you did, in your configuration) and Hazelcast will use the correct addresses to connect.

- If you don't want to upgrade, take a look at https://github.com/hazelcast/hazelcast-aws#ec2-hazelcast-client-configuration and use this plugin on the client-side to discover cluster members. As written in the README, you should use version 2.4 for the 3.12 series. https://github.com/hazelcast/hazelcast-aws#requirements

[Rishav Shandilya]

I have added hazelcast-aws dependency and enabled aws discovery in my hazelcast-client.xml and now getting below exception:

INFO: CLIENT_NAME [test123] [3.12.4] A non-empty group password is configured for the Hazelcast client. Starting with Hazelcast version 3.11, clients with the same group name, but with different group passwords (that do not use authentication) will be accepted to a cluster. The group password configuration will be removed completely in a future release.

13:29:20,025 ERROR [main] info.HazelcastInfo     - Exception:
java.lang.IllegalStateException: Only one discovery method can be enabled at a time. cluster members given explicitly : true, aws discovery: true, gcp discovery: false, azure discovery: false, kubernetes discovery: false, eureka discovery: false, discovery spi enabled : false, hazelcast.cloud enabled : false
at com.hazelcast.client.impl.clientside.ClientDiscoveryServiceBuilder.isDiscoveryConfigurationConsistent(ClientDiscoveryServiceBuilder.java:230)
at com.hazelcast.client.impl.clientside.ClientDiscoveryServiceBuilder.createAddressTranslator(ClientDiscoveryServiceBuilder.java:169)
at com.hazelcast.client.impl.clientside.ClientDiscoveryServiceBuilder.build(ClientDiscoveryServiceBuilder.java:94)
at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.initClientDiscoveryService(HazelcastClientInstanceImpl.java:300)
at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.<init>(HazelcastClientInstanceImpl.java:243)
at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:195)
at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:132)

[Jiri Holusa]

Hi,

as the error message says, you can have only one way of discovery enabled. In your case, you have provided IP addresses in the <cluster-members> element and have <aws enabled="true"> which causes the error above.

Delete the <cluster-members> element and it should work.

BTW I have created a PR for error message enhancement, so that it's easier to understand next time: https://github.com/hazelcast/hazelcast/pull/20714

Jiri