Connecting to remote Hazelcast server (embedded) from client is very slow

[Sutanu Dalui]

We are using an embedded Hazelcast instance server cluster for incoming data processing. We have clients which produce ingestion data to the cluster.

Processing happens fine, but an observation is, that the client connection takes a long time (~1 min) to connect to the cluster. Specifically, HazelcastClient.newHazelcastClient(clientConfig);

Have used,
setConnectionAttemptLimit(1);
setConnectionTimeout(30000);

At times we are also getting a timeout exception,
======
Caused by: java.lang.IllegalStateException: Unable to connect to any address in the config! The following addresses were tried:[/***:5703, /***:5701, /***:5702]
at com.hazelcast.client.spi.impl.ClusterListenerThread.connectToOne(ClusterListenerThread.java:286)
at com.hazelcast.client.spi.impl.ClusterListenerThread.run(ClusterListenerThread.java:89)
Caused by: com.hazelcast.core.HazelcastException: java.util.concurrent.TimeoutException
at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:74)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$OwnerConnectionFuture.createNew(ClientConnectionManagerImpl.java:616)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$OwnerConnectionFuture.access$300(ClientConnectionManagerImpl.java:565)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.ownerConnection(ClientConnectionManagerImpl.java:242)
at com.hazelcast.client.spi.impl.ClusterListenerThread.connect(ClusterListenerThread.java:300)
at com.hazelcast.client.spi.impl.ClusterListenerThread.connectToOne(ClusterListenerThread.java:264)
... 1 more
======

The remote instance is running on port 5701, and I can telnet it successfully.

If it can be suggested how I can try to identify the root cause, that will also suffice. The client-server connection works fine in local, (on same machine), however. Can this be a hardware/network issue? If yes, any pointers how do I narrow down my RCA?

Using Hz 3.4.1

[Peter Veentjer]

What happens if you replace the client by a regular member. Just to see how long it takes for 2 members take to connect over the same environment your client/member are trying to connect.

Normally connection should be quite fast.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/43104779-ca5e-4221-93f0-eb7a9aecc16d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sutanu Dalui]

Hi Peter,

Thanks for your response.

Replacing the connection as regular member it takes lesser time (~10-12 sec). Which I believe is high as well I suppose, still?

Also, for client connection, I see multiple messages on the server side,
=====
Mar 19, 2015 3:55:46 PM com.hazelcast.nio.tcp.SocketAcceptor
INFO: [169.144.57.7]:5701 [NBA-DATAGRID] [3.4.1] Accepting socket connection from /***:48541
Mar 19, 2015 3:55:51 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [169.144.57.7]:5701 [NBA-DATAGRID] [3.4.1] Established socket connection between /***:5701 and ***/***:48541
Mar 19, 2015 3:55:51 PM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [169.144.57.7]:5701 [NBA-DATAGRID] [3.4.1] Connection ***/***:48541] lost. Reason: java.io.EOFException[Could not read protocol type!]
=====

and finally,
=====
INFO: [169.144.57.7]:5701 [NBA-DATAGRID] [3.4.1] Received auth from Connection [/***:5701 -> ***/***:48586], endpoint=null, live=true, type=JAVA_CLIENT, successfully authenticated
=====

It seems the handshake is being retried a number of times?

[Sutanu Dalui]

If I simply try to open a java client Socket, it works fine. Also, am able to Telnet without any issues as well. I tried these out to eliminate any network related issues in the environment.