Hazelcast client reconnection

[Leon Zakaim]

Hi,

We are testing scenarios when one of the HZ members in a cluster goes down.

The expectation is that the client should recognize this and switch automatically to another instance. The client indeed does this but after delay of 5 - 10 seconds. All the operations with the HZ maps are failing during this period of time.

Is there a way to block the operations till new instance is selected for the connection to the cluster?

Details: 

- client is .NET client v.3.12.x

- there are three members in the cluster, all in the same LAN

- we tried creating the instance dynamically

[Stephan Gay]

Hello,

The Hazelcast .NET client, be it v3 or the newer v4, constantly manages the connections to members. As soon as a connection is flagged as "down", it should stop trying to use it, and route requests to other members. It is normal that some map operations (those that go to the member and try to use the connection) fail for a short amount of time.

There is no way to "block the operations 'till new instance is selected" because, if we keep trying to do operations on a dead connection, it means we still don't know that the connection is dead. In other words, if we knew when to block, we just would not need to block and could re-route immediately to another member. 10 seconds seem quite a long delay for the client to realize that a member connection is down. If the member (software) shuts down, detection should happen faster. OTOH... if a LAN cable is unplugged, it may take time for the OS network layer to report the connection as down.

However - in v4 when an operation fails we can retry it - I need to check whether we would be clever enough as to retry it on another connection. And... operations which change the state of the cluster (map.put) are not retried by default. Are you worried about some operations in particular, or all of them in general?

I'll try to think of a way to deal with this situation.

Stephan

Hazelcast .NET Client lead dev

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Leon Zakaim]

Thank you for your  quick reply, can we continue the discussion with you over chat/mail/phone/ etc.? 

[Burak Çelebi]

Hi,

I suggest joining our community slack :)

https://slack.hazelcast.com/

We have different channels there alongside the C# client one.

Hope it helps.