Java Client fails to put to map after reconnecting

[Chad]

I'm seeing a problem in Hazelcast 1.9.4.6 with the HazelcastClient.
Here is the scenario:

1. Cluster up with only 1 node.
2. Client connects and puts a value to a distributed map.
3. Cluster node goes down--there is no longer a cluster.
4. HazelcastClient automatically enters connection retry.
5. Cluster node comes back up (same host:port from before) and creates
a new cluster.
6. HazelcastClient reconnects.
7. The client attempts to put a value into the same distributed map.
I don't know if it matters, but this put is happening within the
client's LifecycleListener.stateChanged() when the LifecycleEvent is
CLIENT_CONNECTION_OPENED.

This results in the following logs, over and over. I can reproduce
this every time, and it seems to never come out of this failure loop.

Feb 09, 2012 12:30:20 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [3] operation=CONCURRENT_MAP_PUT
in 5 seconds.
Feb 09, 2012 12:30:25 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [3] operation=CONCURRENT_MAP_PUT
in 10 seconds.
.
.
.
Feb 09, 2012 1:35:36 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [3] operation=CONCURRENT_MAP_PUT
in 3920 seconds.
Feb 09, 2012 1:35:41 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [3] operation=CONCURRENT_MAP_PUT
in 3925 seconds.

[Chad Gatesman]

I tried this with 1.9.4.8, and the problem still exists.

[Fuad Malikov]

Can you post the code?

Regards,

Fuad Malikov

Sent from Mobile

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Chad Gatesman]

I'll put together a simple sample.

Interesting, Google Groups doesn't have a way to upload a file on
their web frontend??

On Feb 10, 1:00 am, Fuad Malikov <f...@hazelcast.com> wrote:
> Can you post the code?
>
> Regards,
>
> Fuad Malikov
> Sent from Mobile
>

> On Feb 10, 2012, at 2:43 AM, Chad Gatesman <cgates...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I tried this with 1.9.4.8, and the problem still exists.
>

[Fuad Malikov]

You can create an issue and post the code there: http://code.google.com/p/hazelcast/issues/list

-fuad

[Chad Gatesman]

Attached is a sample that reproduces the problem.  Below is the output.  I used com.hazelcast.examples.StartServer for a test cluster.  I killed then restarted StartServer right where you see the CLIENT_CONNECTION_LOST logs.

Feb 10, 2012 9:05:16 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTING
Feb 10, 2012 9:05:16 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Feb 10, 2012 9:05:16 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENED
Feb 10, 2012 9:05:16 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTED
Feb 10, 2012 9:05:23 AM com.hazelcast.client.ConnectionManager
WARNING: Connection to Connection [0] [localhost/127.0.0.1:5701 -> 127.0.0.1:5701] is lost
Feb 10, 2012 9:05:23 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_LOST
Cluster Client: CLIENT_CONNECTION_LOST
Feb 10, 2012 9:05:24 AM com.hazelcast.client.ConnectionManager
INFO: Unable to get alive cluster connection, try in 3,999 ms later, attempt 1 of 5.
Feb 10, 2012 9:05:28 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Cluster Client: CLIENT_CONNECTION_OPENING
Feb 10, 2012 9:05:28 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENED
Cluster Client: CLIENT_CONNECTION_OPENED
Feb 10, 2012 9:05:38 AM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [2] operation=CONCURRENT_MAP_PUT in 5 seconds.
Feb 10, 2012 9:05:43 AM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [2] operation=CONCURRENT_MAP_PUT in 10 seconds.

[Chad Gatesman]

Fuad,

Did you need me to create an issue for this problem?

-Chad

>  ClientTest.java
> 1KViewDownload

[Fuad Malikov]

Hi Chad,

I have fixed the issue in trunk. It will be in 2.0. Actually the bug occurred because you call Hazelcast operation on receiving event. It is Hazelcast thread calling you and by doing a put operation you are blocking it. A good practice is receive the notification and execute whatever you do on a separate thread. That is do not block Hazelcast threads calling you.  

Even though there was a bug and I fixed that you wouldn't hit it, if you would call map.put in another thread of your own. 

-fuad

[Chad Gatesman]

Excellent!  I thought this may have been the case.  I will keep this in mind for other callbacks from Hazelcast.

-Chad