MemberLeftException on getQueue

58 views
Skip to first unread message

Abdülhakim Ünlü

unread,
Oct 20, 2016, 12:37:03 AM10/20/16
to Hazelcast
Hi,

We are using hazelcast 3.6.3.

We have 6 hazelcast server nodes. While a client was initializing, one of the server nodes was killed (other 5 nodes were up and running). The client got the following exception:

com.hazelcast.core.HazelcastException: com.hazelcast.core.MemberLeftException: Member [10.133.203.31]:5701 has left cluster!
        at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:65)
        at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:52)
        at com.hazelcast.client.impl.protocol.task.AbstractMessageTask.sendClientMessage(AbstractMessageTask.java:217)
        at com.hazelcast.client.impl.protocol.task.AbstractMessageTask.handleProcessingFailure(AbstractMessageTask.java:159)
        at com.hazelcast.client.impl.protocol.task.AbstractInvocationMessageTask.onFailure(AbstractInvocationMessageTask.java:56)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture$1.run(InvocationFuture.java:129)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
        at ------ End remote and begin local stack-trace ------.(Unknown Source)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:128)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:95)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:74)
        at com.hazelcast.client.spi.ProxyManager.initialize(ProxyManager.java:301)
        at com.hazelcast.client.spi.ProxyManager.initializeWithRetry(ProxyManager.java:257)
        at com.hazelcast.client.spi.ProxyManager.getOrCreateProxy(ProxyManager.java:238)
        at com.hazelcast.client.impl.HazelcastClientInstanceImpl.getDistributedObject(HazelcastClientInstanceImpl.java:511)
        at com.hazelcast.client.impl.HazelcastClientInstanceImpl.getQueue(HazelcastClientInstanceImpl.java:339)
        at com.hazelcast.client.impl.HazelcastClientProxy.getQueue(HazelcastClientProxy.java:88)
        
Caused by: com.hazelcast.core.MemberLeftException
        at com.hazelcast.client.impl.protocol.ClientExceptionFactory$35.createException(ClientExceptionFactory.java:306)
        at com.hazelcast.client.impl.protocol.ClientExceptionFactory.createException(ClientExceptionFactory.java:613)
        at com.hazelcast.client.impl.protocol.ClientExceptionFactory.createException(ClientExceptionFactory.java:580)
        at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport$ResponseThread.handleClientMessage(ClientInvocationServiceSupport.java:313)
        at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport$ResponseThread.process(ClientInvocationServiceSupport.java:295)
        at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport$ResponseThread.doRun(ClientInvocationServiceSupport.java:288)
        at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport$ResponseThread.run(ClientInvocationServiceSupport.java:265)

I was expecting getQueue operation to be successful with no exceptions, because other nodes were up. 

Why did hazelcast throw an exception in this state ? What am I supposed to do after catching exception ?

thanks,


Abdülhakim Ünlü

unread,
Oct 25, 2016, 7:24:40 AM10/25/16
to Hazelcast
It seems no one got this error before me ?

20 Ekim 2016 Perşembe 07:37:03 UTC+3 tarihinde Abdülhakim Ünlü yazdı:

te...@hazelcast.com

unread,
Oct 25, 2016, 4:44:45 PM10/25/16
to Hazelcast
The client connects to one to many servers based on the resource(s) being requested. Queue's are ordered and therefor they are implemented on a single node with backup(s) residing on other nodes. When the primary node fails the backup node becomes primary and the client can continue.

Check out the documentation on handling-retry-able-operation-failure
You can set a timeout for retrying the operations sent to a member. This can be provided by using the property hazelcast.client.invocation.timeout.seconds in ClientProperties. The client will retry an operation within this given period, of course, if it is a read-only operation or you enabled the redoOperation as stated in the above paragraph. This timeout value is important when there is a failure resulted by either of the following causes:
  • Member throws an exception.
  • Connection between the client and member is closed.
  • Client's heartbeat requests are timed out.
Hope this helps,

Abdülhakim Ünlü

unread,
Nov 1, 2016, 4:10:06 AM11/1/16
to Hazelcast
Setting redoOperation to TRUE does not change the behaviour. 

Also, I get a similar error on queue.take when primary nodes is shutdown. 

com.hazelcast.spi.exception.TargetDisconnectedException: Target[Address[127.0.0.1]:5701] disconnected.
        at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport$CleanResourcesTask.run(ClientInvocationServiceSupport.java:211) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[?:1.7.0_80]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) ~[?:1.7.0_80]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) ~[?:1.7.0_80]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_80]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_80]
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.6.3.jar:3.6.3]
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.6.3.jar:3.6.3]
        at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[?:?]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:128) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:95) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:74) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:37) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.spi.ClientProxy.invokeOnPartitionInterruptibly(ClientProxy.java:135) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.proxy.PartitionSpecificClientProxy.invokeOnPartitionInterruptibly(PartitionSpecificClientProxy.java:43) ~[hazelcast-client-3.6.3.jar:3.6.3]
        at com.hazelcast.client.proxy.ClientQueueProxy.take(ClientQueueProxy.java:192) ~[hazelcast-client-3.6.3.jar:3.6.3]




25 Ekim 2016 Salı 23:44:45 UTC+3 tarihinde te...@hazelcast.com yazdı:

Jaromir Hamala

unread,
Mar 10, 2017, 2:10:19 AM3/10/17
to Hazelcast
Hi,

can you try it with Hazelcast 3.8? 

Cheers,
Jaromir
Reply all
Reply to author
Forward
0 new messages