"No available connection to address" exception – with the

126 views

Skip to first unread message

Jean Luc

unread,

Jan 12, 2017, 8:25:24 PM1/12/17

to haze...@googlegroups.com

Hello,

We've experienced the following error in a production environment (Hazelcast 3.7, deployed on physical machines). One JVM using the client API failed to connected to a cluster node running in another JVM running on the same machine – so it wasn't a network problem. It is not known that triggered the exception so we are trying to understand. Could it be related to https://github.com/hazelcast/hazelcast/issues/9419 or https://github.com/hazelcast/hazelcast/issues/6872?

The exception occurred during a get() on an IMap. Client configuration is first, then the exceptions.

Any insight is very appreciated.

Thank you,

<hz:client id="hazelcastClientInstance" lazy-init="true">
   <hz:group name="${hazelcast.group.name}" password="${hazelcast.group.password}" />
   <hz:network 
         connection-attempt-limit="3"
         connection-attempt-period="3000" 
         connection-timeout="1000"
         redo-operation="true" 
         smart-routing="true">
      <hz:member>localhost:${hazelcast.port}</hz:member>
   </hz:network>
</hz:client>

The first exception:

java.io.IOException: No available connection to address [REDACTED]:5702

com.hazelcast.core.HazelcastException: java.io.IOException: No available connection to address [REDACTED]:5702

at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:73) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:63) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:52) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:83) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:155) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:147) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.proxy.ClientMapProxy.getInternal(ClientMapProxy.java:245) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:240) ~[hazelcast-all-3.7.jar!/:3.7]

at REDACTED(Redacted.java:66) ~[REDACTED.jar!/:0.0.1-SNAPSHOT]

at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at sun.reflect.GeneratedMethodAccessor1909.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at com.sun.proxy.$Proxy102.process(Unknown Source) [na:na]

at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at com.sun.proxy.$Proxy112.process(Unknown Source) [na:na]

at org.springframework.batch.item.support.CompositeItemProcessor.processItem(CompositeItemProcessor.java:61) [spring-batch-infrastructure-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.item.support.CompositeItemProcessor.process(CompositeItemProcessor.java:50) [spring-batch-infrastructure-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.item.SimpleChunkProcessor.doProcess(SimpleChunkProcessor.java:126) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.item.SimpleChunkProcessor.transform(SimpleChunkProcessor.java:293) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:192) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) [spring-tx-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]

at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374) [spring-batch-infrastructure-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) [spring-batch-infrastructure-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144) [spring-batch-infrastructure-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:93) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:90) [spring-batch-core-3.0.5.RELEASE.jar!/:3.0.5.RELEASE]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

Caused by: java.io.IOException: No available connection to address [REDACTED]:5702

at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.getConnection(ClientSmartInvocationServiceImpl.java:77) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnPartitionOwner(ClientSmartInvocationServiceImpl.java:46) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:142) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:122) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientInvocation.run(ClientInvocation.java:153) ~[hazelcast-all-3.7.jar!/:3.7]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_75]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-all-3.7.jar!/:3.7]

at ------ submitted from ------.(Unknown Source) ~[na:na]

at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveAndThrow(ClientInvocationFuture.java:74) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveAndThrow(ClientInvocationFuture.java:30) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:158) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:153) ~[hazelcast-all-3.7.jar!/:3.7]

... 62 common frames omitted

Followed by ~500 instances of the following exception (the number might be related to the data)

com.hazelcast.client.HazelcastClientNotActiveException: Not able to setup owner connection!

at com.hazelcast.client.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:170) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:127) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:152) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:147) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.proxy.ClientMapProxy.getInternal(ClientMapProxy.java:245) ~[hazelcast-all-3.7.jar!/:3.7]

at com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:240) ~[hazelcast-all-3.7.jar!/:3.7]

at REDACTED.process(Redacted.java:25) ~[REDACTED.jar!/:0.0.1-SNAPSHOT]