Hazelcast instance not active! exception on version 3.3.1
1,612 views
Skip to first unread message
Abdulghaffar Al-Labadi
Mar 8, 2015, 1:13:07 AM3/8/15
to haze...@googlegroups.com
Hi,
I am using hazelcast server version 3.3.1 on three machines, but one of them redundantly throw the following exception:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:295)
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:282)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:79)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:77)
at com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51)
at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:130)
at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:388)
at com.hazelcast.instance.HazelcastInstanceImpl.getExecutorService(HazelcastInstanceImpl.java:274)
at com.hazelcast.instance.HazelcastInstanceProxy.getExecutorService(HazelcastInstanceProxy.java:116)
please advice
Peter Veentjer
Mar 8, 2015, 6:39:48 AM3/8/15
to haze...@googlegroups.com
This is an indication that underlying Hazelcast Instance has shut down. So this exception isn't the real problem; the real problem is: why did your HZ instance shutdown. Do you see any exceptions preceding this exception?
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/da6e92bd-728b-4d58-a469-31a7483de6d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Abdulghaffar Al-Labadi
Mar 8, 2015, 8:46:21 AM3/8/15
to haze...@googlegroups.com
Thank you for replying on the post,
What is strange in this exception is the way it occurred. I thing it depends on number of threads trying to access hz instance concurrently.
Peter Veentjer
Mar 9, 2015, 12:36:45 AM3/9/15
to haze...@googlegroups.com
Perhaps there is something else going on.
Can you first check if you see any exceptions in front of your HazelcastInstanceNotActiveException ?
If there is none, perhaps there is something else going on.
Can you first check if you see any exceptions in front of your HazelcastInstanceNotActiveException ?
If there is none, perhaps there is something else going on.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/fbad365c-e01a-45ad-a7a6-0a80bfa1f1ee%40googlegroups.com.
Abdulghaffar Al-Labadi
Mar 9, 2015, 1:06:04 AM3/9/15
to haze...@googlegroups.com
Unfortunately there are nothing else going on.
any way I will reproduce it and back to you.
Thank you again Peter.
--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/tVrWrF74wXI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGuAWdBDqRmFg66qWv5kXvJ5TSY22uhqf_Rc_Z%3DNq0qMZ%3DKp-w%40mail.gmail.com.
Best regards,
Abdulghaffar Ellabadi
Senior Software Developer
Peter Veentjer
Mar 9, 2015, 12:08:23 PM3/9/15
to haze...@googlegroups.com
If there is no previous exception then the problem might be a bit more complicated. How easy is it to reproduce it? Is it doable in a unit test?
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEOg9320Dz2cX7YTeeC8iwms88xpkVNObr2BGeUjs2Wdh4Gkng%40mail.gmail.com.
Abdulghaffar Al-Labadi
Mar 9, 2015, 1:26:49 PM3/9/15
to haze...@googlegroups.com
It is reproducible on one of three machines and thats make the exception more xomplicated for me.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGuAWdB8HzpvukdPhu0uwimTLK_wde3Oc%2Bf8%3DwwHEF26n%2B4P-A%40mail.gmail.com.
Abdulghaffar Al-Labadi
Apr 12, 2015, 6:21:03 AM4/12/15
to haze...@googlegroups.com
Hi,
finally I back to the server that produce the exceptions for my deployment I and find the following exceptions,
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) Exception in thread "ActiveMQ Session Task-121" java.lang.OutOfMemoryError: unable to create new native thread11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.start0(Native Method)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.start(Thread.java:713)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.run(Thread.java:744)11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) Exception in thread "ActiveMQ Transport: tcp:///10.10.8.104:61616@23633" java.lang.OutOfMemoryError: unable to create new native thread11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at java.lang.Thread.start0(Native Method)11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at java.lang.Thread.start(Thread.java:713)11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)11:53:12,120 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:152)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:145)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.tcp.TcpTransport.doStop(TcpTransport.java:538)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.tcp.TcpTransport.stop(TcpTransport.java:582)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.AbstractInactivityMonitor.stop(AbstractInactivityMonitor.java:145)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.WireFormatNegotiator.stop(WireFormatNegotiator.java:91)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.util.ServiceSupport.dispose(ServiceSupport.java:43)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.failover.FailoverTransport.disposeTransport(FailoverTransport.java:240)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:253)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.failover.FailoverTransport$3.onException(FailoverTransport.java:215)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:205)11:53:12,121 ERROR [stderr] (ActiveMQ Transport: tcp:///10.10.8.104:61616@23633) at java.lang.Thread.run(Thread.java:744)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) Exception in thread "ActiveMQ Session Task-11" java.lang.OutOfMemoryError: unable to create new native thread11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.lang.Thread.start0(Native Method)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.lang.Thread.start(Thread.java:713)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)11:54:53,062 ERROR [stderr] (ActiveMQ Session Task-11) at java.lang.Thread.run(Thread.java:744)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) Exception in thread "ActiveMQ Session Task-9" java.lang.OutOfMemoryError: unable to create new native thread11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.lang.Thread.start0(Native Method)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.lang.Thread.start(Thread.java:713)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)11:55:01,066 ERROR [stderr] (ActiveMQ Session Task-9) at java.lang.Thread.run(Thread.java:744)11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) Exception in thread "ActiveMQ Session Task-12" java.lang.OutOfMemoryError: unable to create new native thread11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) at java.lang.Thread.start0(Native Method)11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) at java.lang.Thread.start(Thread.java:713)11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)11:55:04,175 ERROR [stderr] (ActiveMQ Session Task-12) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)11:55:04,176 ERROR [stderr] (ActiveMQ Session Task-12) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)11:55:04,176 ERROR [stderr] (ActiveMQ Session Task-12) at java.lang.Thread.run(Thread.java:744)11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) Exception in thread "hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3" com.hazelcast.nio.serialization.HazelcastSerializationException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:354)11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:310)11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:264)11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.spi.impl.NormalResponse.writeData(NormalResponse.java:78)11:55:54,931 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:140)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:39)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:52)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:219)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:200)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:180)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.spi.impl.BasicOperationService.send(BasicOperationService.java:332)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:119)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.sendResponse(DistributedExecutorService.java:213)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:198)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)11:55:54,932 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at java.lang.Thread.run(Thread.java:744)11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:300)11:55:54,933 ERROR [stderr] (hz._hzInstance_1_MATRIX12.CACHE.TEST.DEV_WS.cached.thread-3) ... 18 more
It seems that i have memory management on the server while it has more 18G of memory reserved for jboss server and the data used for this server is less that 1G.
Abdulghaffar Al-Labadi
Apr 22, 2015, 7:40:18 AM4/22/15
to haze...@googlegroups.com
Hi Peter,
I run the system on three node and add log for hz instances and got the following exception on the node:
2015 04 22 14:11:06:804 - INFO - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] Connection [Address[10.10.8.105]:5193] lost. Reason: Socket explicitly closed
2015 04 22 14:11:06:808 - INFO - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] Connection [Address[10.10.8.104]:5193] lost. Reason: Socket explicitly closed
2015 04 22 14:11:06:831 - INFO - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] Shutting down Hazelcast Management Center Service
2015 04 22 14:11:06:840 - ERROR - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] null
java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at com.hazelcast.map.operation.QueryOperation.runParallel(QueryOperation.java:112)
at com.hazelcast.map.operation.QueryOperation.run(QueryOperation.java:86)
at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:699)
at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:649)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:604)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1400(BasicOperationService.java:591)
at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:543)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:433)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)
2015 04 22 14:11:06:840 - ERROR - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] null
java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at com.hazelcast.map.operation.QueryOperation.runParallel(QueryOperation.java:112)
at com.hazelcast.map.operation.QueryOperation.run(QueryOperation.java:86)
at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:699)
at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:649)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:604)
at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1400(BasicOperationService.java:591)
at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:543)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:433)
at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)
2015 04 22 14:11:06:962 - WARN - Slf4jFactory$Slf4jLogger - [10.10.8.103]:5193 [MATRIX12.CACHE.TEST.DEV_WS] [3.3.1] Could not get results
java.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:312)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:181)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:160)
at com.hazelcast.map.BasicMapContextQuerySupport.getQueryResult(BasicMapContextQuerySupport.java:329)
at com.hazelcast.map.BasicMapContextQuerySupport.addResultsOfPredicate(BasicMapContextQuerySupport.java:316)
at com.hazelcast.map.BasicMapContextQuerySupport.query(BasicMapContextQuerySupport.java:227)
at com.hazelcast.map.proxy.MapProxySupport.query(MapProxySupport.java:1070)
at com.hazelcast.map.proxy.MapProxyImpl.keySet(MapProxyImpl.java:576)
any advice
Peter Veentjer
Apr 24, 2015, 1:35:55 AM4/24/15
to haze...@googlegroups.com
I get only fragments of the problem. So it is a bit difficult to see the whole picture.
One of the exception you get is a from a previous email:11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) Exception
in thread "ActiveMQ Session Task-121" java.lang.OutOfMemoryError: unable
to create new native thread
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.start0(Native Method)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.start(Thread.java:713)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
11:53:12,119 ERROR [stderr] (ActiveMQ Session Task-121) at java.lang.Thread.run(Thread.java:744)
This is a big problem. It could very well be a file handle problem; doesn't need to be a memory problem. Make sure that enough file handles are configured.
Once the OOME happens, depending on the situation, the HazelcastInstance will shut itself down. Because it shuts itself down, all invocations are aborted. So you get a huge amount of HazelcastInstanceNotActiveExceptions.
So what is happening in front of the log fragment you send? Do you see any exceptions?
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/afebc1c5-6637-49e9-aacb-f9c6f843ce11%40googlegroups.com.
Abdulghaffar Al-Labadi
Apr 26, 2015, 7:26:14 AM4/26/15
to haze...@googlegroups.com
Hi,
I tried to reproduce the problem and finally I found the main reason, and it is system memory allocation, where the process could not create new threads because of the Operation system configuration, so what I did is increase Number of thread per process and increase number of threads per user, so every thing is working fine for now and I hope so.
Thank you for supporting me.
Reply all
Reply to author
Forward
0 new messages