SEVERE exception when adding a node to a 2 nodes cluster under heavy load
35 views
Skip to first unread message
Patrizio Munzi
Apr 23, 2014, 12:21:02 PM4/23/14
to haze...@googlegroups.com
Hello guys,
we're doing HZ cluster failover tests and we're having very worrying exception when adding a node to a cluster where a node is already receiving high load.
Here are the steps we're doing:
1. start node 1
2. start heavy load on node 1
3. start node 2
When the node 2 is started the following exceptions appears in node 1:
=========================
23-Apr-2014 15:13:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages packet=Packet [RESPONSE] name=c:com.hypertv.gaming.model.HTFacebookUserBean, connection=Connection [/46.105.110.104:41190 -> Address[46.105.110.104]:5701] live=true, client=false, type=MEMBER,blockId=112, keySize=153, valueSize=0 client=false obj=null callId=14827297
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:13:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
at java.lang.Thread.run(Thread.java:662)
[...]
23-Apr-2014 15:28:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages packet=Packet [RESPONSE] name=c:com.hypertv.gaming.model.HTFacebookUserBean, connection=Connection [/46.105.110.104:41190 -> Address[46.105.110.104]:5701] live=true, client=false, type=MEMBER,blockId=118, keySize=153, valueSize=0 client=false obj=null callId=16687405
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:28:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
at java.lang.Thread.run(Thread.java:662)
[...]
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages processable=com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader@7b1dc9ce
java.lang.NullPointerException
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] null
java.lang.NullPointerException
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.impl.ConcurrentMapManager
WARNING: [5.135.142.61]:5701 [htgamingdatagrid] Store thrown exception for CONCURRENT_MAP_GET
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at com.hazelcast.impl.ConcurrentMapManager.getMap(ConcurrentMapManager.java:2502)
at com.hazelcast.impl.ConcurrentMapManager.setIndexValues(ConcurrentMapManager.java:1427)
at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.doMapStoreOperation(ConcurrentMapManager.java:3828)
at com.hazelcast.impl.ConcurrentMapManager$AbstractMapStoreOperation.run(ConcurrentMapManager.java:3901)
at com.hazelcast.impl.executor.ParallelExecutorService$ParallelExecutorImpl$ExecutionSegment.run(ParallelExecutorService.java:242)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:45)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages processable=com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader@2d7f6fb9
java.lang.NullPointerException
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] null
java.lang.NullPointerException
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
at java.lang.Thread.run(Thread.java:662)
[...]
=========================
No exceptions in node 1
Any idea???
Peter Veentjer
Apr 23, 2014, 2:06:17 PM4/23/14
to haze...@googlegroups.com
Which version of Hazelcast are you using?
And can you make a ticket for it? This makes it easier to track; emails often are forgotten.
And can you make a ticket for it? This makes it easier to track; emails often are forgotten.
Patrizio Munzi
Apr 24, 2014, 4:29:59 AM4/24/14
to haze...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages