INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5700 is STARTING
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.56:5701, timeout: 0, bind-any: true
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5700, timeout: 0, bind-any: true
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.56:5702, timeout: 0, bind-any: true
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.56:5701. Reason: SocketException[Connection refused to address /10.0.0.56:5701]
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5701, timeout: 0, bind-any: true
Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5701 is added to the blacklist.
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.56:5702. Reason: SocketException[Connection refused to address /10.0.0.56:5702]
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5702, timeout: 0, bind-any: true
Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5702 is added to the blacklist.
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.55:5701. Reason: SocketException[Connection refused to address /10.0.0.55:5701]
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.55:5702. Reason: SocketException[Connection refused to address /10.0.0.55:5702]
Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.55]:5701 is added to the blacklist.
Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.55]:5702 is added to the blacklist.
Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Established socket connection between /10.0.0.56:33009
Is it default behaviour to see "Connection refused" out of the box? What's Hazelcast attempting at this point.
Now, when our application actually attempts to use a hazelcast feature, we get the following continuous stream of errors:
WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}
Jul 16, 2016 6:08:30 PM com.hazelcast.spi.OperationService
WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}
Jul 16, 2016 6:08:31 PM com.hazelcast.spi.OperationService
WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}
Jul 16, 2016 6:08:32 PM com.hazelcast.spi.OperationService
On the other node, which was started first, in the catalina.out, once the app on node2 wakes up, we start getting:
Jul 16, 2016 6:09:01 PM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor
SEVERE: [10.0.0.55]:5700 [tomcat1] [3.5.4] Failed to process packet: Packet{header=17, isResponse=false, isOperation=true, isEvent=false, partitionId=-1, conn=Connection [/10.0.0.55:5700 -> /10.0.0.56:33009], endpoint=Address[10.0.0.56]:5700, live=true, type=MEMBER} on hz._hzInstance_1_tomcat1.generic-operation.thread-3
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:380)
at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:331)
at com.hazelcast.nio.serialization.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:489)
at com.hazelcast.cluster.impl.ConfigCheck.readData(ConfigCheck.java:215)
at com.hazelcast.cluster.impl.JoinMessage.readData(JoinMessage.java:80)
at com.hazelcast.cluster.impl.operations.MasterDiscoveryOperation.readInternal(MasterDiscoveryOperation.java:46)
at com.hazelcast.spi.Operation.readData(Operation.java:491)
at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:111)
at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:39)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)
at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:276)
at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:200)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:300)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
Caused by: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at java.lang.Throwable.initCause(Throwable.java:457)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1316)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1196)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:125)
at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:98)
at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:113)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:201)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)
at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:325)
... 15 more
Caused by: java.lang.ClassNotFoundException
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
... 28 more
Config deviating from default is as follows:
<network>
<port auto-increment="true" port-count="20">5700</port>
<outbound-ports>
<ports>33000-33009</ports>
</outbound-ports>
<join>
<multicast enabled="false" />
<tcp-ip enabled="true">
<member>10.0.0.55</member>
<member>10.0.0.56</member>
</tcp-ip>
</join>
<ssl enabled="false"/>
<socket-interceptor enabled="false"/>
<symmetric-encryption enabled="false">
<algorithm>PBEWithMD5AndDES</algorithm>
<salt>***</salt>
<password>***</password>
<iteration-count>19</iteration-count>
</symmetric-encryption>
</network>
So, quite a lot of errors.....if someone has seen this before...much appreciated....
Thanks for your response.You were right. The version was out of date. Simply upgrading with no other change solved the problem.Neil Emerick021 790 9910