Errors from the outset

84 перегляди
Перейти до першого непрочитаного повідомлення

nemer...@gmail.com

не прочитано,
17 лип. 2016 р., 05:54:0517.07.16
Кому: Hazelcast
We're new users to Hazelcast - currently the implementation is 3.5.4 running in tomcat

We're trying to cluster over two nodes....in the application, we're using an IMap to create a lock on certain parts of the application.

When tomcat starts up on the 2nd node, we get the following log entries:

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5700 is STARTING

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.56:5701, timeout: 0, bind-any: true

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5700, timeout: 0, bind-any: true

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.56:5702, timeout: 0, bind-any: true

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.56:5701. Reason: SocketException[Connection refused to address /10.0.0.56:5701]

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5701, timeout: 0, bind-any: true

Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5701 is added to the blacklist.

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.56:5702. Reason: SocketException[Connection refused to address /10.0.0.56:5702]

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Connecting to /10.0.0.55:5702, timeout: 0, bind-any: true

Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.56]:5702 is added to the blacklist.

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.55:5701. Reason: SocketException[Connection refused to address /10.0.0.55:5701]

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.SocketConnector

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Could not connect to: /10.0.0.55:5702. Reason: SocketException[Connection refused to address /10.0.0.55:5702]

Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.55]:5701 is added to the blacklist.

Jul 16, 2016 6:08:29 PM com.hazelcast.cluster.impl.TcpIpJoiner

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Address[10.0.0.55]:5702 is added to the blacklist.

Jul 16, 2016 6:08:29 PM com.hazelcast.nio.tcp.TcpIpConnectionManager

INFO: [10.0.0.56]:5700 [tomcat1] [3.5.4] Established socket connection between /10.0.0.56:33009


Is it default behaviour to see "Connection refused" out of the box? What's Hazelcast attempting at this point.


Now, when our application actually attempts to use a hazelcast feature, we get the following continuous stream of errors:


WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}

Jul 16, 2016 6:08:30 PM com.hazelcast.spi.OperationService

WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}

Jul 16, 2016 6:08:31 PM com.hazelcast.spi.OperationService

WARNING: [10.0.0.56]:5700 [tomcat1] [3.5.4] No Invocation found for response: ErrorResponse{callId=0, urgent=true, cause=com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.}

Jul 16, 2016 6:08:32 PM com.hazelcast.spi.OperationService


On the other node, which was started first, in the catalina.out, once the app on node2 wakes up, we start getting:


Jul 16, 2016 6:09:01 PM com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor

SEVERE: [10.0.0.55]:5700 [tomcat1] [3.5.4] Failed to process packet: Packet{header=17, isResponse=false, isOperation=true, isEvent=false, partitionId=-1, conn=Connection [/10.0.0.55:5700 -> /10.0.0.56:33009], endpoint=Address[10.0.0.56]:5700, live=true, type=MEMBER} on hz._hzInstance_1_tomcat1.generic-operation.thread-3

com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.

        at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:380)

        at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:331)

        at com.hazelcast.nio.serialization.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:489)

        at com.hazelcast.cluster.impl.ConfigCheck.readData(ConfigCheck.java:215)

        at com.hazelcast.cluster.impl.JoinMessage.readData(JoinMessage.java:80)

        at com.hazelcast.cluster.impl.operations.MasterDiscoveryOperation.readInternal(MasterDiscoveryOperation.java:46)

        at com.hazelcast.spi.Operation.readData(Operation.java:491)

        at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:111)

        at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:39)

        at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)

        at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:276)

        at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:200)

        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:300)

        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)

        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)

        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)

        at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)

Caused by: java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.HashMap]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.

        at java.lang.Throwable.initCause(Throwable.java:457)

        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1316)

        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1196)

        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)

        at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:125)

        at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:98)

        at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:113)

        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)

        at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:201)

        at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)

        at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:325)

        ... 15 more

Caused by: java.lang.ClassNotFoundException

        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)

        ... 28 more


Config deviating from default is as follows:


    <network>

        <port auto-increment="true" port-count="20">5700</port>

        <outbound-ports>

            <ports>33000-33009</ports>

        </outbound-ports>

        <join>

            <multicast enabled="false" />

            <tcp-ip enabled="true">

                <member>10.0.0.55</member>

                <member>10.0.0.56</member>

            </tcp-ip>

        </join>

        <ssl enabled="false"/>

        <socket-interceptor enabled="false"/>

        <symmetric-encryption enabled="false">

            <algorithm>PBEWithMD5AndDES</algorithm>

            <salt>***</salt>

            <password>***</password>

            <iteration-count>19</iteration-count>

        </symmetric-encryption>

    </network>



So, quite a lot of errors.....if someone has seen this before...much appreciated....


Vassilis Bekiaris

не прочитано,
18 лип. 2016 р., 02:19:1218.07.16
Кому: Hazelcast, nemer...@gmail.com
Hi, 

see my reply in the other topic you posted about TCP IP joiner; notice that these log entries are INFO level, there is no error, hazelcast is just recording its attempt to locate a running node on a number of IP:PORT combinations, based on your config.

The stack trace here is also the same as the one you posted on the other topic, so please look in your tomcat logs to find out why the webapp is being shut down by tomcat. Please keep one of the two topics alive to avoid duplicating the same conversation :)

Thanks!
Vassilis

Vassilis Bekiaris

не прочитано,
11 серп. 2016 р., 01:29:0711.08.16
Кому: Neil Emerick, haze...@googlegroups.com
Hi Neil,
Glad this worked for you. Please don't hesitate to come back with any other questions. 

Cheers!
Vassilis 

8 Αυγ 2016, 08:15, ο/η Neil Emerick <nemer...@gmail.com> έγραψε:

Thanks for your response. 

You were right. The version was out of date. Simply upgrading with no other change solved the problem. 

Neil Emerick
021 790 9910


Відповісти всім
Відповісти автору
Переслати
0 нових повідомлень