Hazelcast cluster not active

870 views
Skip to first unread message

Петър Иванов

unread,
Apr 26, 2016, 1:26:28 PM4/26/16
to Hazelcast
Hello,

I have one hazelcast node running on separate ubuntu server. There is another web application deployed on tomcat which is connecting to the node. Everything is ok, until we run into this issue :

This is what I see in the node logs :

Apr 26, 2016 11:57:46 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: []:5701 [dev] [3.4] Connection [Address[]:58439] lost. Reason: java.io.EOFException[Remote socket closed!]
Apr 26, 2016 11:57:46 AM com.hazelcast.nio.tcp.ReadHandler
WARNING: []:5701 [dev] [3.4] hz._hzInstance_1_dev.IO.thread-in-1 Closing socket to endpoint Address[]:58439, Cause:java.io.EOFException: Remote socket closed!
Apr 26, 2016 11:57:46 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: []:5701 [dev] [3.4] Connection [Address[]:58438] lost. Reason: java.io.EOFException[Remote socket closed!]
Apr 26, 2016 11:57:46 AM com.hazelcast.nio.tcp.ReadHandler
WARNING: []:5701 [dev] [3.4] hz._hzInstance_1_dev.IO.thread-in-0 Closing socket to endpoint Address[]:58438, Cause:java.io.EOFException: Remote socket closed!
Apr 26, 2016 11:57:46 AM com.hazelcast.client.ClientEndpointManager
INFO: []]:5701 [dev] [3.4] Destroying ClientEndpoint{conn=Connection [0.0.0.0/0.0.0.0:5701 ->:58439], endpoint=Address[]:58439, live=false, type=JAVA_CLIENT, principal='ClientPrincipal{uuid='e725ce7f-f208-4955-a214-1fa42d853deb', ownerUuid='3ce590a6-4f7c-4619-85d2-42e17c978e34'}', firstConnection=false, authenticated=true}
Apr 26, 2016 11:57:46 AM com.hazelcast.client.ClientEndpointManager
INFO: []:5701 [dev] [3.4] Destroying ClientEndpoint{conn=Connection [0.0.0.0/0.0.0.0:5701 -> null], endpoint=Address[]:58438, live=false, type=JAVA_CLIENT, principal='ClientPrincipal{uuid='e725ce7f-f208-4955-a214-1fa42d853deb', ownerUuid='3ce590a6-4f7c-4619-85d2-42e17c978e34'}', firstConnection=true, authenticated=true}
Apr 26, 2016 11:57:49 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: []:5701 [dev] [3.4] Connection [Address[]:58442] lost. Reason: java.io.EOFException[Remote socket closed!]
Apr 26, 2016 11:57:49 AM com.hazelcast.nio.tcp.ReadHandler
WARNING: []:5701 [dev] [3.4] hz._hzInstance_1_dev.IO.thread-in-2 Closing socket to endpoint Address[]:58442, Cause:java.io.EOFException: Remote socket closed!
Apr 26, 2016 11:57:49 AM com.hazelcast.client.ClientEndpointManager
INFO: []:5701 [dev] [3.4] Destroying ClientEndpoint{conn=Connection [/:5701 -> ....153/.53:58443], endpoint=Address[]:58443, live=false, type=JAVA_CLIENT, principal='ClientPrincipal{uuid='4a8422fc-52f9-4ccc-9a34-0b4e44af4227', ownerUuid='3ce590a6-4f7c-4619-85d2-42e17c978e34'}', firstConnection=false, authenticated=true}
Apr 26, 2016 11:57:49 AM com.hazelcast.client.ClientEndpointManager
INFO: []:5701 [dev] [3.4] Destroying ClientEndpoint{conn=Connection [0.0.0.0/0.0.0.0:5701 -> ....153/:58442], endpoint=Address[]:58442, live=false, type=JAVA_CLIENT, principal='ClientPrincipal{uuid='4a8422fc-52f9-4ccc-9a34-0b4e44af4227', ownerUuid='3ce590a6-4f7c-4619-85d2-42e17c978e34'}', firstConnection=true, authenticated=true}
Apr 26, 2016 11:57:51 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: []:5701 [dev] [3.4] Connection [Address[]:58443] lost. Reason: java.io.IOException[Connection reset by peer]
Apr 26, 2016 11:57:51 AM com.hazelcast.nio.tcp.WriteHandler
WARNING: []:5701 [dev] [3.4] hz._hzInstance_1_dev.IO.thread-out-0 Closing socket to endpoint Address[]:58443, Cause:java.io.IOException: Connection reset by peer
Apr 26, 2016 12:30:40 PM com.hazelcast.core.LifecycleService
INFO: []:5701 [dev] [3.4] Address[]:5701 is SHUTTING_DOWN
Apr 26, 2016 12:30:40 PM com.hazelcast.client.spi.impl.ClusterListenerThread
WARNING: Error while listening cluster events! -> ClientConnection{live=true, writeHandler=com.hazelcast.client.connection.nio.ClientWriteHandler@408d849b, readHandler=com.hazelcast.client.connection.nio.ClientReadHandler@afcfd39, connectionId=1, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[connected local=/:34093 remote=/:5701]}, remoteEndpoint=Address[]:5701}, Error: java.io.EOFException: Remote socket closed!

At the same moment on tomcat logs I have:

Apr 26, 2016 11:57:46 AM com.hazelcast.client.spi.impl.ClusterListenerThread
WARNING: Error while listening cluster events! -> ClientConnection{live=false, writeHandler=com.hazelcast.client.connection.nio.ClientWriteHandler@7bede3c0, readHandler=com.hazelcast.client.connection.nio.ClientReadHandler@3f0391d0, connectionId=1, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[closed]}, remoteEndpoint=Address[]:5701}, Error: java.nio.channels.AsynchronousCloseException
Apr 26, 2016 11:57:46 AM com.hazelcast.client.connection.nio.ClientConnection
WARNING: Connection [null] lost. Reason: Socket explicitly closed
Apr 26, 2016 11:57:46 AM com.hazelcast.core.LifecycleService
INFO: HazelcastClient[hz.client_0_dev][3.4] is CLIENT_DISCONNECTED
Apr 26, 2016 11:57:49 AM com.hazelcast.client.connection.nio.ClientConnection
WARNING: Connection [/:5701] lost. Reason: Socket explicitly closed
Apr 26, 2016 11:57:49 AM com.hazelcast.client.connection.nio.ClientConnection
WARNING: Connection [null] lost. Reason: Socket explicitly closed
Apr 26, 2016 11:57:49 AM com.hazelcast.client.spi.impl.ClusterListenerThread
WARNING: Error while listening cluster events! -> ClientConnection{live=false, writeHandler=com.hazelcast.client.connection.nio.ClientWriteHandler@40d15053, readHandler=com.hazelcast.client.connection.nio.ClientReadHandler@6df09a63, connectionId=1, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[closed]}, remoteEndpoint=Address[]:5701}, Error: java.nio.channels.AsynchronousCloseException
Apr 26, 2016 11:57:49 AM com.hazelcast.core.LifecycleService
INFO: HazelcastClient[hz.client_1_dev][3.4] is CLIENT_DISCONNECTED
Apr 26, 2016 11:57:49 AM com.hazelcast.client.spi.ClientPartitionService
SEVERE: Error while fetching cluster partition table!
com.hazelcast.core.HazelcastException: ConnectionManager is not active!!!
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$ConnectionProcessor.call(ClientConnectionManagerImpl.java:365)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$ConnectionProcessor.call(ClientConnectionManagerImpl.java:350)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.hazelcast.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

i`ve just removed the ips from the logs
After this moment the connection is destroyed and I need to restart both applications.

Hazelcast version is 3.4

Can somebody help me in finding the root cause of the issue.

Thanks

Peter Veentjer

unread,
Apr 27, 2016, 12:34:09 AM4/27/16
to haze...@googlegroups.com
Do you have a firewall between the client and server?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/e9172737-f775-4b03-a9e3-9fc8db7e1a94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Петър Иванов

unread,
Apr 27, 2016, 3:34:54 AM4/27/16
to Hazelcast
No the firewall is turned off

Петър Иванов

unread,
Apr 27, 2016, 11:30:45 PM4/27/16
to Hazelcast
I found the problem - it is out of memory error on Tomcat side. 


On Tuesday, April 26, 2016 at 8:26:28 PM UTC+3, Петър Иванов wrote:

Peter Veentjer

unread,
Apr 28, 2016, 4:07:43 AM4/28/16
to haze...@googlegroups.com
Thanks for reporting!

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d0322708-49fa-4d36-863a-89016e623726%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages