hazel cast exception at start up

[Rajeev Mutalik]

Hi,

Whenever I start my module using hazelcast in two servers, it starts
up with error logs in server.log (Glassfish server). Though the two
nodes are able to communicate with each other but this error log gives
a wrong indication at start up.

Please help.

Code:

hzConfig = new Config();
//readConfiguration();
logger.debug("{}: Port for hazel cast is " +
hazelCastConfiguration.getHazelCastDeliverSMPort(), myName);

hzConfig.setPort(hazelCastConfiguration.getHazelCastDeliverSMPort());
hzConfig.setPortAutoIncrement(false);
networkConfig = hzConfig.getNetworkConfig();
join = networkConfig.getJoin();
join.getMulticastConfig().setEnabled(false);

join.getTcpIpConfig().addMember(hazelCastConfiguration.getHazelCastOtherNodeIp()).

addMember(hazelCastConfiguration.getHazelCastThisNodeIp()).setEnabled(true);
//
setRequiredMember(hazelCastConfiguration.getHazelCastThisNodeIp());

networkConfig.getInterfaces().setEnabled(true).addInterface("158.234.57.*");
networkConfig.setJoin(join);
hzConfig.setNetworkConfig(networkConfig);
logger.debug("{}: Creating HazelCast Instance", myName);
hazelcastInstance =
Hazelcast.newHazelcastInstance(hzConfig);
logger.debug("{}: Cluster now....", myName);
Cluster cluster = hazelcastInstance.getCluster();
cluster.addMembershipListener(this);
Set<Member> members = cluster.getMembers();
Iterator<Member> iterator = members.iterator();
while (iterator.hasNext()) {
Member member = (Member) iterator.next();
if
(member.getInetSocketAddress().getAddress().getHostAddress().equalsIgnoreCase(hazelCastConfiguration.getHazelCastOtherNodeIp())
||

member.getInetSocketAddress().getHostName().equalsIgnoreCase(hazelCastConfiguration.getHazelCastOtherNodeIp()))
{
otherNodeActive = true;
}
}
if (otherNodeActive) {
logger.debug("{}: Other node is active", myName);
} else {
logger.debug("{}: Other node is not active", myName);
}




Error logs in server.log



[#|2012-02-13T09:02:14.617+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,617 DEBUG
[Thread-19253] - ConnectionManager.run(): Creating hazelcast instance
|#]

[#|2012-02-13T09:02:14.647+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,647 DEBUG
[Thread-19253] - QueueManager: readConfiguration() - enter
|#]

[#|2012-02-13T09:02:14.656+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,656 DEBUG
[Thread-19253] - QueueManager: readConfiguration() - exit
|#]

[#|2012-02-13T09:02:14.657+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,657 DEBUG
[Thread-19253] - QueueManager: Intialising
|#]

[#|2012-02-13T09:02:14.739+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,739 DEBUG
[Thread-19253] - QueueManager: Port for hazel cast is 6901
|#]

[#|2012-02-13T09:02:14.740+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:14,740 DEBUG
[Thread-19253] - QueueManager: Creating HazelCast Instance
|#]

[#|2012-02-13T09:02:16.055+0100|INFO|sun-appserver2.1|
com.hazelcast.system|_ThreadID=136;_ThreadName=Thread-19253;|[dev]
Hazelcast 1.9.3.4 (20110815) starting at Address[158.234.57.26:6901]|
#]

[#|2012-02-13T09:02:16.058+0100|INFO|sun-appserver2.1|
com.hazelcast.system|_ThreadID=136;_ThreadName=Thread-19253;|[dev]
Copyright (C) 2008-2011 Hazelcast.com|#]

[#|2012-02-13T09:02:16.252+0100|INFO|sun-appserver2.1|
com.hazelcast.impl.LifecycleServiceImpl|
_ThreadID=136;_ThreadName=Thread-19253;|[dev]
Address[158.234.57.26:6901] is STARTING|#]

[#|2012-02-13T09:02:16.289+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:16.298+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:16.301+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:16.304+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:17.298+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:17.302+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:17.310+0100|WARNING|sun-appserver2.1|
com.hazelcast.nio.OutSelector|_ThreadID=137;_ThreadName=hz.
1.OutThread;_RequestID=2c378033-8a09-42c1-9ab3-0a19b6482818;|[dev]
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
500)
at com.hazelcast.nio.OutSelector
$Connector.run(OutSelector.java:100)
at
com.hazelcast.nio.SelectorBase.processSelectionQueue(SelectorBase.java:
102)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:117)
at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-02-13T09:02:23.354+0100|INFO|sun-appserver2.1|
com.hazelcast.cluster.ClusterManager|_ThreadID=138;_ThreadName=hz.
1.ServiceThread;|[dev]

Members [2] {
Member [158.234.57.22:6901]
Member [158.234.57.26:6901] this
}
|#]

[#|2012-02-13T09:02:25.658+0100|INFO|sun-appserver2.1|
com.hazelcast.impl.management.ManagementCenterService|
_ThreadID=136;_ThreadName=Thread-19253;|[dev] Hazelcast Management
Center started at port 7001.|#]

[#|2012-02-13T09:02:25.659+0100|INFO|sun-appserver2.1|
com.hazelcast.impl.LifecycleServiceImpl|
_ThreadID=136;_ThreadName=Thread-19253;|[dev]
Address[158.234.57.26:6901] is STARTED|#]

[#|2012-02-13T09:02:25.659+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:25,659 DEBUG
[Thread-19253] - QueueManager: Clustering now....
|#]

[#|2012-02-13T09:02:25.662+0100|INFO|sun-appserver2.1|
javax.enterprise.system.stream.out|
_ThreadID=136;_ThreadName=Thread-19253;|2012-02-13 08:02:25,661 DEBUG
[Thread-19253] - QueueManager: Other node is active

[Talip Ozturk]

So you are sure that port 6901 is accessible on both. telnet this port
from the other server:
1. start node1
2. on node2 telnet node1-ip 6901 and type 'stats' and hit enter

let me know what happens.

Can you start the Glassfish servers JVMs with
-Djava.net.preferIPv4Stack=true option.

-talip

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To post to this group, send email to haze...@googlegroups.com.
> To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.
>

[Rajeev Mutalik]

Hi Talip,

Yes both the nodes are able to communicate with each other on these
ports

-bash-3.00$ telnet 158.234.57.26 6901
Trying 158.234.57.26...
Connected to 158.234.57.26.
Escape character is '^]'.
stats
STAT uptime 76820
STAT threads 1
STAT waiting_requests 1
STAT curr_connections 1
STAT total_connections 1
STAT bytes 0
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_delete 0
STAT get_hits 0
STAT get_misses 0
END


-bash-3.00$ telnet 158.234.57.22 6901
Trying 158.234.57.22...
Connected to 158.234.57.22.
Escape character is '^]'.
stats
STAT uptime 76785
STAT threads 1
STAT waiting_requests 1
STAT curr_connections 1
STAT total_connections 1
STAT bytes 0
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_delete 0
STAT get_hits 0
STAT get_misses 0
END

I may not be able to start JVM on glassfish with the options u have
mentioned as there are quite a few other modules running on glassfish.

Thanks and Regards,
Rajeev