hazelcast hangs during startup on linux box

110 views
Skip to first unread message

Alex

unread,
Apr 11, 2011, 5:45:31 PM4/11/11
to Hazelcast
We are having trouble getting hazelcast to startup reliably. We have
three instances running on Linux boxes and they do not always find
each other That is, most of the time two of them will startup and find
each other and the third one will either hang or not join the group.
Which machine is failing to join varies.

The hazelcast configuration file is:
<hazelcast>
<properties>
<hazelcast.max.no.heartbeat.seconds>180</
hazelcast.max.no.heartbeat.seconds>
</properties>
<group>
<name>testiewestie</name>
<password>dev-pass</password>
</group>
<network>
<port auto-increment="true">6453</port>
<join>
<multicast enabled="false">
<multicast-group>224.2.2.3</multicast-group>
<multicast-port>54327</multicast-port>
</multicast>
<tcp-ip enabled="true">
<interface>10.170.0.57</interface>
<interface>10.170.0.195</interface>
<interface>10.170.0.190</interface>
</tcp-ip>
</join>
<interfaces enabled="false">
<interface>10.170.0.*</interface>
</interfaces>
<symmetric-encryption enabled="false">
<!--
encryption algorithm such as
DES/ECB/PKCS5Padding,
PBEWithMD5AndDES,
AES/CBC/PKCS5Padding,
Blowfish,
DESede
-->
<algorithm>PBEWithMD5AndDES</algorithm>
<!-- salt value to use when generating the secret key -->
<salt>thesalt</salt>
<!-- pass phrase to use when generating the secret key -->
<password>thepass</password>
<!-- iteration count to use when generating the secret key
-->
<iteration-count>19</iteration-count>
</symmetric-encryption>
<asymmetric-encryption enabled="false">
<!-- encryption algorithm -->
<algorithm>RSA/NONE/PKCS1PADDING</algorithm>
<!-- private key password -->
<keyPassword>thekeypass</keyPassword>
<!-- private key alias -->
<keyAlias>local</keyAlias>
<!-- key store type -->
<storeType>JKS</storeType>
<!-- key store password -->
<storePassword>thestorepass</storePassword>
<!-- path to the key store -->
<storePath>keystore</storePath>
</asymmetric-encryption>
</network>
<executor-service>
<core-pool-size>16</core-pool-size>
<max-pool-size>64</max-pool-size>
<keep-alive-seconds>300</keep-alive-seconds>
</executor-service>
<queue name="default">
<!--
Maximum size of the queue. When a JVM's local queue size
reaches the maximum,
all put/offer operations will get blocked until the queue
size
of the JVM goes down below the maximum.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<max-size-per-jvm>10000</max-size-per-jvm>
<!--
Maximum number of seconds for each item to stay in the
queue. Items that are
not consumed in <time-to-live-seconds> will automatically
get evicted from the queue.
Any integer between 0 and Integer.MAX_VALUE. 0 means
infinite. Default is 0.
-->
<time-to-live-seconds>0</time-to-live-seconds>
</queue>
<map name="default">
<!--
Number of backups. If 1 is set as the backup-count for
example,
then all entries of the map will be copied to another JVM
for
fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
-->
<backup-count>1</backup-count>
<!--
Valid values are:
NONE (no eviction),
LRU (Least Recently Used),
LFU (Least Frequently Used).
NONE is the default.
-->
<eviction-policy>NONE</eviction-policy>
<!--
Maximum size of the map. When max size is reached,
map is evicted based on the policy defined.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<max-size>0</max-size>
<!--
When max. size is reached, specified percentage of
the map will be evicted. Any integer between 0 and 100.
If 25 is set for example, 25% of the entries will
get evicted.
-->
<eviction-percentage>25</eviction-percentage>
<!--
While recovering from split-brain (network partitioning),
map entries in the small cluster will merge into the
bigger cluster
based on the policy set here. When an entry merge into the
cluster, there might an existing entry with the same key
already.
Values of these entries might be different for that same
key.
Which value should be set for the key? Conflict is
resolved by
the policy set here. Default policy is hz.ADD_NEW_ENTRY

There are built-in merge policies such as
hz.NO_MERGE ; no entry will merge.
hz.ADD_NEW_ENTRY ; entry will be added if the merging
entry's key
doesn't exist in the cluster.
hz.HIGHER_HITS ; entry with the higher hits wins.
hz.LATEST_UPDATE ; entry with the latest update wins.
-->
<merge-policy>hz.ADD_NEW_ENTRY</merge-policy>
</map>
<merge-policies>
<!-- Add your own map merge policy implementations here -->
<!--
<map-merge-policy name="MY_MERGE_POLICY">
<class-name>com.acme.MyOwnMergePolicy</class-name>
</map-merge-policy>
-->
</merge-policies>

</hazelcast>

The code being executed is simply:

public static void main(String[] args) {
com.hazelcast.core.Hazelcast.getLock("foobar"); // <-- this is
where it hangs
System.out.println("got lock!");
java.io.Console.readLine();
com.hazelcast.core.Hazelcast.shutdownAll()
}

The hanging process prints out:

Apr 11, 2011 2:36:50 PM com.hazelcast.config.XmlConfigBuilder
INFO: Using configuration file at /home/asantoro/hazelcast.xml
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.InSelector
FINEST: [testiewestie] Started Selector at 6454
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] Starting ExecutorManager
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service x:default
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service x:hz.client
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service
x:hz.migration
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service x:hz.query
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service x:hz.store
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.ExecutorManager
FINEST: [testiewestie] creating new named executor service x:hz.events
Apr 11, 2011 2:36:50 PM com.hazelcast.system
INFO: [testiewestie] Hazelcast 1.9 (20100912) starting at
Address[10.170.0.195:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.system
INFO: [testiewestie] Copyright (C) 2008-2010 Hazelcast.com
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.LifecycleServiceImpl
INFO: [testiewestie] Address[10.170.0.195:6454] is STARTING
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] We are asked to start and completelyShutdown is
false
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] Starting thread hz.InThread
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] Starting thread hz.OutThread
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] Starting thread hz.ServiceThread
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] Adding ShutdownHook
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] finished starting threads, calling join
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.195:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.195:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.190:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.190:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.57:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.cluster.ClusterManager
FINEST: [testiewestie] ClusterManager adding Member
[10.170.0.195:6454] this
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.57:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.57:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.57:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.57:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.190:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.195:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.190:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.190:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connecting to Address[10.170.0.195:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6455]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6452]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6451]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connected to Address[10.170.0.57:6454]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.190:6451],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.57:6451],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connected to Address[10.170.0.190:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connected to Address[10.170.0.195:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connected to Address[10.170.0.57:6453]
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.195:6455],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.190:6455],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.195:6451],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.190:6452],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.57:6455],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.195:6452],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.57:6452],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:50 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connected to Address[10.170.0.190:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.CMap
FINEST: [testiewestie] c:__hz_Proxies Cleanup , dirty:0, purge:0,
evict:0, unknown:0, stillOwned:0, backupPurge:0
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] we are going to try to connect to each address,
but no more than five times
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 0 to
Address[10.170.0.195:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.195:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 0 to
Address[10.170.0.195:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.195:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 0 to
Address[10.170.0.190:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.190:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.195:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 0 to
Address[10.170.0.190:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.195:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] found and sending join request for
Address[10.170.0.190:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.190:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.190:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.195:6452],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.190:6455],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 1 to
Address[10.170.0.57:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.195:6451],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.57:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connecting to Address[10.170.0.57:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 1 to
Address[10.170.0.57:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] found and sending join request for
Address[10.170.0.57:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connection check. connected: false,
Address[10.170.0.57:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 2 to
Address[10.170.0.57:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINE: [testiewestie] ** setting master address to
Address[10.170.0.57:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Couldn't connect to Address[10.170.0.57:6455],
cause: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] found and sending join request for
Address[10.170.0.57:6454]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.Connection
FINE: [testiewestie] Connection lost Address[10.170.0.57:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] Selector exception at hz.OutThread, cause=
java.nio.channels.CancelledKeyException
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:116)
at java.lang.Thread.run(Thread.java:619)
Apr 11, 2011 2:36:51 PM com.hazelcast.cluster.ClusterManager
FINEST: [testiewestie] Connection is removed Address[10.170.0.57:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 3 to
Address[10.170.0.57:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINE: [testiewestie] closing selector hz.OutThread
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.57:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 3 to
Address[10.170.0.57:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.57:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.cluster.ClusterManager
FINEST: [testiewestie] Removing Address Address[10.170.0.57:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.57:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 3 to
Address[10.170.0.190:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] found and sending join request for
Address[10.170.0.190:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 4 to
Address[10.170.0.195:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.195:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.195:6455]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 4 to
Address[10.170.0.190:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.190:6452]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 4 to
Address[10.170.0.190:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.nio.OutSelector
FINEST: [testiewestie] connect to Address[10.170.0.190:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] number of join requests is greater than 5, no
join request will be sent for Address[10.170.0.190:6451]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] connection attempt 4 to
Address[10.170.0.195:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] found and sending join request for
Address[10.170.0.195:6453]
Apr 11, 2011 2:36:51 PM com.hazelcast.impl.Node
FINEST: [testiewestie] FOUND true
Apr 11, 2011 2:37:01 PM com.hazelcast.impl.CMap
FINEST: [testiewestie] c:__hz_Proxies Cleanup , dirty:0, purge:0,
evict:0, unknown:0, stillOwned:0, backupPurge:0


And some version information:

os = Linux 2.6.9-89.0.26 (64-bit) Redhat
jdk = java version "1.6.0_18"
hazelcast.version = 1.9
hazelcast.build = 20100912

Note that the problem happens 50% of the time, not always. Any
suggestions as to how I can make this work reliably?

Fuad Malikov

unread,
Apr 12, 2011, 1:42:28 AM4/12/11
to haze...@googlegroups.com, Alex
Alex,

What is the purpose of the code. The member starts gets the lock and shutdowns immediately after that. It seems that, when the other nodes try to connect the first node shutdowns itself.
Can you describe what you try to achieve with the code?

BTW, We strongly recommend you to upgrade to 1.9.2.2

Fuad


--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.




--

@fuadm

Alex

unread,
Apr 12, 2011, 2:07:38 PM4/12/11
to Hazelcast
The code I put there is just test code. It starts, gets a lock, waits
for the user to press a key and exits. This way I can start and stop
various instances easily and see if they can find each other. The
production code follows the same pattern, but instead of waiting for a
key press it executes our actual functionality. The end goal is to
have three processes running, with one executing and the two others in
standby mode. If the running process fails, one of the standby
processes will acquire the lock and take over processing.

I will download 1.9.2.2 and give it a try. Maybe I will have better
luck with it.

Thanks for your help!
Message has been deleted

Alex

unread,
Apr 12, 2011, 5:06:38 PM4/12/11
to Hazelcast
The move to 1.9.2.2 seems to have done the trick. Started and stopped
the processes several times and they seem to have successfully found
each other.

Thanks for your help!

P.S. Removed a previous post where I claimed locks were not working,
it was my mistake, things were working fine.

Fuad Malikov

unread,
Apr 13, 2011, 2:42:51 AM4/13/11
to haze...@googlegroups.com, Alex
Cool. When you posted the last message we were trying to figure out what could be wrong on your site:)

Fuad

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.




--

@fuadm
Reply all
Reply to author
Forward
0 new messages