HA 3 member cluster, issue after slave reboot

[Nicolas Menoux]

Hi,

I'm testing a 3 member HA neo4j-enterprise-1.8.2 cluster on Ubunutu 12.04.2 LTS boxes. I've reset my database and here are the conf (server.1 = 192.168.1.231, server.2 = 192.168.1.232, server.3 = 192.168.1.233)

same conf for 3 nodes

coord.cfg

-------------

server.1=192.168.1.231:2888:3888

server.2=192.168.1.232:2888:3888

server.3=192.168.1.233:2888:3888

clientPort=2181

Node 1

=====

(neo4j.properties)

ha.server_id=1

ha.coordinators=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181

#ha.cluster_name =

ha.server = 192.168.1.231:6001

(neo4j-server.properties)

org.neo4j.server.webserver.address=192.168.1.231

org.neo4j.server.database.mode=HA

and 

/data/coordinator/myid contains 1

Node 2

=====

(neo4j.properties)

ha.server_id=2

ha.coordinators=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181

#ha.cluster_name =

ha.server = 192.168.1.232:6001

(neo4j-server.properties)

org.neo4j.server.webserver.address=192.168.1.232

org.neo4j.server.database.mode=HA

and 

/data/coordinator/myid contains 2

Node 3

=====

(neo4j.properties)

ha.server_id=3

ha.coordinators=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181

#ha.cluster_name =

ha.server = 192.168.1.233:6001

(neo4j-server.properties)

org.neo4j.server.webserver.address=192.168.1.233

org.neo4j.server.database.mode=HA

and 

/data/coordinator/myid contains 3

coordinator and neo4f services are configured to be start at reboot (chkconfig)

Once the coordinator directory versions2 is removed from the 3 nodes and the coord service relaunched, the cluster could be started and it work correctly.

I stop the node 2, node 1 remains master

I stop the node 3, node 1 is still working

When I restart node 2... the neo4j could not start but coordinator seems to be ok.

I manually restart the neo4j service and it works, the cluster is running and node 2 is slave.

I do not understand why the service could not start at the server reboot phase but only by relaunch it manually.

I've looked at the zookeeper log and here are the rows where we could see that the node 2 is unable to contact himself for coordinator service.

1) REBOOT PHASE

08-03-2013 10:53:29,795 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT

08-03-2013 10:53:29,831 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:host.name=neo4j2

08-03-2013 10:53:29,831 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_15

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-i386/jre

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/local/neo4j-enterprise-1.8.2/lib/concurrentlinkedhashmap-lru-1.3.1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/geronimo-jta_1.1_spec-1.1.1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/lucene-core-3.5.0.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-backup-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-com-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-consistency-check-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-cypher-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-graph-algo-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-graph-matching-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-ha-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-jmx-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-kernel-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-lucene-index-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-management-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-shell-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-udc-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/org.apache.servicemix.bundles.jline-0.9.94_1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/org.apache.servicemix.bundles.netty-3.2.5.Final_1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/scala-library-2.9.1-1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/server-api-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/slf4j-api-1.6.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/zookeeper-3.3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/antlr-2.7.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-3.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-analysis-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-commons-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-tree-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-util-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/bcprov-jdk16-140.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/blueprints-core-1.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/blueprints-neo4j-graph-1.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-beanutils-1.8.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-beanutils-core-1.8.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-collections-3.2.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-configuration-1.6.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-digester-1.8.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-io-1.4.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-lang-2.4.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/gremlin-groovy-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/gremlin-java-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/groovy-1.8.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-core-asl-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-jaxrs-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-mapper-asl-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/janino-2.5.10.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jansi-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jcl-over-slf4j-1.6.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-core-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-multipart-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-server-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jettison-1.3.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jetty-6.1.25.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jetty-util-6.1.25.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jsr311-api-1.1.2.r612.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-access-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-classic-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-core-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/mimepull-1.6.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-cypher-plugin-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-gremlin-plugin-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2-static-web.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2-tests.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-advanced-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-enterprise-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/pipes-1.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/rrd4j-2.0.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/servlet-api-2.5-20081211.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/stax-api-1.0.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/coordinator/lib/log4j-1.2.16.jar:/usr/local/neo4j-enterprise-1.8.2/conf/

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

08-03-2013 10:53:29,832 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.arch=i386

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.2.0-38-generic-pae

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.name=root

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root

08-03-2013 10:53:29,833 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/local/neo4j-enterprise-1.8.2

08-03-2013 10:53:29,836 CET INFO main org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181 sessionTimeout=5000 watcher=org.neo4j.kernel.ha.zookeeper.ZooKeeperClusterClient$WatcherImpl@4f9c6d

08-03-2013 10:53:29,854 CET INFO main-SendThread() org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.233:2181

08-03-2013 10:53:29,889 CET INFO main org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181 sessionTimeout=5000 watcher=org.neo4j.kernel.ha.zookeeper.ZooClient$WatcherImpl@182e445

08-03-2013 10:53:29,891 CET INFO main-SendThread() org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.233:2181

08-03-2013 10:53:31,538 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 1683ms for sessionid 0x0, closing socket connection and attempting reconnect

08-03-2013 10:53:31,569 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 1678ms for sessionid 0x0, closing socket connection and attempting reconnect

08-03-2013 10:53:31,678 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.232:2181

08-03-2013 10:53:31,694 CET WARN main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

08-03-2013 10:53:31,876 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.232:2181

08-03-2013 10:53:32,166 CET WARN main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

08-03-2013 10:53:32,459 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.231:2181

08-03-2013 10:53:32,473 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 10:53:32,479 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

08-03-2013 10:53:32,810 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.231:2181

08-03-2013 10:53:32,824 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 10:53:32,826 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

08-03-2013 10:53:33,020 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.233/192.168.1.233:2181

08-03-2013 10:53:33,362 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.233/192.168.1.233:2181

08-03-2013 10:53:34,688 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2104ms for sessionid 0x0, closing socket connection and attempting reconnect

08-03-2013 10:53:35,030 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2104ms for sessionid 0x0, closing socket connection and attempting reconnect

08-03-2013 10:53:35,167 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.232/192.168.1.232:2181

08-03-2013 10:53:35,168 CET WARN main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

08-03-2013 10:53:35,759 CET INFO main-SendThread(192.168.1.233:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.232/192.168.1.232:2181

08-03-2013 10:53:35,760 CET WARN main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

08-03-2013 10:53:36,954 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.231/192.168.1.231:2181

08-03-2013 10:53:36,956 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 10:53:36,957 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

08-03-2013 10:53:37,033 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.231/192.168.1.231:2181

08-03-2013 10:53:37,035 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 10:53:37,036 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

08-03-2013 10:53:37,318 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.233/192.168.1.233:2181

.... (same connection refused many times)

08-03-2013 10:55:31,937 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.231/192.168.1.231:2181

08-03-2013 10:55:31,939 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 10:55:31,941 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

08-03-2013 10:55:32,803 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeerConfig - Reading configuration from: conf/coord.cfg

08-03-2013 10:55:32,812 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeerConfig - Defaulting to majority quorums

08-03-2013 10:55:32,926 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeerMain - Starting quorum peer

08-03-2013 10:55:32,960 CET INFO main org.apache.zookeeper.server.NIOServerCnxn - binding to port 0.0.0.0/0.0.0.0:2181

08-03-2013 10:55:33,045 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeer - tickTime set to 3000

08-03-2013 10:55:33,045 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeer - minSessionTimeout set to -1

08-03-2013 10:55:33,045 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeer - maxSessionTimeout set to -1

08-03-2013 10:55:33,045 CET INFO main org.apache.zookeeper.server.quorum.QuorumPeer - initLimit set to 20

08-03-2013 10:55:33,174 CET INFO main org.apache.zookeeper.server.persistence.FileSnap - Reading snapshot data/coordinator/version-2/snapshot.400000001

08-03-2013 10:55:33,235 CET INFO Thread-1 org.apache.zookeeper.server.quorum.QuorumCnxManager - My election bind port: 3888

08-03-2013 10:55:33,247 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.QuorumPeer - LOOKING

08-03-2013 10:55:33,248 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.FastLeaderElection - New election. My id =  2, Proposed zxid = 17179869185

08-03-2013 10:55:33,254 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 2 (n.leader), 17179869185 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,255 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,256 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,256 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,256 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,284 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)

08-03-2013 10:55:33,486 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 400

08-03-2013 10:55:33,886 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 800

08-03-2013 10:55:34,687 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 1600

08-03-2013 10:55:35,189 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.231:41843

08-03-2013 10:55:35,192 CET WARN NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running

08-03-2013 10:55:35,193 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /192.168.1.231:41843 (no session established for client)

08-03-2013 10:55:36,254 CET WARN WorkerSender Thread org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /192.168.1.233:3888

java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:366)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:335)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)

        at java.lang.Thread.run(Thread.java:722)

08-03-2013 10:55:36,260 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)

08-03-2013 10:55:36,461 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.QuorumPeer - FOLLOWING

08-03-2013 10:55:36,480 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.Learner - TCP NoDelay set to: true

08-03-2013 10:55:36,492 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:host.name=neo4j2

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.version=1.7.0_15

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.vendor=Oracle Corporation

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-i386/jre

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.class.path=/usr/local/neo4j-enterprise-1.8.2/lib/zookeeper-3.3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/coordinator/lib/log4j-1.2.16.jar

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA>

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=i386

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=3.2.0-38-generic-pae

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=root

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=/root

08-03-2013 10:55:36,493 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/usr/local/neo4j-enterprise-1.8.2

08-03-2013 10:55:36,494 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.ZooKeeperServer - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir data/coordinator/version-2 snapdir data/coordinator/version-2

08-03-2013 10:55:36,521 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.Learner - Getting a snapshot from leader

08-03-2013 10:55:36,523 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.Learner - Setting leader epoch 6

08-03-2013 10:55:36,532 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.persistence.FileTxnSnapLog - Snapshotting: 600000000

08-03-2013 10:55:37,089 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.231:41847

08-03-2013 10:55:37,092 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Client attempting to renew session 0x33d4957a8b80000 at /192.168.1.231:41847

08-03-2013 10:55:37,093 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.quorum.Learner - Revalidating client: 233423396534747136

08-03-2013 10:55:37,104 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.NIOServerCnxn - Established session 0x33d4957a8b80000 with negotiated timeout 6000 for client /192.168.1.231:41847

08-03-2013 10:55:38,932 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.231:41848

08-03-2013 10:55:38,933 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Client attempting to renew session 0x33d49550e6b0000 at /192.168.1.231:41848

08-03-2013 10:55:38,933 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.quorum.Learner - Revalidating client: 233423385356075008

08-03-2013 10:55:38,936 CET INFO QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.NIOServerCnxn - Established session 0x33d49550e6b0000 with negotiated timeout 6000 for client /192.168.1.231:41848

08-03-2013 10:55:39,252 CET WARN WorkerSender Thread org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /192.168.1.233:3888

java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:366)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:335)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)

        at java.lang.Thread.run(Thread.java:722)

08-03-2013 10:55:39,260 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 2 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:39,261 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LEADING (n.state), 1 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:42,252 CET WARN WorkerSender Thread org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /192.168.1.233:3888

java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:366)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:335)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)

        at java.lang.Thread.run(Thread.java:722)

08-03-2013 10:55:42,254 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 2 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:42,255 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LEADING (n.state), 1 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:45,252 CET WARN WorkerSender Thread org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /192.168.1.233:3888

java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:366)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:335)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)

        at java.lang.Thread.run(Thread.java:722)

08-03-2013 10:55:45,253 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LOOKING (n.state), 2 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:45,255 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), LEADING (n.state), 1 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:48,252 CET WARN WorkerSender Thread org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /192.168.1.233:3888

java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:366)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:335)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)

        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)

        at java.lang.Thread.run(Thread.java:722)

08-03-2013 10:55:48,253 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), FOLLOWING (n.state), 2 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:48,254 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), FOLLOWING (n.state), 2 (n.sid), FOLLOWING (my state)

08-03-2013 10:55:48,254 CET INFO WorkerReceiver Thread org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 1 (n.leader), 21474836480 (n.zxid), 6 (n.round), FOLLOWING (n.state), 2 (n.sid), FOLLOWING (my state)

2) RESTART SERVICE MANUALLY

08-03-2013 11:01:22,577 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT

08-03-2013 11:01:22,578 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:host.name=neo4j2

08-03-2013 11:01:22,578 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_15

08-03-2013 11:01:22,578 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

08-03-2013 11:01:22,578 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-i386/jre

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/local/neo4j-enterprise-1.8.2/lib/concurrentlinkedhashmap-lru-1.3.1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/geronimo-jta_1.1_spec-1.1.1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/lucene-core-3.5.0.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-backup-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-com-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-consistency-check-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-cypher-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-graph-algo-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-graph-matching-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-ha-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-jmx-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-kernel-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-lucene-index-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-management-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-shell-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/neo4j-udc-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/org.apache.servicemix.bundles.jline-0.9.94_1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/org.apache.servicemix.bundles.netty-3.2.5.Final_1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/scala-library-2.9.1-1.jar:/usr/local/neo4j-enterprise-1.8.2/lib/server-api-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/slf4j-api-1.6.2.jar:/usr/local/neo4j-enterprise-1.8.2/lib/zookeeper-3.3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/antlr-2.7.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-3.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-analysis-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-commons-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-tree-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/asm-util-3.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/bcprov-jdk16-140.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/blueprints-core-1.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/blueprints-neo4j-graph-1.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-beanutils-1.8.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-beanutils-core-1.8.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-collections-3.2.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-configuration-1.6.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-digester-1.8.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-io-1.4.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/commons-lang-2.4.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/gremlin-groovy-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/gremlin-java-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/groovy-1.8.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-core-asl-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-jaxrs-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jackson-mapper-asl-1.9.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/janino-2.5.10.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jansi-1.5.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jcl-over-slf4j-1.6.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-core-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-multipart-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jersey-server-1.9.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jettison-1.3.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jetty-6.1.25.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jetty-util-6.1.25.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/jsr311-api-1.1.2.r612.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-access-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-classic-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/logback-core-0.9.30.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/mimepull-1.6.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-cypher-plugin-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-gremlin-plugin-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2-static-web.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-1.8.2-tests.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-advanced-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/neo4j-server-enterprise-1.8.2.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/pipes-1.0.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/rrd4j-2.0.7.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/servlet-api-2.5-20081211.jar:/usr/local/neo4j-enterprise-1.8.2/system/lib/stax-api-1.0.1.jar:/usr/local/neo4j-enterprise-1.8.2/system/coordinator/lib/log4j-1.2.16.jar:/usr/local/neo4j-enterprise-1.8.2/conf/

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.arch=i386

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.2.0-38-generic-pae

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.name=root

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root

08-03-2013 11:01:22,579 CET INFO main org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/local/neo4j-enterprise-1.8.2

08-03-2013 11:01:22,580 CET INFO main org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181 sessionTimeout=5000 watcher=org.neo4j.kernel.ha.zookeeper.ZooKeeperClusterClient$WatcherImpl@4f9c6d

08-03-2013 11:01:22,591 CET INFO main-SendThread() org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.231:2181

08-03-2013 11:01:22,606 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.231/192.168.1.231:2181, initiating session

08-03-2013 11:01:22,609 CET WARN QuorumPeer:/0:0:0:0:0:0:0:0:2181 org.apache.zookeeper.server.quorum.Learner - Got zxid 0x600000001 expected 0x1

08-03-2013 11:01:22,609 CET INFO SyncThread:2 org.apache.zookeeper.server.persistence.FileTxnLog - Creating new log file: log.600000001

08-03-2013 11:01:22,610 CET INFO main org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.231:2181,192.168.1.232:2181,192.168.1.233:2181 sessionTimeout=5000 watcher=org.neo4j.kernel.ha.zookeeper.ZooClient$WatcherImpl@1c1dfa9

08-03-2013 11:01:22,612 CET INFO main-SendThread() org.apache.zookeeper.ClientCnxn - Opening socket connection to server /192.168.1.232:2181

08-03-2013 11:01:22,623 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.232:42851

08-03-2013 11:01:22,623 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.232/192.168.1.232:2181, initiating session

08-03-2013 11:01:22,624 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Client attempting to establish new session at /192.168.1.232:42851

08-03-2013 11:01:22,719 CET INFO CommitProcessor:2 org.apache.zookeeper.server.NIOServerCnxn - Established session 0x23d496d5bce0000 with negotiated timeout 6000 for client /192.168.1.232:42851

08-03-2013 11:01:22,721 CET INFO main-SendThread(192.168.1.231:2181) org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.1.231/192.168.1.231:2181, sessionid = 0x13d496d5bed0000, negotiated timeout = 6000

08-03-2013 11:01:22,722 CET INFO main-SendThread(192.168.1.232:2181) org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.1.232/192.168.1.232:2181, sessionid = 0x23d496d5bce0000, negotiated timeout = 6000

08-03-2013 11:01:24,181 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.232:42858

08-03-2013 11:01:24,181 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Processing conf command from /192.168.1.232:42858

08-03-2013 11:01:24,183 CET INFO Thread-4 org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /192.168.1.232:42858 (no session established for client)

08-03-2013 11:01:24,184 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.232:42859

08-03-2013 11:01:24,184 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Processing envi command from /192.168.1.232:42859

08-03-2013 11:01:24,185 CET INFO Thread-5 org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /192.168.1.232:42859 (no session established for client)

08-03-2013 11:01:24,185 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from /192.168.1.232:42860

08-03-2013 11:01:24,185 CET INFO NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 org.apache.zookeeper.server.NIOServerCnxn - Processing srvr command from /192.168.1.232:42860

08-03-2013 11:01:24,186 CET INFO Thread-6 org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /192.168.1.232:42860 (no session established for client)

mmm.. is it normal that the coordinator service is trying to contact himself or am I missing something in the conf ?

Thanks for your help,

Regards,

Nicolas

[Michael Hunger]

I contacted our team to help you with this. Have not enough insight into the intricasies of ZK myself.

[Michael Hunger]

Nicolas,

can you share the messages.log from all db instances with us?

On Friday, March 8, 2013 11:25:45 AM UTC+1, Nicolas Menoux wrote:

[Nicolas Menoux]

Hi Michael,

I'm not sure if my previous post was really published, so I share my 3 syslog files again. Sorry if I double my post.

Nicolas

[Michael Hunger]

Nicolas how critical is this to your operation?

We want to prioritize it correctly. So the manual startup is a working workaround right now, or?

[Nicolas Menoux]

it is not critical Michael and I confirm that neo4j manual restart is working but not during the reboot phase where service are automatically relaunched.

Thanks.

[Dave Stern (FiftyThree)]

Just a quick sanity check: have you checked the order of operations in your startup? If the coordinator starts AFTER the neo4j process, I could see this being the effect.

[Nicolas Menoux]

Hi Dave,

it was one of my thought at the beginning, the order of services launching but I've not gone deeper at that time. I will check that point and I will keep you informed. Thanks.

[Nicolas Menoux]

Yessss ! you're right, it was indeed the service order which was the root of that issue !

I've modified the names of the files in the rcX.d  directories like this :

mv /etc/rc0.d/K01neo4j /etc/rc0.d/K02neo4j

mv /etc/rc1.d/K01neo4j /etc/rc1.d/K02neo4j

mv /etc/rc2.d/S02neo4j /etc/rc2.d/S03neo4j

mv /etc/rc3.d/S02neo4j /etc/rc3.d/S03neo4j

mv /etc/rc4.d/S02neo4j /etc/rc4.d/S03neo4j

mv /etc/rc5.d/S02neo4j /etc/rc5.d/S03neo4j

in order to start neo4j after the coordinator. In the original form, both services were starting together....with this kind of deadlock !

Thanks Michael and Dave for your help, I've learnt at the same time how to modify service order :-)

Nicolas

Le mardi 12 mars 2013 15:52:50 UTC+1, Dave Stern (FiftyThree) a écrit :

[Dave Stern (FiftyThree)]

Glad this worked out for you!

Dave

[Michael Hunger]

Thanks so much Dave for pointing this out !!

Cheers

Michael

(neo4j.org) <-[:WORKS_ON]- (@mesirii) -[:TAKES_CARE_OF]-> (you) -[:WORKS_WITH]->(@Neo4j)

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.