On Neo4j 1.5 having HA issues

[Gagan]

After every 24 hours I get into a situation where error(given at the
last) occurs at master. After that, the whole cluster goes for a toss.
On the master I see "
WARNING: Could not set last committed tx id
org.neo4j.kernel.ha.zookeeper.ZooKeeperTimedOutException: Connection
to ZooKeeper server timed out, keeper state=Expired"


and on slaves
java.lang.NullPointerException
at org.neo4j.kernel.ha.SlaveIdGenerator.nextId(SlaveIdGenerator.java:
144)
OR

java.lang.NullPointerException
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.pullUpdates(HighlyAvailableGraphDatabase.java:
494)

Some how this waitForSyncConnected() thing really is waiting only on
keeperstate change , which does not change it seems...

Any ideas?

I'm running 3 machine cluster with HA on each machine. Write happen
both on slave and master.

------------Error at master where it all begins ---------------

2012-03-03 23:59:13.501+0530:
org.neo4j.kernel.ha.zookeeper.ZooClient@e5d391d, Sat Mar 03 23:59:13
IST 2012 Got event: WatchedEvent state:Disconnected type:None
path:null(path=null)
2012-03-03 23:59:13.936+0530:
org.neo4j.kernel.ha.zookeeper.ZooClient@e5d391d, Sat Mar 03 23:59:13
IST 2012 Got event: WatchedEvent state:Expired type:None
path:null(path=null)
2012-03-03 23:59:13.942+0530:
org.neo4j.kernel.ha.zookeeper.ZooClient@e5d391d, Sat Mar 03 23:59:13
IST 2012 Got event: WatchedEvent state:SyncConnected type:None
path:null(path=null)
2012-03-03 23:59:13.946+0530: getMaster 2 based on [(1, 8671356, 796),
(2, 8671356, 795)]
2012-03-03 23:59:13.946+0530: Get master before write:(null,
MachineInfo[ID:2, sequence:795, last committed tx id:8671356, server:
(192.168.27.237, 6001)])
2012-03-03 23:59:13.952+0530: Wrote HA server 192.168.27.238:6001 to
zoo keeper
2012-03-03 23:59:13.957+0530: Did setup, seq=0000000797 new
sessionId=231316217222594565
2012-03-03 23:59:13.960+0530: getMaster 3 based on [(1, 8671356, 796),
(2, 8671356, 795), (3, 8671356, 797)]
2012-03-03 23:59:13.960+0530: Get master after write:(null,
MachineInfo[ID:3, sequence:797, last committed tx id:8671356, server:
(192.168.27.238, 6001)])
2012-03-03 23:59:13.961+0530: Setting 'master-notify' to 3
2012-03-03 23:59:13.964+0530: master-notify set to 3
2012-03-03 23:59:13.965+0530: Did set 'master-notify' to 3
2012-03-03 23:59:13.965+0530: newMaster called
2012-03-03 23:59:19.005+0530: ZooKeeper exception in newMaster
Connection to ZooKeeper server timed out, keeper state=Expired
org.neo4j.kernel.ha.zookeeper.ZooKeeperTimedOutException: Connection
to ZooKeeper server timed out, keeper state=Expired
at
org.neo4j.kernel.ha.zookeeper.ZooClient.waitForSyncConnected(ZooClient.java:
248)
at
org.neo4j.kernel.ha.zookeeper.AbstractZooKeeperManager.getAllMachines(AbstractZooKeeperManager.java:
190)
at
org.neo4j.kernel.ha.zookeeper.AbstractZooKeeperManager.getMasterFromZooKeeper(AbstractZooKeeperManager.java:
135)
at
org.neo4j.kernel.ha.zookeeper.ZooKeeperBroker.getMasterReally(ZooKeeperBroker.java:
190)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.reevaluateMyself(HighlyAvailableGraphDatabase.java:
545)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.doNewMaster(HighlyAvailableGraphDatabase.java:
940)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.newMaster(HighlyAvailableGraphDatabase.java:
925)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.newMaster(HighlyAvailableGraphDatabase.java:
918)
at org.neo4j.kernel.ha.zookeeper.ZooClient.process(ZooClient.java:
141)
at org.apache.zookeeper.ClientCnxn
$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:
506)
2012-03-03 23:59:19.005+0530: Error in ZooClient.process Connection to
ZooKeeper server timed out, keeper state=Expired
org.neo4j.kernel.ha.zookeeper.ZooKeeperTimedOutException: Connection
to ZooKeeper server timed out, keeper state=Expired
at
org.neo4j.kernel.ha.zookeeper.ZooClient.waitForSyncConnected(ZooClient.java:
248)
at
org.neo4j.kernel.ha.zookeeper.AbstractZooKeeperManager.getAllMachines(AbstractZooKeeperManager.java:
190)
at
org.neo4j.kernel.ha.zookeeper.AbstractZooKeeperManager.getMasterFromZooKeeper(AbstractZooKeeperManager.java:
135)
at
org.neo4j.kernel.ha.zookeeper.ZooKeeperBroker.getMasterReally(ZooKeeperBroker.java:
190)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.reevaluateMyself(HighlyAvailableGraphDatabase.java:
545)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.doNewMaster(HighlyAvailableGraphDatabase.java:
940)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.newMaster(HighlyAvailableGraphDatabase.java:
925)
at
org.neo4j.kernel.HighlyAvailableGraphDatabase.newMaster(HighlyAvailableGraphDatabase.java:
918)
at org.neo4j.kernel.ha.zookeeper.ZooClient.process(ZooClient.java:
141)
at org.apache.zookeeper.ClientCnxn
$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:
506)

[Peter Neubauer]

Hi there,
Have you updated to 1.5.2. And verified it?

[Peter Neubauer]

Also,
is this consistently every exact 24h? Is there any event happening
like external activity or a full GC?

/peter

[Michael Hunger]

Or some firewall, network, DNS reset?

What exact version of 1.5 are you using?

Can you provide us with the messages.log files from all the ha instances and the zookeeper logs as well?

Thanks

Michael