exhibitor api - config update results in zoo.cfg with no ports for servers

[Dann S Washko]

Enter code here...

I am trying to update exhibitor configurations via the API. When I issue a POST to set-rolling the first exhibitor host attempts to restart fails to progress. When I look in the logs I see this:

Wed Sep 28 18:04:10 UTC 2016 INFO ZooKeeper Server: Starting zookeeper ... STARTED Wed Sep 28 18:04:09 UTC 2016 ERROR ZooKeeper Server: Using config: /opt/zookeeper/zookeeper-3.4.8/bin/../conf/zoo.cfg Wed Sep 28 18:04:09 UTC 2016 ERROR ZooKeeper Server: ZooKeeper JMX enabled by default Wed Sep 28 18:04:09 UTC 2016 INFO Process started via: /opt/zookeeper/zookeeper-3.4.8/bin/zkServer.sh Wed Sep 28 18:04:08 UTC 2016 INFO Kill attempted result: 0 Wed Sep 28 18:04:08 UTC 2016 INFO Attempting to start/restart ZooKeeper Wed Sep 28 18:04:08 UTC 2016 INFO Attempting to stop instance

It just hangs there. I took a look at the zookeeper.out log and saw this:

2016-09-28 18:07:41,381 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper/zookeeper-3.4.8/bin/../conf/zoo.cfg

2016-09-28 18:07:41,475 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.84.76.50 to address: /10.84.76.50

2016-09-28 18:07:41,475 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.84.77.184 to address: /10.84.77.184

2016-09-28 18:07:41,476 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.84.77.37 to address: /10.84.77.37

2016-09-28 18:07:41,476 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.84.77.19 to address: /10.84.77.19

2016-09-28 18:07:41,476 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.84.79.55 to address: /10.84.79.55

2016-09-28 18:07:41,476 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid config, exiting abnormally

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/zookeeper/zookeeper-3.4.8/bin/../conf/zoo.cfg

at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Caused by: java.lang.IllegalArgumentException: clientPort is not set

at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:249)

at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)

... 2 more

Invalid config, exiting abnormally

Prior to executing the API call this is my zoo.cfg:

#Auto-generated by Exhibitor - Wed Sep 28 18:04:09 UTC 2016

#Wed Sep 28 18:04:09 UTC 2016

server.1=10.84.77.184\:2888\:3888

initLimit=10

syncLimit=5

clientPort=2181

server.17=10.84.76.50\:2888\:3888

server.16=10.84.77.19\:2888\:3888

tickTime=2000

server.14=10.84.79.55\:2888\:3888

server.9=10.84.77.37\:2888\:3888

dataDir=/var/lib/zookeeper

dataLogDir=/tmp/zookeeper

After executing the API call this is the file:

#Auto-generated by Exhibitor - Wed Sep 28 18:07:40 UTC 2016 #Wed Sep 28 18:07:40 UTC 2016 server.3=10.84.76.50\:0\:0 server.1=10.84.77.184\:0\:0 initLimit=10 syncLimit=5 clientPort=0 tickTime=2000 server.9=10.84.77.37\:0\:0 server.8=10.84.77.19\:0\:0 dataDir=/var/lib/zookeeper dataLogDir=/tmp/zookeeper server.5=10.84.79.55\:0\:0

What I was doing was taking the json response from get-state and using that to pass back to set-rolling. This is my call:

curl http://localhost:8080/exhibitor/v1/config/get-state {"version":"v1.5.5","running":true,"backupActive":false,"standaloneMode":false,"extraHeadingText":null,"nodeMutationsAllowed":true,"config":{"rollInProgress":false,"rollStatus":"n/a","rollPercentDone":0,"hostname":"10.84.79.55","serverId":14,"logIndexDirectory":"/tmp/zookeeper_log_indexes","zookeeperInstallDirectory":"/opt/zookeeper/*","zookeeperDataDirectory":"/var/lib/zookeeper","zookeeperLogDirectory":"/tmp/zookeeper","serversSpec":"1:10.84.77.184,9:10.84.77.37,14:10.84.79.55,15:10.84.76.50,16:10.84.77.19","backupExtra":"","zooCfgExtra":{"tickTime":"2000","initLimit":"10","syncLimit":"5"},"javaEnvironment":"","log4jProperties":"","clientPort":2181,"connectPort":2888,"electionPort":3888,"checkMs":30000,"cleanupPeriodMs":300000,"cleanupMaxFiles":20,"backupMaxStoreMs":0,"backupPeriodMs":0,"autoManageInstances":1,"autoManageInstancesSettlingPeriodMs":0,"observerThreshold":0,"autoManageInstancesFixedEnsembleSize":0,"autoManageInstancesApplyAllAtOnce":1,"controlPanel":{}}}

I don't see anything in the documentation that serversSpec requires the ports.

[Dann S Washko]

Okay after doing some more testing on this it appears what is happening is when I pass the config json for some reason the values for clientPort, connectPort, and electionPort are being set to 0.

I also see that in my initial post I did not post the right command. Here is the POST to set-rolling I have been using:

curl -H "Content-Type: application/json" -X POST  http://localhost:8080/exhibitor/v1/config/set-rolling -d '{"clientPort":2181,"connectPort":2888,"electionPort":3888,"rollInProgress":false,"rollStatus":"n/a","rollPercentDone":0,"hostname":"10.84.79.55","serverId":5,"logIndexDirectory":"/tmp/zookeeper_log_indexes","zookeeperInstallDirectory":"/opt/zookeeper/zookeeper-3.4.8","zookeeperDataDirectory":"/var/lib/zookeeper","zookeeperLogDirectory":"/tmp/zookeeper","serversSpec":"1:10.84.79.90,3:10.84.79.38,4:10.84.77.89,6:10.84.79.51,7:10.84.79.138","backupExtra":"","zooCfgExtra":{"tickTime":"2000","initLimit":"10","syncLimit":"5"},"javaEnvironment":"","log4jProperties":"","checkMs":40000,"cleanupPeriodMs":300000,"cleanupMaxFiles":20,"backupMaxStoreMs":0,"backupPeriodMs":0,"autoManageInstances":1,"autoManageInstancesSettlingPeriodMs":0,"observerThreshold":0,"autoManageInstancesFixedEnsembleSize":0,"autoManageInstancesApplyAllAtOnce":1,"controlPanel":{}}'

On the first server that is trying to do a rolling restart this is the output of get-state:

curl http://localhost:8080/exhibitor/v1/config/get-state

{"version":"v1.5.5","running":false,"backupActive":false,"standaloneMode":false,"extraHeadingText":null,"nodeMutationsAllowed":true,"config":{"rollInProgress":true,"rollStatus":"Applying to \"10.84.77.89\" (next will be \"10.84.79.138\")","rollPercentDone":1,"hostname":"10.84.77.89","serverId":4,"logIndexDirectory":"/tmp/zookeeper_log_indexes","zookeeperInstallDirectory":"/opt/zookeeper/zookeeper-3.4.8","zookeeperDataDirectory":"/var/lib/zookeeper","zookeeperLogDirectory":"/tmp/zookeeper","serversSpec":"1:10.84.79.90,3:10.84.79.38,4:10.84.77.89,6:10.84.79.51,7:10.84.79.138","backupExtra":"","zooCfgExtra":{"tickTime":"2000","initLimit":"10","syncLimit":"5"},"javaEnvironment":"","log4jProperties":"","clientPort":0,"connectPort":0,"electionPort":0,"checkMs":0,"cleanupPeriodMs":0,"cleanupMaxFiles":0,"backupMaxStoreMs":0,"backupPeriodMs":0,"autoManageInstances":0,"autoManageInstancesSettlingPeriodMs":0,"observerThreshold":0,"autoManageInstancesFixedEnsembleSize":0,"autoManageInstancesApplyAllAtOnce":0,"controlPanel":{}}}

Here are the values of get-state before the attempted change:

curl http://localhost:8080/exhibitor/v1/config/get-state

{"version":"v1.5.5","running":true,"backupActive":false,"standaloneMode":false,"extraHeadingText":null,"nodeMutationsAllowed":true,"config":{"rollInProgress":false,"rollStatus":"n/a","rollPercentDone":0,"hostname":"10.84.77.89","serverId":8,"logIndexDirectory":"/tmp/zookeeper_log_indexes","zookeeperInstallDirectory":"/opt/zookeeper/*","zookeeperDataDirectory":"/var/lib/zookeeper","zookeeperLogDirectory":"/tmp/zookeeper","serversSpec":"1:10.84.79.90,3:10.84.79.38,6:10.84.79.51,7:10.84.79.138,8:10.84.77.89","backupExtra":"","zooCfgExtra":{"tickTime":"2000","initLimit":"10","syncLimit":"5"},"javaEnvironment":"","log4jProperties":"","clientPort":2181,"connectPort":2888,"electionPort":3888,"checkMs":30000,"cleanupPeriodMs":300000,"cleanupMaxFiles":20,"backupMaxStoreMs":0,"backupPeriodMs":0,"autoManageInstances":1,"autoManageInstancesSettlingPeriodMs":0,"observerThreshold":0,"autoManageInstancesFixedEnsembleSize":0,"autoManageInstancesApplyAllAtOnce":1,"controlPanel":{}}}

So I am beginning to thing somewhere in this mess there is a value that is corrupting other values. It seems like everything after zooCfgExtra is being set to 0.

[Dann S Washko]

I should have noted that those tests were with version 1.5.5 of Exhibitor. I have now moved to version 1.5.6. When I issue the set-rolling on 1.5.6 nothing happens. I see it say succeeded in the json report and in the web ui it states it is currently performing a rolling restart. It never goes beyond that point. I see nothing in the logs for exhibitor or zookeeper that anything is actually happening. I have to then issues a cancel and rollback.