Exhibitor Status Changing

137 views

Skip to first unread message

Nick Kitmitto

unread,

Jul 9, 2018, 7:40:48 PM7/9/18

to exhibitor-users

I'm currently working on reverse engineering a Zookeeper/Exhibitor implementation. We're utilizing a docker image (https://hub.docker.com/r/mbabineau/zookeeper-exhibitor/). Unfortunately, when I curl the exhibitor status (curl http://<ip>:8181/exhibitor/v1/cluster/status), nodes are routinely put in place and taken out. All could be marked as down, and the next minute marked as up. We have 5 nodes running, but never seen more than 4 in the Exhibitor list and Zookeeper isn't actually starting.

Looking at zkServer.sh and running status, I can see the service continually rebooting.









Jul  9 23:30:49 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Starting zookeeper ... STARTED [pool-3-thread-62]
Jul  9 23:31:17 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: serving [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: serving [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Server list has changed [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to stop instance [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to start/restart ZooKeeper [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Kill attempted result: 0 [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Process started via: /opt/zookeeper/bin/zkServer.sh [ActivityQueue-0]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: JMX enabled by default [pool-3-thread-62]
Jul  9 23:31:48 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Using config: /opt/zookeeper/bin/../conf/zoo.cfg [pool-3-thread-62]
Jul  9 23:31:49 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Starting zookeeper ... STARTED [pool-3-thread-63]
Jul  9 23:32:18 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: not serving [ActivityQueue-0]
Jul  9 23:32:27 <HOSTNAME> dhclient[2671]: XMT: Solicit on eth0, interval 117980ms.
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: serving [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Server list has changed [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to stop instance [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to start/restart ZooKeeper [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Kill attempted result: 0 [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Process started via: /opt/zookeeper/bin/zkServer.sh [ActivityQueue-0]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: JMX enabled by default [pool-3-thread-63]
Jul  9 23:32:48 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Using config: /opt/zookeeper/bin/../conf/zoo.cfg [pool-3-thread-63]
Jul  9 23:32:49 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Starting zookeeper ... STARTED [pool-3-thread-61]
Jul  9 23:34:18 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: not serving [ActivityQueue-0]
Jul  9 23:34:25 <HOSTNAME> dhclient[2671]: XMT: Solicit on eth0, interval 113790ms.
Jul  9 23:34:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper down/not-serving waiting 30005 of 40000 ms before restarting [ActivityQueue-0]
Jul  9 23:35:11 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Automatic Instance Management will change the server list: 32:10.181.19.157,33:10.181.20.177,38:10.181.18.24,40:10.181.18.167 ==> 32:10.181.19.157,33:10.181.20.177,38:10.181.18.24 [ActivityQueue-0]
Jul  9 23:35:11 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Could not initiate Automatic Instance Management config change. Another process is already making a config change. [ActivityQueue-0]
Jul  9 23:35:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: not serving [ActivityQueue-0]
Jul  9 23:35:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Server list has changed [ActivityQueue-0]
Jul  9 23:35:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to stop instance [ActivityQueue-0]
Jul  9 23:35:48 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Attempting to start/restart ZooKeeper [ActivityQueue-0]
Jul  9 23:35:49 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Kill attempted result: 0 [ActivityQueue-0]
Jul  9 23:35:49 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Process started via: /opt/zookeeper/bin/zkServer.sh [ActivityQueue-0]
Jul  9 23:35:49 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: JMX enabled by default [pool-3-thread-64]
Jul  9 23:35:49 <HOSTNAME> zk-exhibitor[4128]: ERROR com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Using config: /opt/zookeeper/bin/../conf/zoo.cfg [pool-3-thread-64]
Jul  9 23:35:50 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  ZooKeeper Server: Starting zookeeper ... STARTED [pool-3-thread-65]
Jul  9 23:36:18 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  State: serving [ActivityQueue-0]
Jul  9 23:36:19 <HOSTNAME> dhclient[2671]: XMT: Solicit on eth0, interval 120150ms.
Jul  9 23:36:59 <HOSTNAME> zk-exhibitor[4128]: INFO  com.netflix.exhibitor.core.activity.ActivityLog  Automatic Instance Management will change the server list: 32:x.x.x.157,33:x.x.x.177,38:x.x.x.24,39:x.x.x.167 ==> 32:x.x.x.157,33:x.x.x.177,38:x.x.x.24 [ActivityQueue-0]

Reply all

Reply to author

Forward

0 new messages