Exhibitor deletes myid file

[Jason Kolb]

Hi, I'm attempting to use Exhibitor to manage my zookeeper install. I'd like to get to auto-discovery, but as a first step I'm just trying to get it to manage starting and restarting servers.

The problem I'm running into is that when Exhibitor starts zookeeper, it blows away the "myid" file, which then causes Zookeeper to not start (because it can't find it).

Is this known behavior... and if so, what is the purpose? I thought providing a myid file would circumvent the auto-discovery and simply let Exhibitor manage starting the service, but that doesn't seem to be exactly the case.

I'm running the servers on EC2 if that matters.

Thanks!

Jason

[Jordan Zimmerman]

Exhibitor manages the zoo.cfg, myid, etc. files for you. That's the raison d'etre of Exhibitor. 

-JZ

--
You received this message because you are subscribed to the Google Groups "exhibitor-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to exhibitor-use...@googlegroups.com.
To post to this group, send email to exhibit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/56262fad-8820-454e-b252-e72ce38a06d3%40googlegroups.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David Capwell]

I am currently testing how my apps handle ZK crashes on AWS, so I have an ASG with 3 nodes.  The test I just started running was "what happens when all 3 nodes die".  ASG happened and started to replace the nodes, but it seems the nodes coming up are not able to boot ZK anymore.  If I bring up a new ZK cluster or add a node to a cluster, ASG kicks in and exhibitor boots zk and the cluster adds or gets created.  

This is what I see in the logs:

02:47:14.117 39509 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - State: down

02:47:14.126 39518 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - Attempting to stop instance

02:47:14.126 39518 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - Attempting to start/restart ZooKeeper

02:47:14.412 39804 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - jps didn't find instance - assuming ZK is not running

02:47:14.421 39813 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - Starting in standalone mode

02:47:14.428 39820 [pool-5-thread-1] ERROR c.n.e.core.activity.ActivityLog - ZooKeeper Server: JMX enabled by default

02:47:14.430 39822 [ActivityQueue-0] INFO  c.n.e.core.activity.ActivityLog - Process started via: /opt/apache/zookeeper/zookeeper-3.4.5/bin/zkServer.sh

02:47:14.440 39832 [pool-5-thread-1] ERROR c.n.e.core.activity.ActivityLog - ZooKeeper Server: Using config: /opt/apache/zookeeper/zookeeper-3.4.5/bin/../conf/zoo.cfg

02:47:15.454 40846 [pool-5-thread-2] INFO  c.n.e.core.activity.ActivityLog - ZooKeeper Server: Starting zookeeper ... STARTED

[ec2-user@ip-10-144-80-140 jetty]$ ps -ef | grep java

root      2201     1  1 02:46 pts/0    00:00:00 /usr/bin/java -Djetty.logs=/mnt/logs/jetty -Djetty.home=/usr/share/jetty-8.1.5 -Djava.io.tmpdir=/mnt/tmp/jetty -jar /usr/share/jetty-8.1.5/start.jar --pre=etc/jetty-logging.xml

root      2215  2201 48 02:46 pts/0    00:00:26 /usr/java/jdk1.6.0_32/jre/bin/java -Xmx6g -Xms6g -XX:ErrorFile=/mnt/logs/jetty/java_error%p.log -XX:HeapDumpPath=/mnt/logs/jetty/ -XX:+UseConcMarkSweepGC -XX:ParallelCMSThreads=2 -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=80 -Djetty.home=/usr/share/jetty-8.1.5 -Djava.io.tmpdir=/mnt/tmp/jetty -Dcom.sun.management.jmxremote -cp /usr/share/jetty-8.1.5/lib/jetty-xml-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/servlet-api-3.0.jar:/usr/share/jetty-8.1.5/lib/jetty-http-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-continuation-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-server-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-security-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-servlet-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-webapp-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-deploy-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-servlets-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-annotations-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/annotations/javax.annotation-1.1.0.v201108011116.jar:/usr/share/jetty-8.1.5/lib/annotations/org.objectweb.asm-3.1.0.v200803061910.jar:/usr/share/jetty-8.1.5/lib/jdbc/derby/derby-10.6.1.0.jar:/usr/share/jetty-8.1.5/lib/jdbc/derby/derbytools-10.6.1.0.jar:/usr/share/jetty-8.1.5/lib/jetty-jmx-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jsp/com.sun.el-2.2.0.v201108011116.jar:/usr/share/jetty-8.1.5/lib/jsp/javax.el-2.2.0.v201108011116.jar:/usr/share/jetty-8.1.5/lib/jsp/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/usr/share/jetty-8.1.5/lib/jsp/javax.servlet.jsp-2.2.0.v201112011158.jar:/usr/share/jetty-8.1.5/lib/jsp/org.apache.jasper.glassfish-2.2.2.v201112011158.jar:/usr/share/jetty-8.1.5/lib/jsp/org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar:/usr/share/jetty-8.1.5/lib/jsp/org.eclipse.jdt.core-3.7.1.jar:/usr/share/jetty-8.1.5/lib/jta/javax.transaction-1.1.1.v201105210645.jar:/usr/share/jetty-8.1.5/lib/jta/atomikos/atomikos-util-3.7.0.jar:/usr/share/jetty-8.1.5/lib/jta/atomikos/transactions-3.7.0.jar:/usr/share/jetty-8.1.5/lib/jta/atomikos/transactions-api-3.7.0.jar:/usr/share/jetty-8.1.5/lib/jta/atomikos/transactions-jdbc-3.7.0.jar:/usr/share/jetty-8.1.5/lib/jta/atomikos/transactions-jta-3.7.0.jar:/usr/share/jetty-8.1.5/lib/logging/logback-access-1.0.12.jar:/usr/share/jetty-8.1.5/lib/logging/logback-classic-1.0.12.jar:/usr/share/jetty-8.1.5/lib/logging/logback-core-1.0.12.jar:/usr/share/jetty-8.1.5/lib/logging/slf4j-api-1.7.2.jar:/usr/share/jetty-8.1.5/lib/jetty-jndi-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-plus-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jndi/javax.activation-1.1.0.v201105071233.jar:/usr/share/jetty-8.1.5/lib/jndi/javax.mail.glassfish-1.4.1.v201005082020.jar:/usr/share/jetty-8.1.5/resources:/usr/share/jetty-8.1.5/lib/jetty-websocket-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-util-8.1.5.v20120716.jar:/usr/share/jetty-8.1.5/lib/jetty-io-8.1.5.v20120716.jar org.eclipse.jetty.xml.XmlConfiguration /mnt/tmp/jetty/start8233930833469659809.properties /usr/share/jetty-8.1.5/etc/jetty-logging.xml /usr/share/jetty-8.1.5/etc/jetty-jmx.xml /usr/share/jetty-8.1.5/etc/jetty.xml /usr/share/jetty-8.1.5/etc/jetty-hightide.xml /usr/share/jetty-8.1.5/etc/jetty-requestlog.xml /usr/share/jetty-8.1.5/etc/jetty-deploy.xml /usr/share/jetty-8.1.5/etc/jetty-webapps.xml /usr/share/jetty-8.1.5/etc/jetty-contexts.xml /usr/share/jetty-8.1.5/etc/jetty-testrealm.xml /usr/share/jetty-8.1.5/etc/jetty-jaas.xml

ec2-user  2314  1404  0 02:47 pts/0    00:00:00 grep java

[ec2-user@ip-10-144-80-140 jetty]$ sudo /opt/apache/zookeeper/zookeeper-3.4.5/bin/zkServer.sh start

JMX enabled by default

Using config: /opt/apache/zookeeper/zookeeper-3.4.5/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[ec2-user@ip-10-144-80-140 jetty]$ cat zookeeper.out 

2013-06-08 02:47:39,885 [myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /opt/apache/zookeeper/zookeeper-3.4.5/bin/../conf/zoo.cfg

2013-06-08 02:47:39,897 [myid:] - INFO  [main:QuorumPeerConfig@334] - Defaulting to majority quorums

2013-06-08 02:47:39,899 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid config, exiting abnormally

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/apache/zookeeper/zookeeper-3.4.5/bin/../conf/zoo.cfg

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:121)

        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)

        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Caused by: java.lang.IllegalArgumentException: /mnt/data/zookeeper/myid file is missing

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:344)

        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:117)

        ... 2 more

Invalid config, exiting abnormally

[David Capwell]

Ok, so it seems that that only happened to the first node that came up.  The second two are waiting on the 3 dead nodes.  I am assuming that the nodes will discover that those nodes are dead after a while and take over.

for right now i just did sudo init 0 to kill the node that is complaining about myid.

[Jordan Zimmerman]

That sounds right. So, everything's good.

-JZ

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/50ea99bb-94f3-4167-871b-b5ccee26afe7%40googlegroups.com?hl=en.

[David Capwell]

I deleted the first node and the new one to replace it joined the cluster. Is there a reason the first node saw this?

[Jordan Zimmerman]

There could be a race, I'm not sure. I see the log message you saw occasionally, but it corrects itself.

-JZ

On Jun 8, 2013, at 8:30 AM, David Capwell <dcap...@gmail.com> wrote:

> I deleted the first node and the new one to replace it joined the cluster. Is there a reason the first node saw this?
>

> --
> You received this message because you are subscribed to the Google Groups "exhibitor-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to exhibitor-use...@googlegroups.com.
> To post to this group, send email to exhibit...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/de40c3e2-925b-4a46-b5b9-d28a237ecce1%40googlegroups.com?hl=en.