Exhibitor not starting zookeeper

[Doug Garstang]

Having had no luck getting exhibitor to work with a file, I switched to S3. However, when using S3 as the config source, exhibitor doesn't start zookeeper:

  

-jar /opt/exhibitor/1.5.5.jar \

  --configtype s3 \

  --hostname 172.41.9.72 \

  --port 8080 \

  --s3config com-foobar-test-cfg:zookeeper-cfg/exhibitor.properties \

  --s3region us-east-1 \

Running in debug mode I can verify it's getting the exhibitor.properties file from S3. When I was using file as the config it was starting zookeeper, but now it's not. What's different? Frustrated. :(

Doug.

[Don Laidlaw]

The file stored in S3, and the --defaultconfig file are very different. 

What I did was start the zookeeper ensemble using the --defaultconfig command line parameter as documented, then saved the properties to S3 by using the Config tab (turn editing on, hit Commit). I was then able to capture that file that got saved into S3 and use it as a model for my own template to be generated at install time.

Best regards,

-Don

[Doug Garstang]

Don,

Are you saying that --defaultconfig and --s3config don't point to the same content? Highly confused. Is this documented somewhere? What goes in each one? Am I supposed to put all the config except for the servers-spec into the defaultconfig and the servers-spec into the s3 config or something else altogether?

Doug.

--
You received this message because you are subscribed to a topic in the Google Groups "exhibitor-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/exhibitor-users/v98UNOY5L_I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to exhibitor-use...@googlegroups.com.
To post to this group, send email to exhibit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/a534bf10-aa22-42ed-95ca-5c8e15b05205%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jordan Zimmerman]

There is config for the Exhibitor application and the shared ZK config that is stored in S3 (or wherever). You’re probably confused on this. The shared ZK config was never meant to be exposed to users but folks have gotten creative with Chef etc. and play around with it (it’s just a Properties file). So, —defaultconfig is initial values for the Exhibitor app itself and is documented here (see the values at the bottom of the page):

https://github.com/Netflix/exhibitor/wiki/Running-Exhibitor

You received this message because you are subscribed to the Google Groups "exhibitor-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to exhibitor-use...@googlegroups.com.

To post to this group, send email to exhibit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/CAGM7N-RG_tmC3eE3Vq0Rtc1O56P2kySgFQrZvgS8U_uQGf7BGQ%40mail.gmail.com.

[Don Laidlaw]

I feel your pain. I went through the exact same frustration. To help you along, here is basically what I have for each file:

In S3:

#Default shared config generated at install time
com.netflix.exhibitor.java-environment=
com.netflix.exhibitor.zookeeper-data-directory=/var/zookeeper/data
com.netflix.exhibitor.servers-spec=INITIAL_ZOOKEEPER_SERVERS
com.netflix.exhibitor.cleanup-period-ms=43200000
com.netflix.exhibitor.auto-manage-instances-fixed-ensemble-size=INITIAL_ZOOKEEPER_ENSEMBLE_SIZE
com.netflix.exhibitor.zookeeper-install-directory=/opt/zookeeper
com.netflix.exhibitor.check-ms=30000
com.netflix.exhibitor.zookeeper-log-directory=/var/zookeeper/datalog
com.netflix.exhibitor.log-index-directory=
com.netflix.exhibitor.backup-period-ms=60000
com.netflix.exhibitor.client-port=2181
com.netflix.exhibitor.cleanup-max-files=3
com.netflix.exhibitor.backup-max-store-ms=86400000
com.netflix.exhibitor.connect-port=2888
com.netflix.exhibitor.backup-extra=
com.netflix.exhibitor.observer-threshold=999
com.netflix.exhibitor.log4j-properties=log4j.rootLogger\=INFO, ROLLINGFILE\nlog4j.appender.ROLLINGFILE\=org.apache.log4j.RollingFileAppender\nlog4j.appender.ROLLINGFILE.Threshold\=DEBUG\nlog4j.appender.ROLLINGFILE.File\=/var/log/zookeeper/zookeeper.log\nlog4j.appender.ROLLINGFILE.MaxFileSize\=10MB\nlog4j.appender.ROLLINGFILE.MaxBackupIndex\=3\nlog4j.appender.ROLLINGFILE.layout\=org.apache.log4j.PatternLayout\nlog4j.appender.ROLLINGFILE.layout.ConversionPattern\=%d{ISO8601} [myid\:%X{myid}] - %-5p [%t\:%C{1}@%L] - %m%n\n
com.netflix.exhibitor.auto-manage-instances-apply-all-at-once=0
com.netflix.exhibitor.election-port=3888
com.netflix.exhibitor.zoo-cfg-extra=initLimit\=10&syncLimit\=5&tickTime\=2000&autopurge.snapRetainCount\=10&autopurge.purgeInterval\=24
com.netflix.exhibitor.auto-manage-instances-settling-period-ms=180000
com.netflix.exhibitor.auto-manage-instances=1

Note that the above is a template where I replace the INITIAL_ZOOKEEPER_SERVERS and INITIAL_ZOOKEEPER_ENSEMBLE_SIZE strings with the actual values I want to use. You could parameterize more or less of that file as you need. 

This is the file I use on the --defaultconfig command line:

zookeeper-install-directory=/opt/zookeeper
zookeeper-data-directory=/var/zookeeper/data
zookeeper-log-directory=/var/zookeeper/datalog
auto-manage-instances=true

It has pretty much the minimum you need to get Exhibitor to read the other content from S3.

As you can see, these files are the same format (Java properties files) but with very different property names for exactly the same thing. And the S3 form does not seem to be documented.

Best regards,

-Don

[Doug Garstang]

Jordan,

Thanks for the clarification. Where can I find documentation of properties are relevant to each? The link you provided states the properties that are relevant to the defaultconfig file, (although it omits the com.netflix.exhibitor suffix which some say is required), but it doesn't say what properties are relevant for the s3config file.

Would a typical use be to put the servers-spec into the shared s3config, and the rest into the defaultconfig local file? Can I even do that? If I pass both to exhibitor, will it combine them bother together? That's not documented anywhere that I can see.

If I use the UI to write the servers back to the s3config file, will it write _ONLY_ the servers, or will it write everything that's also in the defaultconfig? I hope not, because if so that makes putting everything else into the local defaultconfig file.

Doug.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/69FFC9D2-428B-407C-A4F1-754CDE87EF12%40jordanzimmerman.com.

[Doug Garstang]

Thanks Don.

Ugh, so you have com.netflix.exhibitor prefixed in your s3config, but not in your defaultconfig. When I run exhibitor with the --file option, it writes an initial defaultconfig that contains properties from BOTH of your files, i.e. it combines them together. If they are supposed to be separate, why does exhibitor combine them like this? Or, again, does exhibitor not care? Does it combine the s3config and defaultconfig settings together?

Doug.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/53d71653-6c8c-477f-9122-30676b2e778d%40googlegroups.com.

[Jordan Zimmerman]

The defaultconfig stuff matches the UI on the Config panel. The properties file that is stored at S3 was never meant to be User accessible and is not currently documented. People have hacked solutions though. To fully document it I’d have to look through the code which is why I’ve never done it. The closest doc there is are the Javadoc comments for the enums that match the S3 properties. See:

https://github.com/Netflix/exhibitor/blob/master/exhibitor-core/src/main/java/com/netflix/exhibitor/core/config/IntConfigs.java

https://github.com/Netflix/exhibitor/blob/master/exhibitor-core/src/main/java/com/netflix/exhibitor/core/config/StringConfigs.java

Another thing to do is to run Exhibitor and get things as you like and then copy the properties file from S3. I know others have done this. There’s a bunch of noise in the file (for rolling config changes, etc.) but it shouldn’t matter.

-JZ 

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/CAGM7N-SOu%2BEhX1tfjWm_h4rfP-nCUxNwnfGNeOOgGbz_9cJhcQ%40mail.gmail.com.

[Don Laidlaw]

Yes, this is how I have done it. I basically got the ensemble running the way I wanted it, the from the Exhibitor console, on the Config tab, enabled Editing and hit the Commit button. This saved the file to S3. I downloaded that, and filtered out all the rolling config properties, and used the rest to construct my template.

I need a template because in our cloud operations all the installation is done hands off. Nobody will ever see the Exhibitor console. So there was no way to manually configure it, then push it to S3. I had to create the S3 file and put it there before the Zookeeper install and startup.

But if you have the opportunity to do hands-on manipulations of the console, then you do not ever need to concern yourself about the S3 file.

-Don

[Doug Garstang]

David,

With regards to:

"But if you have the opportunity to do hands-on manipulations of the console, then you do not ever need to concern yourself about the S3 file."

Isn't a benefit of putting the servers into S3, is that making the change on one of the node UI's, allows the config to be written back to S3, where the other instances can pick it up? This isn't documented but I know it's trying to write the file back to S3 as I get a permissions error after committing the change (even though my IAM policy has s3:* - trying to work out what's going on there...)

Doug.

--
You received this message because you are subscribed to a topic in the Google Groups "exhibitor-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/exhibitor-users/v98UNOY5L_I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to exhibitor-use...@googlegroups.com.
To post to this group, send email to exhibit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/1a29c133-cdae-4d01-9dd7-ee038f4b26e4%40googlegroups.com.

[Don Laidlaw]

I don't think the --defaultconfig file is required at all. I was told before that the S3 config file was basically overriding the defaultconfig file. But I did find that the first time that zookeeper starts up, if it does not have the paths populated (install dir, snapshot dir, transaction dir) then it will not even attempt to download the S3 file, it instead enters a latent state and stays there. So I provide those settings in the defaultconfig so that the server does not enter a latent state.

Again, I do most of this because I am running in a completely hands-off environment. Nobody has access to the Exhibitor console. If we run into a problem on a server we just terminate it and let Amazon's autoscaling create a replacement.

Best regards,

-Don

[Jordan Zimmerman]

BTW folks - I realize I haven’t given Exhibitor much support recently. PLEASE send PRs with doc updates/whatever. We can all pitch in to make it a better project.

-Joredan

--

You received this message because you are subscribed to the Google Groups "exhibitor-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to exhibitor-use...@googlegroups.com.

To post to this group, send email to exhibit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/a0628a68-fc65-4d17-a347-b15e368b92e5%40googlegroups.com.

[Doug Garstang]

So it seems like I have two possible ways to use exhibitor.

1. Put everything, including the servers, into the --defaultconfig file, and have chef push out a common file to all nodes. Updates to the nodes in the cluster would be done via chef updating the --defaultconfig file. This is the approach I started with, and had no success. I can try again.

2. Put most stuff into --defaultconfig (managed by chef), and put the servers into the --s3config file. Updates to nodes would be done by updating the list of servers on one of the nodes, where exhibitor would push it back to S3, and the other instances would pick it up.

Does this sound accurate?

Doug.

--
You received this message because you are subscribed to a topic in the Google Groups "exhibitor-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/exhibitor-users/v98UNOY5L_I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to exhibitor-use...@googlegroups.com.
To post to this group, send email to exhibit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/exhibitor-users/a0628a68-fc65-4d17-a347-b15e368b92e5%40googlegroups.com.

[Doug Garstang]

Now I'm having a new issue.

I'm using:

--s3config com-slicetest-test-cfg:zookeeper-cfg/s3config.properties

It looks like from the debug output that exhibitor is trying to put the lock files in the root directory of the S3 bucket, (yes I know there are no directories in S3), but it seems to be ignoring the zookeeper-cfg at the front of the key, and instead trying to put lock files in zookeeper-cfg/. :(

Doug.

[Doug Garstang]

This is what I get when exhibitor writes the config to S3. It has everything.... So confused about what's suppose to go where. :(

#Auto-generated by Exhibitor 172.41.9.72

#Tue Dec 01 22:28:04 UTC 2015

com.netflix.exhibitor-rolling-hostnames=

com.netflix.exhibitor-rolling.zookeeper-data-directory=/var/lib/zookeeper

com.netflix.exhibitor-rolling.servers-spec=1\:172.41.9.72,2\:172.41.10.49

com.netflix.exhibitor.java-environment=

com.netflix.exhibitor.zookeeper-data-directory=/var/lib/zookeeper

com.netflix.exhibitor-rolling-hostnames-index=0

com.netflix.exhibitor-rolling.java-environment=

com.netflix.exhibitor-rolling.observer-threshold=0

com.netflix.exhibitor.servers-spec=1\:172.41.9.72,2\:172.41.10.49

com.netflix.exhibitor.cleanup-period-ms=300000

com.netflix.exhibitor.auto-manage-instances-fixed-ensemble-size=0

com.netflix.exhibitor.zookeeper-install-directory=/opt/zookeeper/*

com.netflix.exhibitor.check-ms=30000

com.netflix.exhibitor.zookeeper-log-directory=

com.netflix.exhibitor-rolling.auto-manage-instances=1

com.netflix.exhibitor-rolling.cleanup-period-ms=300000

com.netflix.exhibitor-rolling.auto-manage-instances-settling-period-ms=0

com.netflix.exhibitor-rolling.check-ms=30000

com.netflix.exhibitor.log-index-directory=/tmp/zookeeper_log_indexes

com.netflix.exhibitor-rolling.log-index-directory=/tmp/zookeeper_log_indexes

com.netflix.exhibitor.backup-period-ms=0

com.netflix.exhibitor-rolling.connect-port=2888

com.netflix.exhibitor-rolling.election-port=3888

com.netflix.exhibitor-rolling.backup-extra=

com.netflix.exhibitor.client-port=2181

com.netflix.exhibitor-rolling.zoo-cfg-extra=tickTime\=2000&initLimit\=10&syncLimit\=5

com.netflix.exhibitor-rolling.zookeeper-install-directory=/opt/zookeeper/*

com.netflix.exhibitor.cleanup-max-files=20

com.netflix.exhibitor-rolling.auto-manage-instances-fixed-ensemble-size=0

com.netflix.exhibitor-rolling.backup-period-ms=0

com.netflix.exhibitor-rolling.client-port=2181

com.netflix.exhibitor.backup-max-store-ms=0

com.netflix.exhibitor-rolling.cleanup-max-files=20

com.netflix.exhibitor-rolling.backup-max-store-ms=0

com.netflix.exhibitor.connect-port=2888

com.netflix.exhibitor.backup-extra=

com.netflix.exhibitor.observer-threshold=0

com.netflix.exhibitor.log4j-properties=

com.netflix.exhibitor.auto-manage-instances-apply-all-at-once=1

com.netflix.exhibitor.election-port=3888

com.netflix.exhibitor-rolling.auto-manage-instances-apply-all-at-once=1

com.netflix.exhibitor.zoo-cfg-extra=tickTime\=2000&initLimit\=10&syncLimit\=5

com.netflix.exhibitor-rolling.zookeeper-log-directory=

com.netflix.exhibitor.auto-manage-instances-settling-period-ms=0

com.netflix.exhibitor-rolling.log4j-properties=

com.netflix.exhibitor.auto-manage-instances=1

Doug

[Don Laidlaw]

Hi Doug,

I can only show you what works for me, which I have done above. And that is the absolute minimal --defaultconfig file with everything in S3. The absolute minimum --defaultconfig file gets the exhibitor boot up to the point where it can download the config from S3, which is the real config it will use.

I have not tested to see if the parameters in the local --defaultconfig file get overridden by those in S3. There are so few of them that it has been no issue for me.

Best regards,

-Don