New Consul discovery plugin for ElasticSearch

[Mike Tougeron]

Hi all, we (Lithium) just released a new plugin for ElasticSearch node discovery for Consul https://github.com/lithiumtech/elasticsearch-consul-discovery. It is based on grantr/elasticsearch-srv-discovery but instead of being based on SRV DNS lookups, which generally returns 3 IPs, it uses API calls to Consul instead. It also accepts tags as parameters so that you can get a more fine-grained result of servers in the service.

We made this new plugin because with our larger ElasticSearch clusters the # of nodes in the cluster are much larger than the number of IPs that DNS responded with. We want to find all of the healthy nodes on the initial lookup.

-Mike

[Brian Lalor]

Woah, sweet! How stable is this?

-- 
Brian Lalor
bla...@bravo5.org

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/ae1bd864-3eed-4cf3-98b0-485a300246b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Tougeron]

We've been using it in non-prod for ~2 weeks now and prod for a few days.

[Timothy Van Heest]

This is pretty awesome Mike.

Does your plugin do anything dynamically, or is the configuration (e.g. the discovered nodes) fixed on startup of the cluster?

We also use consul with elasticsearch, but up to this point we're been using unicast with a fixed list of 3 master nodes.

[Mike Tougeron]

It's a discovery plugin so it keeps the list of servers up-to-date based on the results of the Consul Service API call.

[Jigar Joshi]

This is also using uni cast

https://github.com/lithiumtech/elasticsearch-consul-discovery/blob/master/src/main/java/org/elasticsearch/discovery/consul/ConsulUnicastHostsProvider.java#L68

But instead of fixed set of masters it resolves it based on what is healthy to consul, It doesn't just resolve it during startup, It does it dynamically as MikeT pointed

On Wednesday, September 9, 2015 at 10:44:22 AM UTC-7, Timothy Van Heest wrote: