ES+Zookeeper Behavior during client disconnects..

[Matt Wise]

Hey I just ran into this bug today with Zookeeper 3.4.6: https://issues.apache.org/jira/browse/ZOOKEEPER-885#comment-13749751

The short version is that a short blip of IO delay on one of our Zookeeper nodes disconnected several of our clients. Some of those clients were our ElasticSearch nodes. The nodes took longer than their session timeout to get re-connected, which triggered our ElasticSearch farm to start re-allocating shards across it. This re-allocation took up a whole ton of IO and triggered several alerts for awhile.

We obviously will tune our ES farm so that the reallocation is less impactful when it happens ... but can you implement some settings into the Zookeeper plugin so that we can increase the time the ES farm waits before re-allocating data everywhere? I'd love to be able to have it wait 5 minutes for a node to come back up before deciding that the ES farm needs to re-allocate all of the shards.

[Igor Motov]

That sounds like a good idea, but I think it should be implemented inside shard allocator not in the discovery, because zen discovery could benefit from it as well. I would suggest opening an issue on https://github.com/elasticsearch/elasticsearch requesting this feature.

[Matt Wise]

https://github.com/elasticsearch/elasticsearch/issues/3569

Matt Wise

Sr. Systems Architect

Nextdoor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch-zookeeper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch-zoo...@googlegroups.com.
To post to this group, send email to elasticsear...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch-zookeeper/1e353ecd-b7ed-45d1-b884-791be0047930%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.