Production readiness of elasticsearch-zookeeper

[Aaron Dixon]

I want to run large ES clusters in EC2 and continue to get split-brains even after taking advice on increasing ec2 discovery ping timeouts and discovery.zen.minimum_master_nodes to N/2+1.

Is there a way to reliably prevent split-brains without using elasticsearch-zookeeper? What's the production readiness of elasticsearch-zookeeper?

[Igor Motov]

Hi Aaron,

Reoccurring split-brains within the same cluster is a symptom of cluster overload or severe network connectivity issues. These issues have to be addressed regardless of the discovery service that you are planning to use. No discovery service will be able to make your cluster stable if nodes in the cluster are running out of memory, going into long GC pauses or disconnect from the network. You need to start monitoring jvm heap, CPU and network on your cluster and figure out the root cause of these issues. That's the only reliable way to prevent split-brains without using elasticsearch-zookeeper. It's also the only way get to a stable and functional cluster with or without elasticsearch-zookeeper.

There are companies that are using elasticsearch-zookeeper in production, but it's definitely less field tested than zen discovery.  

Igor