prometheus target discovery for elastic search

[Johny]

I am setting up monitoring set up for Elastic Search using Prometheus. I am running into one hurdle and need some help in making a design decision.

I am planning to run metric exporter on all master-eligible nodes in Elastic Search (3) for resiliency and also because masters have low footprint. The exporter fetches stats from local process and exports metrics for Prometheus to scrape. 

ES requires a quorum of at least n/2 master eligible nodes to elect a master. This implies one or more master-eligible nodes may not be part of quorum hence export out of sync metrics. If I set up all exporters as targets in Prometheus, it will not be feasible to de-duplicate metrics in Prometheus (due to split-brain issue). 

So the question is on setting up targets for scraping in Prometheus. Ideally, I would like to set up the current active master as the target. However, I am unable to solve service discovery issue without setting up another service. ES exposes current active master as http://cluster:9200/_cat/master but it is not in format Prometheus expects. Also, there is no metric that shows current active master.

Does anyone have any experience dealing with such an issue?

thanks

[Sandip Bhattacharya]

On 10.09.21 21:01, Johny wrote:
> I am planning to run metric exporter on all master-eligible nodes in
> Elastic Search (3) for resiliency and also because masters have low
> footprint. The exporter fetches stats from local process and exports
> metrics for Prometheus to scrape.

If you are using the most popular elasticsearch_exporter, I believe
filtering your metrics using something like this for any ES metrics,
should give you the current master metrics.

{job="elasticsearch", es_master_node="true"}

HTH,
Sandip

[Johny]

This label is set to 'true' on all master-eligible nodes and it gets the metrics for just the master nodes, not for other nodes.