Best suitable Service Discovery for prometheus

46 views

Skip to first unread message

pratyush ranjan

unread,

Jun 22, 2020, 11:07:05 AM6/22/20

to Prometheus Users

I have three service discovery in my mind for Prometheus those are consul_sd_config, ec2_sd_config & file_sd_config. Which service discovery I should choose?

My monitoring architecture is like I have around 35-40 projects and each project has its separate Amazon VPC. Each project has multiple instances for elasticsearch , mongodb, redis, MySQL, applications etc.

My main prometheus.yml file is around 8000+ lines .

Stuart Clark

unread,

Jun 22, 2020, 11:44:06 AM6/22/20

to pratyush ranjan, Prometheus Users

All three are useful methods, and you may find using several is the right answer for you.

Ultimately it is about collecting the information about what to scrape from the sources of truth.

File is the simplest, with information either hard coded or generated by a configuration management tool. I find it a useful mechanism for blackbox scrapes as well as any services which aren't in AWS.

The AWS method is great for EC2 instances, using tags. Similarly the Consul method is great for things registered there - which to use really depends on how things are setup for you. You will often find that you might need to adjust your service registration in Consul or the tags associated with EC2 instances slightly to expose things correctly for Prometheus.

Where we use the EC2 method we have a tag per exporter (e.g. Node, push gateway, etc.) and then the scrape config matches based on those, as many boxes will need to be scraped on several ports.