auto discover of targets in Prometheus

[HJS]

For a particular job in Prometheus, it seems like the typical config is something like this:

static_configs:
- targets: ['localhost:9090']

But in the case where I want a dynamic list of hosts((auto discover of targets)), this option is available in prometheus and if available how can config??

Thank you in advance!

[Julius Volz]

Have a look at the Prometheus configuration documentation, which includes ~10 different service discovery mechanisms:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Here's an example on how to use the Consul SD one:

https://www.robustperception.io/finding-consul-services-to-monitor-with-prometheus

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/25b2c0b0-6435-404a-8c19-5bc9358dc77b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[HARSH AGARWAL]

You can use Prometheus-service-discovery(SD) to automatically discover the targets.
https://prometheus.io/docs/prometheus/latest/configuration/configuration

Using relabel-configs, you can specify which targets you want to keep/discard/change-labels, etc.

Prometheus supports SD for many providers. You can also use your own custom-service-discovery mechanisms using file_sd.

[Gajendra Ambi]

Everyone gives the same link https://prometheus.io/docs/prometheus/latest/configuration/configuration when asked about auto discovery of physical nodes but the entire page has only 2 occurences of the word auto-discovery. 

We have 700+ physical servers and Prometheus is running on k8s (targets are not part of k8s). Can someone please point to a blog or the actual part of the documentation which has auto discovery of physical nodes, not the same link please. I could not find anything there. It says, you can do it using 1 of the auto discovery methods but the link does not have a single method of auto discovery listed there.

[Julien Pivotto]

On 19 Feb 14:00, Gajendra Ambi wrote:
> Everyone gives the same link
> https://prometheus.io/docs/prometheus/latest/configuration/configuration when
> asked about auto discovery of physical nodes but the entire page has only 2
> occurences of the word auto-discovery.
> We have 700+ physical servers and Prometheus is running on k8s (targets are
> not part of k8s). Can someone please point to a blog or the actual part of
> the documentation which has auto discovery of physical nodes, not the same
> link please. I could not find anything there. It says, you can do it using
> 1 of the auto discovery methods but the link does not have a single method
> of auto discovery listed there.

Hello,

Prometheus does not do *auto* discovery. We do *service* discovery. You
will need a source of truth where your register your nodes.

>
> On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL wrote:
>
> > You can use Prometheus-service-discovery(SD) to automatically discover the
> > targets.
> > https://prometheus.io/docs/prometheus/latest/configuration/configuration
> >
> > Using relabel-configs

> > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E>,

> > you can specify which targets you want to keep/discard/change-labels, etc.
> >
> > Prometheus supports SD for many providers. You can also use your
> > own custom-service-discovery mechanisms using file_sd

> > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E>
> > .

> >
> >
> > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
> >>
> >> For a particular job in Prometheus, it seems like the typical config is
> >> something like this:
> >>
> >> static_configs:
> >> - targets: ['localhost:9090']
> >>
> >> But in the case where I want a dynamic list of hosts((auto discover of
> >> targets)), this option is available in prometheus and if available how can
> >> config??
> >>
> >>
> >> Thank you in advance!
> >>
> >>
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Gajendra Ambi]

Hi,

I too just discovered that it does not do dynamic discovery. I have to tell prometheus to see it by using file based discovery in prometheus. This is very unfortunate where we have 100s of nodes and growing and each group of devs who own them get to add them when they want. Currently it seems We have to do it manually. Thanks for the quick info.

[Stuart Clark]

On 19/02/2021 22:19, Gajendra Ambi wrote:
> Hi,
> I too just discovered that it does not do dynamic discovery. I have to
> tell prometheus to see it by using file based discovery in prometheus.
> This is very unfortunate where we have 100s of nodes and growing and
> each group of devs who own them get to add them when they want.
> Currently it seems We have to do it manually. Thanks for the quick info.
>

Prometheus needs some way of obtaining a list of the targets to scrape
for each job.

For cloud based instances that is fairly easy - Prometheus can ask them
to list all instances and using features like tags you can decide what
is scraped.

For virtual machines you could do something similar. For physical
machines it really depends on how you have things setup. Do you have a
IPAM/asset management system? Do you run a tool like Consul?

If you ignore Prometheus for a second: If you were asked to produce a
list of all the servers with details about what is running on them (so
you knew what metrics to fetch) how would you do that?

Whatever the answer, that's what you should aim to have Prometheus use.
If the answer is to produce a manual list, then unfortunately that's
what you need to do - Prometheus has to look somewhere for the list.

--
Stuart Clark

[Ben Kochie]

Prometheus absolutely does do dynamic discovery.

See the scrape_config section of the docs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

There are 16 available XXX_sd_configs options there. If your source of truth isn't listed, you can either write your own adapter using file_sd_configs as an API. Or if you think your system is popular enough, contribute it to the codebase.

There is a growing list of adapters here: https://prometheus.io/docs/operating/integrations/#file-service-discovery

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/be0fbf93-357f-403a-8f8f-b70140ead2den%40googlegroups.com.

[Gajendra D Ambi]

Hi Team,

Yes we do have a IPAM available. When we run prometheus node exporter agent it starts showing up metrics at ip/metrics. I was hoping for a way to give a range of ip addresses to prometheus, rather than adding them each as a static target. What would be the best way for us then! with 100s of baremetal servers, Also these baremetals are managed by openstack, I do see openstack auto discovery but will it also discover bare metal servers managed by openstack or just the VM instances? because prometheus wasn't explicit on that part in the documentation. If you could clarify in here or documentation then it would be awesome.

Thanks and Regards,

https://ambig.one/2/

Gajendra D Ambi |AWS-CSAA, VCAP6-DCA, 2xVCA, 7xVCP, VCE-CIA, EMC-ISA, Comptia Server+/A+, MCSE, 2xMCTS vExpert (2017/2018) |

[Stuart Clark]

On 20/02/2021 15:08, Gajendra D Ambi wrote:
> Hi Team,
> Yes we do have a IPAM available. When we run prometheus node exporter
> agent it starts showing up metrics at ip/metrics. I was hoping for a
> way to give a range of ip addresses to prometheus, rather than adding
> them each as a static target. What would be the best way for us then!
> with 100s of baremetal servers, Also these baremetals are managed by
> openstack, I do see openstack auto discovery but will it also discover
> bare metal servers managed by openstack or just the VM instances?
> because prometheus wasn't explicit on that part in the documentation.
> If you could clarify in here or documentation then it would be awesome.

The OpenStack SD can do both -
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config
- It has support for both hypervisor and instance for all Nova servers.

You can use the File SD
(https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config)
in conjunction with your IPAM to manage scrape configs too. A common
method is to run a regular scheduled job to interrogate the IPAM for
new, changed or removed devices and then produce JSON or YAML files for
your jobs - depending on the level of information you could produce
lists for the SNMP exporter (e.g. network equipment), node exporter
(Unix servers), MySQL exporter (and other databases), IPMI exporter, etc.

--
Stuart Clark

[Gajendra D Ambi]

the ipam sd sounds intriguing but i wonder if this is true with prometheus on k8s since the configmap is presented as a secret to prometheus.