Multiple metrics path for Prometheus

[rakshit gupta]

Hi, 

How can (is it possible) I use multiple endpoints for metrics scraping in a job ?

# The HTTP resource path on which to fetch metrics from targets.

 [ metrics_path: <path> | default = /metrics ]  

In my use case, a service( registered to consul) can expose metrics in any one of the endpoints - /metrics, /actuator/info, etc, and I need generic configuration in Prometheus to scrape data by probably iterating through the endpoints. I cannot hard code the endpoint there that /metrics is my endpoint.

Thanks for your help in advance!

[Christian Hoffmann]

Hi,

On 12/9/20 12:42 PM, rakshit gupta wrote:
> How can (is it possible) I use multiple endpoints for metrics scraping
> in a job ?
>
> # The HTTP resource path on which to fetch metrics from targets.
>  [ metrics_path: <path>

> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#path>

> | default = /metrics ]
>
> In my use case, a service( registered to consul) can expose metrics in
> any one of the endpoints - /metrics, /actuator/info, etc, and I need
> generic configuration in Prometheus to scrape data by probably iterating
> through the endpoints. I cannot hard code the endpoint there that
> /metrics is my endpoint.

I don't think this is possible or a good idea. You would need to decide
which metrics path to use. If you have this information (e.g. during
service discovery), then you can place those targets in different scrape
jobs or even re-use the sample scrape job and relabel __metrics_path__
accordingly.

If SD doesn't provide this information, another option would be to
provide it via your configuration management system (e.g. probe your
targets regularly and see what endpoint is the right one; then record
this information).

If this information is not available to you, then the only other option
would be to create scrape jobs for all variants. Two of those three
would always fail. I don' think it would be a good idea or simple to
work with.

Kind regards,
Christian

[Bjoern Rabenstein]

There is probably same nuance in arguing if and when this is a good
idea and when not.

But in fact, the famous
https://github.com/kubernetes/kube-state-metrics is doing it. It's not
using different paths, but different ports, but that's kind of
similar.

On the Prometheus side, however, you need separate scrape
targets. There is currently no way of "iterating" through multiple
ports or paths of a target. From the Prometheus side, a different
port, a different path, or a different host is just all the same thing
in defining a different target. (And that probably won't change
anytime soon.)

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Vitaly Shupak]

This is possible to do using relabel_configs:

scrape_configs:

  - job_name: 'job'

    static_configs:

        - targets:

            - 'example.com/a'

            - 'example.com/b'

            - 'example.com/c'

            - 'example.com/d'

    relabel_configs:

        - source_labels: [__address__]

          target_label: __metrics_path__

          regex: '(.*)/(.*)'

          replacement: '/$2'

        - source_labels: [__address__]

          target_label: instance

        - source_labels: [__address__]

          regex: '(.*)/(.*)'

          replacement: '$1'

          target_label: __address__

[Julius Volz]

On Fri, Jan 29, 2021 at 3:50 PM Vitaly Shupak <vitaly...@gmail.com> wrote:

This is possible to do using relabel_configs:

scrape_configs:

  - job_name: 'job'

    static_configs:

        - targets:

            - 'example.com/a'

            - 'example.com/b'

            - 'example.com/c'

            - 'example.com/d'

    relabel_configs:

        - source_labels: [__address__]

          target_label: __metrics_path__

          regex: '(.*)/(.*)'

          replacement: '/$2'

        - source_labels: [__address__]

          target_label: instance

        - source_labels: [__address__]

          regex: '(.*)/(.*)'

          replacement: '$1'

          target_label: __address__

This does not require relabeling, as you can just attach a "__metrics_path__" label directly to each target, e.g.:

scrape_configs:
  - job_name: 'job'
    static_configs:
        - targets:

            - 'example.com'

          labels:

            __metrics_path__: a

        - targets:
            - 'example.com'

          labels:

            __metrics_path__: b

        - targets:
            - 'example.com'

          labels:

            __metrics_path__: c

        - targets:
            - 'example.com'

          labels:

            __metrics_path__: b

I'd probably still just group them under different scrape configs though, as you will get different metrics from each of the endpoints, and thus you'll likely want to group them under a different job name (although you could also override the "job" label like above, in one scrape config).

 

On Monday, December 14, 2020 at 1:32:23 PM UTC-5 bjo...@rabenste.in wrote:

There is probably same nuance in arguing if and when this is a good
idea and when not.

But in fact, the famous
https://github.com/kubernetes/kube-state-metrics is doing it. It's not
using different paths, but different ports, but that's kind of
similar.

On the Prometheus side, however, you need separate scrape
targets. There is currently no way of "iterating" through multiple
ports or paths of a target. From the Prometheus side, a different
port, a different path, or a different host is just all the same thing
in defining a different target. (And that probably won't change
anytime soon.)

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/c604b921-828a-427b-8d10-471ad7ccdae9n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Julius Volz]

(I mean /a, /b, etc. of course, not a, b, ...)