Is it possible to scrape metrics from multiple ports using same service?

[Anoop]

Hi,

Hi,

I have a kubernetes pod running with 3 containers in it. I am exporting metrics to different port and path for each container. Is there any way to enable monitoring for three ports and paths using a single service?

prometheus.io/path: /perf-metrics
prometheus.io/scrape_port: '9288'

prometheus.io/path: /auth-metrics
prometheus.io/scrape_port: '9277'

prometheus.io/path: /app-metrics
prometheus.io/scrape_port: '9266'

Thanks,

[Anoop Mohan]

Hi,

Any suggestions?

Thanks,

Anoop

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ca7ecdde-3efa-40ab-a5f1-62613a5ec742n%40googlegroups.com.

[Brian Candler]

You didn't say anything about where this list of targets is coming from.

If it's static, then very simply:

  - job_name: blah
    static_configs:
      - targets:

          - 1.2.3.4:9288

        labels:

          __metrics_path__: /perf-metrics

      - targets:

          - 1.2.3.4:9277

        labels:

          __metrics_path__: /auth-metrics

      - targets:

          - 1.2.3.4:9266

        labels:

          __metrics_path__: /app-metrics

Or being a bit fancier with target relabelling, something like this (untested):

  - job_name: blah
    static_configs:
      - targets:

          - 1.2.3.4:9288/perf-metrics

          - 1.2.3.4:9277/auth-metrics

          - 1.2.3.4:9266/app-metrics

    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

      - source_labels: [__address__]
        regex: '([^/]+)(.*)'

        target_label: __metrics_path__

        replacement: '$2'

      - source_labels: [__address__]
        regex: '([^/]+)(.*)'

        target_label: __address__

        replacement: '$1'

If it's coming from some service discovery mechanism, then you can use target rewriting rules to replace the __address__ label with <addr>:<port> and __metrics_path__ with <path>, where <addr>, <port> and <path> come from some data returned by the service discovery mechanism. For example, with kubernetes SD you can access the label values and annotations on nodes/pods/services in __meta_XXX labels.

[Anoop Mohan]

Thanks Brian for answering my question.

I am using Kubernetes SD and I believe my prometheus configuration is discovering targets correctly. My current use case is to see whether we can add multiple ports and paths configurations under the annotation section in a service? 

I am running multiple containers in a single pod and using a single service for this. Each container exposes metrics to their own port and metric path (not in /metrics path). So, how to configure the annotation in that single service to monitor the ports and paths of multiple containers?

Thanks & Regards,

Anoop

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bf343662-e122-40a3-8840-3d44b10e8e57n%40googlegroups.com.

[Brian Candler]

> how to configure the annotation

I'd say that's a kubernetes question, not a prometheus question.

From the prometheus point of view: first you work out what exactly it is you're trying to scrape (pods, services etc).  Then you read in a list of those entities using kubernetes_sd.  Then you have some rewriting rules to convert the metadata/labels/annotations into __address__ and __metrics_path__.  That's it.

You say you have multiple containers in the same pod which need scraping separately.  I don't think Kubernetes exposes metadata at the container level, only the pod level.  Some options you could consider:

1. Create separate scrape jobs (one for /perf-metrics, one for /app-metrics, one for /auth-metrics). Each job uses kubernetes_sd to get a list of suitable pods to scrape.

Or:

2. Create a separate 'service' for each container, and then use kubernetes_sd to get a list of services to scrape.

Or:

3. Looking at the kubernetes_sd docs:

The endpointslice role discovers targets from existing endpointslices. For each endpoint address referenced in the endpointslice object one target is discovered. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

So maybe that will work too. That will just give you a list of port numbers, so you may need to use fixed rewriting rules to map these to the correct __metrics_path__.

[Anoop Mohan]

Thanks Brian for sharing the details 👍

Thanks & Regards,

Anoop

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/70a3faf2-859b-4706-92af-c2f75c5a4fbbn%40googlegroups.com.