Changing the scrape path for kubernetes_sd_config

[Thomas Stringer]

I'm a little unsure how to alter the scrape path of a kubernetes_sd_config job with a role of node: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#node

The current relabel_configs look like this:

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}:4194/proxy/metrics

This is not the path I want to hit, though. I want to scrape `<node_name>:8080/metrics`.

As per the description on the docs for the node role of this sd config: "The node role discovers one target per cluster node with the address defaulting to the Kubelet's HTTP port. The target address defaults to the first existing address of the Kubernetes node object in the address type order of NodeInternalIP, NodeExternalIP, NodeLegacyHostIP, and NodeHostName."

Highlighted is a problem for me, I think. I want to use my custom port of 8080, not the kubelet port. I'm wondering if the following relabel configs would get me the metrics path of `<node_name>:8080/metrics`:

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - source_labels: [__address__]
            regex: (.+):(\d+)
            target_label: __address__
            replacement: ${1}:8080
          - source_labels: [__metrics_path__]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /metrics

Is that the right format for regex? Would that pull out the hostname into ${1} from `hostname:port` format? And the last one with the __metrics_path__, is that the nicest/easiest/right way to just set the metrics path?

Lastly, is there any way to see what Prometheus is doing with these relabels? In other words, how do I verify that when Prometheus is running, it is scraping from <node_name>:8080/metrics?

Thanks so much in advance!

[Simon Pasquier]

The regex for __address__ seems right to me. You don't even need to relabel __metrics_path since '/metrics' should be the default value.

 

Lastly, is there any way to see what Prometheus is doing with these relabels? In other words, how do I verify that when Prometheus is running, it is scraping from <node_name>:8080/metrics?

Check the Targets page in the Prometheus UI.

 

Thanks so much in advance!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3661ab60-208e-4ab2-a0f7-73f88be22535%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Stringer]

Perfect, that worked!! Thanks for the help!

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To post to this group, send email to promethe...@googlegroups.com.