Node labels in prometheus metrics

[Yuvraj Singh]

I am trying to get node labels, i.e.- nodegroup for all node level metrics. So for example I want this nodegroup label with- node_memory_MemAvailable_bytes this metric. I have tried multiple ways to get this but dont have any success with this one yet.

I am getting node labels in - kube_node_labels metric but when I try to combine this by using group_left I get many to many matching is not allowed as I try to use instance label which is common in both.

I just want to get `nodegroup` label with all my node level metrics as its not easy to do it by using instance label which only gives ip address. I see that there is an option to use ec2-sd-config but I am looking for a better approach which will be generic for all cloud providers. 

If you have read this far, Thanks a lot

I also tried by adding this config in prometheus scrape config:-

- job_name: 'kubernetes-nodes'
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)

Sharing output of both queries:- 
node_memory_MemAvailable_bytes
node_memory_MemAvailable_bytes{endpoint="metrics", instance="192.168.40.7:9100", job="node-exporter", namespace="monitoring", pod="prometheus-prometheus-node-exporter-tsh8z", service="prometheus-prometheus-node-exporter"}

kube_node_labels

kube_node_labels{endpoint="http", instance="192.168.43.233:8080", job="kube-state-metrics", label_beta_kubernetes_io_arch="amd64", label_beta_kubernetes_io_instance_type="t3a.xlarge", label_beta_kubernetes_io_os="linux", label_failure_domain_beta_kubernetes_io_region="ap-south-1", label_failure_domain_beta_kubernetes_io_zone="ap-south-1a", label_kubernetes_io_arch="amd64", label_kubernetes_io_os="linux", label_nodegroup="services", namespace="monitoring", node="ip-192-168-43-233.ap-south-1.compute.internal", pod="prometheus-kube-state-metrics-64b9c87fdf-kr5x3", service="prometheus-kube-state-metrics"}

[Alex Duzsardi]

if you have a label before applying the re-label config like __meta_kubernetes_node_label_nodegroup , you could do something like this in the job config

    relabel_configs:
      - source_labels: [__meta_kubernetes_node_label_nodegroup]
        target_label: nodegroup

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a802fcd0-c0e2-46fb-afb6-ffcd48cee972n%40googlegroups.com.

--

Alexandru Duzsardi,

DevOps Engineer

Skype: alexinno83

GPG/PGP Key: https://keybase.io/aduzsardi/pgp_keys.asc

GitLab: https://gitlab.com/aduzsardi

GitHub: https://github.com/aduzsardi

LinkedIn: https://www.linkedin.com/in/aduzsardi

E-mail: alex.d...@infinitpartners.com

InFinIT Partners,

Address: Str. Macinului Nr. 17, Cluj-Napoca, Romania 

Web:  www.infinitpartners.com