Re: [prometheus-users] Querying across metrics

[Simon Pasquier]

You need to use the label_replace() function with on()/group_left() clauses.

I can't give you the exact query but you can read more on this excellent and detailed blog post [1] from Fish.

[1] https://5pi.de/2017/11/09/use-prometheus-vector-matching-to-get-kubernetes-utilization-across-any-pod-label/

On Wed, Jun 13, 2018 at 8:49 PM, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:

Hello,

I am trying to work out how to get capacity / sum across a subset of information in Prometheus and ultimately Grafana with cube-state-metrics

I can get the total max number of pods in a kubernetes cluster, or even for a single node but I need to get this by certainly labels.

kube-state-metrics gives you this:-

kube_node_labels{ job="kube-state-metrics",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_kubernetes_io_hostname="host1.example.com", label_region="internet",label_tier="app",node=" host1.example.com "} 1

kube_node_labels{ job="kube-state-metrics",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_kubernetes_io_hostname="host2.example.com", label_region="intranet",label_tier="app",node=" host2.example.com "} 1

kube_node_labels{ job="kube-state-metrics",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_kubernetes_io_hostname="host3.example.com", label_region="intranet",label_tier="app",node=" host3.example.com "} 1

kube_node_labels{ job="kube-state-metrics",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_kubernetes_io_hostname="host4.example.com", label_region="intranet",label_tier="app",node=" host4.example.com "} 1

 

 

kube_node_status_capacity_pods{ job="kube-state-metrics",node="host1.example.com"}            29

kube_node_status_capacity_pods{ job="kube-state-metrics",node="host2.example.com"}            23

kube_node_status_capacity_pods{ job="kube-state-metrics",node="host3.example.com"}            24

kube_node_status_capacity_pods{ job="kube-state-metrics",node="host4.example.com"}            36

So what I need to do is get the capacity by a "label_region" so I need some magic between kube_node_labels and the "node", and to map this to the "node" in "kube_node_state_capacity_pods" and then add them all together.

I can create the variables in Grafana easily enough with "label_values" but I am having trouble getting the mapping between the labels and the nodes.

Any help would be much appreciated.

Thanks

Paul

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/23d94346-095c-4a49-a487-347d5ddab318%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Seymour]

On Thursday, 14 June 2018 09:19:59 UTC+1, Simon Pasquier wrote:

You need to use the label_replace() function with on()/group_left() clauses.

I can't give you the exact query but you can read more on this excellent and detailed blog post [1] from Fish.

[1] https://5pi.de/2017/11/09/use-prometheus-vector-matching-to-get-kubernetes-utilization-across-any-pod-label/

Thanks a lot.

I did it with this sort of thing.

sum(kube_pod_info * on (node) group_left(label_region) kube_node_labels{label_region=~"$region"})

Which seemed to do the trick.

Cheers

Paul