tricky query for joining labels between 3 metrics but keep the left metric value

154 views
Skip to first unread message

hw17...@gmail.com

unread,
May 16, 2019, 10:49:48 AM5/16/19
to Prometheus Users
Hi All

I tried to get the IO latency from my elastic search pods in a kubernetes environment
Unfortunately, the labels to make the link between the node, the device and the pod are in 3 different metrics.

I have the node-exporter with the IO latency by device and instance
node_disk_io_time_seconds_total{device="xvdco",instance="ip-192-168-40-33.compute.internal"}

I have the instance on which my pod is running
container_memory_swap{instance="ip-192-168-40-33.compute.internal",pod="elasticsearch-data-0"}

finally I have the device from my elasticsearch pod (name=pod here)
elasticsearch_filesystem_io_stats_device_read_operations_count{device="xvdco",name="elasticsearch-data-0"}

I tried the following query
node_disk_io_time_seconds_total
+ on (instance) group_left(pod) container_memory_swap{pod="elasticsearch-data-0",container="POD"}
+ on (device) group_left(name) elasticsearch_filesystem_io_stats_device_read_operations_count{name="elasticsearch-data-0"}

the result is
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}   83759.864

the syntax is working, there is well only 1 result which is good BUT the value is wrong because of the + operator
All I want is to have the value from the left metric unchanged, ie only do the join on labels
like this
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}  14125.864

Is it possible with on group operand?
Or should I use a combination of label_join/label_replace instead?

Thanks


Brian Brazil

unread,
May 16, 2019, 10:54:20 AM5/16/19
to hw17...@gmail.com, Prometheus Users
On Thu, 16 May 2019 at 15:49, <hw17...@gmail.com> wrote:
Hi All

I tried to get the IO latency from my elastic search pods in a kubernetes environment
Unfortunately, the labels to make the link between the node, the device and the pod are in 3 different metrics.

I have the node-exporter with the IO latency by device and instance
node_disk_io_time_seconds_total{device="xvdco",instance="ip-192-168-40-33.compute.internal"}

I have the instance on which my pod is running
container_memory_swap{instance="ip-192-168-40-33.compute.internal",pod="elasticsearch-data-0"}

finally I have the device from my elasticsearch pod (name=pod here)
elasticsearch_filesystem_io_stats_device_read_operations_count{device="xvdco",name="elasticsearch-data-0"}

I tried the following query
node_disk_io_time_seconds_total
+ on (instance) group_left(pod) container_memory_swap{pod="elasticsearch-data-0",container="POD"}
+ on (device) group_left(name) elasticsearch_filesystem_io_stats_device_read_operations_count{name="elasticsearch-data-0"}

the result is
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}   83759.864

the syntax is working, there is well only 1 result which is good BUT the value is wrong because of the + operator

Multiply the 2nd two values by 0, then that will all work.

Brian
 
All I want is to have the value from the left metric unchanged, ie only do the join on labels
like this
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}  14125.864

Is it possible with on group operand?
Or should I use a combination of label_join/label_replace instead?

Thanks


--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e208130a-7914-4129-af4b-138ff3f78d51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

hw17...@gmail.com

unread,
May 16, 2019, 12:06:36 PM5/16/19
to Prometheus Users
Geat ! Many thanks Brian
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages