tricky query for joining labels between 3 metrics but keep the left metric value

[hw17...@gmail.com]

Hi All

I tried to get the IO latency from my elastic search pods in a kubernetes environment
Unfortunately, the labels to make the link between the node, the device and the pod are in 3 different metrics.

I have the node-exporter with the IO latency by device and instance
node_disk_io_time_seconds_total{device="xvdco",instance="ip-192-168-40-33.compute.internal"}

I have the instance on which my pod is running
container_memory_swap{instance="ip-192-168-40-33.compute.internal",pod="elasticsearch-data-0"}

finally I have the device from my elasticsearch pod (name=pod here)
elasticsearch_filesystem_io_stats_device_read_operations_count{device="xvdco",name="elasticsearch-data-0"}

I tried the following query
node_disk_io_time_seconds_total
+ on (instance) group_left(pod) container_memory_swap{pod="elasticsearch-data-0",container="POD"}
+ on (device) group_left(name) elasticsearch_filesystem_io_stats_device_read_operations_count{name="elasticsearch-data-0"}

the result is
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}   83759.864

the syntax is working, there is well only 1 result which is good BUT the value is wrong because of the + operator
All I want is to have the value from the left metric unchanged, ie only do the join on labels
like this
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}  14125.864

Is it possible with on group operand?
Or should I use a combination of label_join/label_replace instead?

Thanks

[Brian Brazil]

On Thu, 16 May 2019 at 15:49, <hw17...@gmail.com> wrote:

Hi All

I tried to get the IO latency from my elastic search pods in a kubernetes environment
Unfortunately, the labels to make the link between the node, the device and the pod are in 3 different metrics.

I have the node-exporter with the IO latency by device and instance
node_disk_io_time_seconds_total{device="xvdco",instance="ip-192-168-40-33.compute.internal"}

I have the instance on which my pod is running
container_memory_swap{instance="ip-192-168-40-33.compute.internal",pod="elasticsearch-data-0"}

finally I have the device from my elasticsearch pod (name=pod here)
elasticsearch_filesystem_io_stats_device_read_operations_count{device="xvdco",name="elasticsearch-data-0"}

I tried the following query
node_disk_io_time_seconds_total
+ on (instance) group_left(pod) container_memory_swap{pod="elasticsearch-data-0",container="POD"}
+ on (device) group_left(name) elasticsearch_filesystem_io_stats_device_read_operations_count{name="elasticsearch-data-0"}

the result is
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}   83759.864

the syntax is working, there is well only 1 result which is good BUT the value is wrong because of the + operator

Multiply the 2nd two values by 0, then that will all work.

Brian

 

All I want is to have the value from the left metric unchanged, ie only do the join on labels
like this
{...,device="xvdco",instance="ip-192-168-40-33.compute.internal",name="elasticsearch-data-0",pod="elasticsearch-data-0",role="app"}  14125.864

Is it possible with on group operand?
Or should I use a combination of label_join/label_replace instead?

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e208130a-7914-4129-af4b-138ff3f78d51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[hw17...@gmail.com]

Geat ! Many thanks Brian

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e208130a-7914-4129-af4b-138ff3f78d51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io