Best way to compare time series in Prometheus

[chuanjia xing]

I have aws ec2 CPUUtilization data in Prometheus. Let's say I have data for 100 ec2 instances for last 24 hours in a 5min granular. Now I want to find the top 10 ec2 instances with the highest CPUUtilization in past 24 hours. How should I compare them to select? 

I tried to use the topk() function, but that seems like returning more than 10 instances. My understanding is it will return the topk instances in each timestamp, which will result in way more than k instances;

I also tried to use ave_over_time function, after applying this function on each time series, the results are still bunch of time series and still I don't know how to compare them. 

So what are the options for me for this?

 Thanks

[Ben Kochie]

https://www.robustperception.io/graph-top-n-time-series-in-grafana

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ce826926-7a73-4a46-941e-c01e7625b1e9n%40googlegroups.com.

[Julien Pivotto]

On 02 Apr 13:15, Ben Kochie wrote:
> https://www.robustperception.io/graph-top-n-time-series-in-grafana


Note that now you can use the promql query:


node_scrape_collector_duration_seconds and topk(5, avg_over_time(node_scrape_collector_duration_seconds[${__range_s}s]) @ end()


https://prometheus.io/blog/2021/02/18/introducing-the-@-modifier/

> > <https://groups.google.com/d/msgid/prometheus-users/ce826926-7a73-4a46-941e-c01e7625b1e9n%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpTavAtWHmy6wAUU1cN2QnBCPNzAzYkAJ4hoiKvCRNepg%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[chuanjia xing]

Thank you both for the pointers! It's what I am looking for.