Question about max_over_time() function

50 views

Skip to first unread message

chuanjia xing

unread,

May 21, 2021, 8:50:39 PM5/21/21

to Prometheus Users

I am collecting cpu utilization data for ec2 instances using prometheus. For each data entry, it includes instance_id, and cluster_id and some other information. A cluster can have multiple instances, and the # of instances vary due to scaling up and down.

Now I want to create a dashboard for following:

Show the cpu utilization data for all cluster with # of instances >= 50, and with avg cpu utilization <= 10%.

To do this, I first need to select all the cluster with avg # of instances >= 50. I used following query to select these clusters:

count by (cluster_id)(avg_over_time($Metric{}[1200s])) >= 50

I am following this blog (https://www.robustperception.io/graph-top-n-time-series-in-grafana ) to do this selection.

When using the avg_over_time function above, I found it’s not clear to me what this function is actually doing. From its definition:

max_over_time(range-vector): the maximum value of all points in the specified interval.

So couple questions:

My understanding is that, my raw data is a time series, and after applying this max_over_time function, the result will be another time series, is this right?
Which interval to use? And its impact of the result?

I used 1200s in above query, instead of the $_range_s suggested in the above blog. I used 1200s because I noticed it can give me some reasonable result comparing to $_range_s. But again, I don’t know how this parameters affecting the selecting result. Anyone can help to understand?

What’s the time range this query apply on? Does it automatically apply to all the available date, for example, if my data is 1 year long, does it apply this query for all the data points in past year? Can I configure it to be like last 7 days?
Finally, is my above query correct? Will it give me all the cluster with # of instances >= 50? I want to verify this but I don’t find a good way to validate.