How to get number of CPUs within Prometheus PromQL

[yasi...@gmail.com]

Hi,

  

  I want to generate the High Load Average Alert, Currently I have following Alert Configuration:

  rules:

  - alert: High Load

    expr: avg_over_time(node_load1[4m]) > 0.8

    labels:

      severity: Major

    annotations:

      description: '{{ $labels.instance }} of job {{ $labels.job }} is under high

        load.'

      summary: Instance {{ $labels.instance }} under high load

Problem is node_load1 is factor of number of CPUs, if there are 8 CPUs, the true load average is node_load1[4m]/8. How I can get the number of CPUs dynamically using PromQL?

Thanks,

Yasir.

[Ben Kochie]

Rather than look at load1, which is not CPU usage, but run queue length, I would look at the real CPU utilization with these recording rules:

https://github.com/prometheus/node_exporter/blob/master/example-rules.yml

Then you can alert like this:

- alert: High Load
  expr: avg_over_time(instance:node_cpu_utilization:ratio[4m]) * 100 > 80

I added the * 100 so the ratio can be shown as a percent in the description with {{ $value | printf "%.2f" }}

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1873ab58-83d2-48e2-b267-31bdc997a1d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yasir Saleem]

Hi Ben,

  Thanks for the help. I did not understand the Grammar of this expression. This does not seems to be PromQL. Is there separate language or grammar for alerting rules expression? - Where I can get more help on this? - Also, how to test those expressions?

Thanks,

[Ben Kochie]

Yes, those are recording rules.

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9a081167-20d1-4c41-be70-4165a0f1236d%40googlegroups.com.

[Yasir Saleem]

I am confused about the syntax of these recording rules. This document says that 'expr:' is a PromQL expression, but when I test 'avg_over_time(instance:node_cpu_utilization:ratio[4m]) * 100 > 80' in the prometheus browser it does not work. Plus, I do not find any documentation that explains 'instance:node_cpu_utilization:ratio' this syntax of PromQL expression? - Where I can get help on that specific format/syntax of PromQL

Thanks.