Efficient Alert rule for Specific and Common threshold for over 1000 instances ?

[swe...@gmail.com]

Hi, 

I tried to find proper soluion in this group but I still can't find possible one.

I am monitoring over 1000 AWS instances.

and I have 3 Alert rules for CPU, Memory and Disk.

There is common threshold for all instances something like,

CPU : 0.7, Memory : 0.7, Disk: 0.7

-> e.g. For Common (All instances)

- alert: CPU

  expr: CPU > 0.7

  ...

- alert: MEM

  expr: MEM > 0.7

  ...

Now, I need to add different threshold for specific instances which it want to use different threshold for CPU.

- alert: CPU

  expr: CPU{instance="AAA"} > 0.8

Simplest approach like below, But If there are more instances with specific case, file will be complexed.

# For specific one

- alert: CPU

  expr: CPU{instance="AAA"} > 0.8

- alert: CPU

  expr: CPU{instance="BBB"} > 0.9

# For common

- alert: CPU

  expr: CPU{instance!~"AAA|BBB} > 0.7

- alert: MEM

  expr: MEM > 0.7

In this case, Isn't there more efficient way to make rules for specific and common.

Thanks for your answer in advance.

[Brian Candler]

That exact issue is dealt with in this post:

https://www.robustperception.io/using-time-series-as-alert-thresholds

[Ben Kochie]

You might want to read up on modern alerting practices so you can avoid this problem entirely.

https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit

https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/

https://landing.google.com/sre/sre-book/chapters/practical-alerting/

http://www.brendangregg.com/usemethod.html

https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0cfd686c-106a-4941-b99b-7dc3dec7ba48n%40googlegroups.com.

[swe...@gmail.com]

thanks for your reply. It's a little bit understand but I will read it deeply.

2020년 9월 10일 목요일 오전 2시 26분 13초 UTC+9에 b.ca...@pobox.com님이 작성:

[swe...@gmail.com]

wow, It's helpful information for me. thanks for your reply!!

2020년 9월 10일 목요일 오후 4시 40분 2초 UTC+9에 sup...@gmail.com님이 작성:

[swe...@gmail.com]

Yes. I checked that post and this is absolutely what I want.

I can't understand 100% of promql, but It works what I want.

thanks a lot!!!

2020년 9월 10일 목요일 오전 2시 26분 13초 UTC+9에 b.ca...@pobox.com님이 작성:

That exact issue is dealt with in this post:

https://www.robustperception.io/using-time-series-as-alert-thresholds