Setting custom threshold for Disk Storage Space.

[yagyans...@gmail.com]

Hi. I am using Prometheus version 2.22.1 and I am using custom thresholds for many of my alerts using the recording rules. But for Disk Space Utilization I am getting stuck where I need to set a custom threshold only for a single mount point of a particular IP. For eg. My default disk space utilization threshold is 90%. On the IP x.x.x.x, there are 2 mounts - vol1 and vol2. I want to set the threshold for vol2 as 95%. To set the threshold to 95% for all the mount points of an IP is easy and can be done in the way I have done below but How do I approach setting the custom threshold for one of the mount point?

Sample of custom thresholding that I am doing for one of the other alerts - Number of Threads.

  - record: custom_critical
    expr: (up{job=~"node.*",instance="x.x.x.x:9100"}) + 9999

Alert Expression:
((node_processes_threads) > on(instance) group_left() (custom_critical or on(instance) count by (instance)(node_processes_threads) * 0 + 4000))

Thanks in advance!

[Stuart Clark]

A common way is to use timeseries to set the thresholds:
https://www.robustperception.io/using-time-series-as-alert-thresholds

Also, rather than simple % thresholds the predict_linear function can be
very useful - alerting instead when space is predicted to be exhausted
within a particular time period. See
https://www.robustperception.io/tag/predict_linear

--
Stuart Clark

[Yagyansh S. Kumar]

Hi Stuart,

A common way is to use timeseries to set the thresholds:
https://www.robustperception.io/using-time-series-as-alert-thresholds

>> I have already referred the link and is exactly what I have used in the example thresholding that I have shared. But how do I use this for a single mountpoint for a single IP is where I am getting stuck at.