predict_linear - disk space alerts

[Pete Leese]

Hi there,

I have an alert setup using predict_linear.

# Predictive alert for disk filling within 4 hours
  - alert: DiskWillFillIn4Hours
    expr: predict_linear(wmi_logical_disk_free_bytes[1h], 4 * 3600) < 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} {{ $labels.volume }} storage running out"
      description: "{{ $labels.instance }} is predicated that the {{ $labels.volume }} will run out at the current rate in under 4 hours."

The alert has triggered because there was a dip in available storage, however has not continued to decrease. My problem is that the alert keeps on running, even though the disk space is no longer decreasing.

How can I improve the accuracy of this so that it does not alert if there is not a continued drop in available free storage?

[Ben Kochie]

There are a few options.

You can increase the `for` time, so that it takes more consistent prediction to avoid the alert.

You can combine it with a threshold, so that you only get the alert if the disk space is below a threshold (predict_linear() and (space_percent < 25)

You can increase the look-back time longer than 1 hour, so that a short dip in space doesn't predict a fullness event.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/973750cd-a969-4af0-8041-ccedbae8e749%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pete Leese]

Thanks Ben, I'll have a play, I think option 2 may be the better solution. but at least I have options.