Alerting with result of expression

[Kolja Krückmann]

Hi y'all

I'm looking for a possibility to add the "result" of an expression to the alerting description.

My expression is to alert when the c:\ Drive is below 10%. Now I want to add the actual value of the expression: 

((windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) * 100) <= 10

in the alerting mail. Is this somehow possible so that the free size is within the mail?

Kind regards

[Kolja Krückmann]

Small correction here:

I want to have the expression "windows_logical_disk_free_bytes{volume="C:"}/1000/1000/1000" (if this is the actual GB of free disk space (or do I need to device by 1024?)) in my alerting mail. And not as above the percentage.

[Brian Candler]

The "result" of the expression is available as {{ $value }}, and there are functions to convert this into a more human-readable value. See https://prometheus.io/docs/prometheus/latest/configuration/template_reference/

Examples:

expr: windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes

annotations:

  description: "Low free disk space: {{ $value | humanizePercentage }}"

expr: windows_logical_disk_free_bytes{volume="C:"} < 1000000000

annotations:

  description: "Low free disk space: {{ $value | humanize }}"    # or humanize1024: depends if you want Gigabytes or Gibibytes. https://en.wikipedia.org/wiki/Gigabyte

If you want to do the threshold based on percent, but report the absolute value, I would use something like this (untested):

expr: windows_logical_disk_free_bytes and (windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes < 0.1)

annotations:

  description: "Low free disk space: {{ $value | humanize }}"

I believe it's also possible to embed a completely separate query in a template (to look up a separate value to include in the annotations), but I've never done it, and can't find any examples.

Aside: I find these sort of static alerts annoying. Sometimes a filesystem has 8% disk free space and that's a good and normal situation for it to be in. Therefore, either you're lost in a sea of unimportant repeating alerts, or you're jumping through hoops for setting separate static thresholds per filesystem.

https://groups.google.com/g/prometheus-users/c/wHLxUPtrb-A/m/idIcdJIrBgAJ

Another approach you could consider:

https://groups.google.com/g/prometheus-users/c/0ncUqLm0LhU/m/mAiwaADXAgAJ

This looks at how quickly the filesystem is filling up, and tells you how long before it expects to be full.

[Kolja Krückmann]

Thanks alot!

My question now is, that my current expressions only returns the percentage as value - can I just add another expression for that specific alert?

Currentliy my alerts for diskSpace (one at 10% as warning and one at 5% as crit):

  - alert: LowDiskSpace
    expr: ((windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) * 100) <= 10
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "10% C-Disk Space"
      description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch 10% frei ({{$value | humanize}})."
   
  - alert: LowDiskSpace
    expr: ((windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) * 100) <= 5
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "5% C-Disk Space"
      description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch 5% frei ({{$value | humanize}})."

I want the values (bold above) to get returned as gigabyte values -> so can I just add an expression which calculates the free space?

[Brian Candler]

Repeating what I said before, I would do it by adjusting the alerting expr to return the actual amount of free space in bytes:

  - alert: LowDiskSpace
    expr: windows_logical_disk_free_bytes and ((windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) * 100) <= 10

    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "10% C-Disk Space"
      description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch 10% frei ({{$value | humanize}})."
   
  - alert: LowDiskSpace

    expr: windows_logical_disk_free_bytes and ((windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) * 100) <= 5

    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "5% C-Disk Space"
      description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch 5% frei ({{$value | humanize}})."

Since the percentage is never returned as part of the query, I would also simplify the expression to remove the factor of 100.

    expr: windows_logical_disk_free_bytes and (windows_logical_disk_free_bytes{volume="C:"} / windows_logical_disk_size_bytes) < 0.1

Note: you can simply enter the entire "expr" into the PromQL browser in the Prometheus web interface to test it. If it returns one or more values, that means an alert condition is active. You'll also be able to see the values and labels returned by the alerting expression. I have tested with the following expression:

    node_filesystem_avail_bytes and node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.7

The "and" operator in PromQL is documented here: https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators