Oddities with Alert Humanize

[Pete Leese]

Hi all,

Having a few quirks with using Humanize in my alerts.

I am pulling back available free storage but in some instances it states the metric symbol and others it does not.

Here is my alert rule:

# Alert for Available Storage Warning

  - alert: Available Storage Warning

    expr: wmi_logical_disk_free_bytes{volume !~"HarddiskVolume.+"} * 1e-9 < 3

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} {{ $labels.volume }} has under 3GB available storage"

      description: "{{ $labels.instance }} volume {{ $labels.volume }} has {{ $value | humanize}} available storage"

When this triggers an alert  when the available storage

Server1.domain.local volume P: has 2.971 available storage

But when the storage dips into the 100's of MB's it displays the metric symbol in the alert.

Server2.domain.local volume P: has 551.6m available storage

So, my question is how do I consistently get the metric symbol to display ?

Cheers.

Pete 

[Brian Brazil]

It's up to you to put in the unit, so you need to add a B. 

--

Brian Brazil

www.robustperception.io

[Pete Leese]

That's a little odd then as it was adding its own unit for megabyte hence why I raised the question :) 

[Julius Volz]

So the "m" isn't the actual unit, it's a metric prefix: https://github.com/prometheus/prometheus/blob/67c13ba156a86afe6181f8d6e9fac1ec0f17c25f/template/template.go#L150-L174

In this case, m = "milli". You can see the humanize implementation here: https://github.com/prometheus/prometheus/blob/67c13ba156a86afe6181f8d6e9fac1ec0f17c25f/template/template.go#L150-L174

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3108f8df-2b86-4aeb-9d16-0e524e5130d8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

Sorry, the first link was supposed to go to https://en.wikipedia.org/wiki/Metric_prefix

On Tue, Jul 17, 2018 at 8:28 PM, Julius Volz <juliu...@gmail.com> wrote:

So the "m" isn't the actual unit, it's a metric prefix: https://github.com/prometheus/prometheus/blob/67c13ba156a86afe6181f8d6e9fac1ec0f17c25f/template/template.go#L150-L174

In this case, m = "milli". You can see the humanize implementation here: https://github.com/prometheus/prometheus/blob/67c13ba156a86afe6181f8d6e9fac1ec0f17c25f/template/template.go#L150-L174

[Pete Leese]

Just for my understanding, why would it be returning “m” as part of my alert value?

[Pete Leese]

Here what is being outputted in another example [Current Value: 766m% Free]

This should be outputting a %

# Alert for critical Memory utilisation > 5 minutes.
  - alert: Critical Memory Utilisation
    expr: (avg by (instance) (wmi_os_physical_memory_free_bytes / wmi_cs_physical_memory_bytes) * 100)  < 2
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} Memory Utilisation Critical"
      description: "{{ $labels.instance }} memory utilisation is running at a critical level of above 98% for over 5 minutes {{ $value | humanize}}% Free]"

 

{instance="SERVER.DOMAIN"

0.7173032601962498

[Alin Sînpălean]

In your first example, the value (as is returned by the alert expression) is in GB, where G stands for "giga" (i.e. 10^9) and B stands for bytes. What humanize does is convert a value that may be a few orders of magnitude above or below what a human can comfortably parse into a human readable value with a metric prefix appended. E.g. it would likely produce 1.234G for an input of 1234567890 and 23n for 0.000000023. Now it's your responsibility to append the unit to that, whether it's B (for bytes), m (for meters) or s (for seconds). The output would then look like 1.234GB or 23ns.

The issue with your first example is that you're already starting with a non-standard unit (GB instead of B), so in the best case what you'll end up with is 551.6mGB (milli-gigabytes) which is not very useful. If you insist on using humanize, then you have to change your alert expression to

  expr: wmi_logical_disk_free_bytes{volume !~"HarddiskVolume.+"} < 3 * 1e9

and your description to

  description: "{{ $labels.instance }} volume {{ $labels.volume }} has {{ $value | humanize -}}B available storage"

The issue with your second example is that you shouldn't use metric prefixes to begin with. There is no milli-percent or giga-percent, so you should simply output the value (maybe with a printf "%.2f" $value to limit the decimals).

Cheers,

Alin.

[Simon Pasquier]

IIUC you should use "{{ $value | printf "%.2f" }}% Free" and you'd get "0.78% Free".

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/812572a5-b3c0-4478-be26-e175d45050bc%40googlegroups.com.

[Pete Leese]

Thankyou - I think that makes sense, and now I understand humanize a bit better.

I'll experiment with printf ! thanks for the advice as always

[Pete Leese]

Almost there :)

      description: "{{ $labels.instance }} memory utilisation is running 95% for over 20 minutes [Current Value: {{ $value | printf "%.2f" }}% Free]"

For some reason running

Checking baseline.rules
  FAILED:
yaml: line 63: found character that cannot start any token

produces the above error? I literally cannot see anything wrong ?

[Simon Pasquier]

You need to escape the double quotes around %.2f: \"%.2f\"

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d9ca477b-50cb-45fa-b92a-91b53b4413c3%40googlegroups.com.