Alerts - Returning useful values

[Pete Leese]

Hi everyone,

I have some alerts setup to trigger when free memory falls below a certain % - great that works fine.

(avg by (instance) (wmi_os_physical_memory_free_bytes / wmi_cs_physical_memory_bytes) * 100)  < 5

What I would like to do is get some usuful information in here to display the amount of free memory available in MB into my alert rule.

# Alert for high Memory utilisation > 20 minutes.

  - alert: High Memory Utilisation

    expr: (avg by (instance) (wmi_os_physical_memory_free_bytes / wmi_cs_physical_memory_bytes) * 100)  < 5

    for: 20m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Memory Utilisation High"

      description: "{{ $labels.instance }} memory utilisation has been running over 95% for 20 minutes"

Cheers

Pete 

[Christian Hoffmann]

On 07/16/2018 06:10 PM, 'Pete Leese' via Prometheus Users wrote:
> I have some alerts setup to trigger when free memory falls below a
> certain % - great that works fine.
>
> (avg by (instance) (wmi_os_physical_memory_free_bytes /
> wmi_cs_physical_memory_bytes) * 100)  < 5
>
> What I would like to do is get some usuful information in here to
> display the amount of free memory available in MB into my alert rule.

The actual percentage should be available via "{{ $value }}". If you
want some different metric (e.g. absolute numbers) which is not the
result of your "expr", you may need to perform another query. I haven't
done that myself, but it seems to be supported:

https://prometheus.io/docs/prometheus/latest/configuration/template_reference/#queries

https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#using-console-url-parameters

Kind regards,
Christian

[Julius Volz]

Yep, see an example here: https://gitlab.com/gitlab-com/runbooks/blob/4cf9fb8c42c37b2f060e04f81bb81a2bf06dbce6/alerts/postgresqls.yml#L188-194

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/eb851dc9-2e7e-6264-23d2-8d3219a46297%40hoffmann-christian.info.

For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

And an example for a query with multiple results: https://gitlab.com/gitlab-com/runbooks/blob/4cf9fb8c42c37b2f060e04f81bb81a2bf06dbce6/alerts/postgresqls.yml#L219-222

[Pete Leese]

<error expanding template: error parsing template __alert_High Memory Utilisation: template: __alert_High Memory Utilisation:1: malformed character constant: 'wmi_os_physical_memory_free_bytes{instance='>

Here is my alert;

      summary: "Instance {{ $labels.instance }} Memory Utilisation High"

      description: "{{ $labels.instance }} memory utilisation is running 95% for over 20 minutes [Current Value: {{query 'wmi_os_physical_memory_free_bytes{instance=\'$labels.instance\'}']"

What have I done wrong here? :) looks right compared to examples.

[Pete Leese]

Morning guys, any clues with this one? :)

[Matthias Rampke]

Looks like an issue quoting the single quotes. Try using double-backslashes or something? Or try without the outer double quotes (YAML _should_ be okay without them) and use double quotes in instance="$labels.instance"?

/MR

PS: Not everyone on this list is a guy, so you're more likely to get a quick response if you address everyone. I liked this explanation of how it's not a neutral term: https://www.vox.com/2015/6/11/8761227/you-guys-sexism-language

On Wed, Jul 18, 2018 at 9:28 AM 'Pete Leese' via Prometheus Users <promethe...@googlegroups.com> wrote:

Morning guys, any clues with this one? :)

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/997c3a46-ae21-4aff-8e2b-2910ebad1f24%40googlegroups.com.

[Pete Leese]

Tried that. Still not working

[Pete Leese]

Here's my query but still states 

 

 {{query 'wmi_os_physical_memory_free_bytes{instance=\"$labels.instance\"}'MB]"

 still producing the error:

<error expanding template: error parsing template __alert_High Memory Utilisation: template: __alert_High Memory Utilisation:1: malformed character constant: 'wmi_os_physical_memory_free_bytes{instance="$labels.instance"}'>

[Matthias Rampke]

You're not closing the templating tag before MB. add }}

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/df648dac-80a9-4691-ae67-2b9529d8dc78%40googlegroups.com.

[Pete Leese]

<error expanding template: error parsing template __alert_High Memory Utilisation: template: __alert_High Memory Utilisation:1: malformed character constant: 'query wmi_os_physical_memory_free_bytes{instance="$labels.instance"'>

 

 {{ 'query wmi_os_physical_memory_free_bytes{instance=\"$labels.instance\"'} }} MB]"

grrrr lol :) 

 

[Matthias Rampke]

wait, you're quoting the whole "function call" so it gets treated as a single string. Try

{{ query 'wmi_os_physical_memory_free_bytes{instance=\"$labels.instance\"'} }} MB]"

(note how the first single quote moved)

If that doesn't work, I've had better success and less quoting hell with this construction:

{{ printf `wmi_os_physical_memory_free_bytes{instance="%s"}` $labels.instance | query | first | value }}

(I *think* "| value" is necessary to get the actual value of the query result, and "| first" to make sure it doesn't blow up if for some reason there's two matches)

/MR

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e96558bc-3404-4b9d-b774-64aee868118a%40googlegroups.com.

[Pete Leese]

Ahha, a little closer now ! I now have a value being returned with your second query, albeit with a small tweak to escape the "

The value being returned is 3.65522944e+08

When I do a query the following is returned in Prometheus UI I get … 365522944

This should read 365MB

[Matthias Rampke]

Try adding `| humanize` to the pipeline? The available functions are documented here:

https://prometheus.io/docs/prometheus/latest/configuration/template_reference/

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/48e03fc7-5eac-4f23-94ab-8fb3dd2695ef%40googlegroups.com.

[Pete Leese]

Thanks for the pointers.

 I've had mixed results using humanize, how would one incorporate printf \"%.2f\" into the mix so that only the first 2 decimal places into the query? 

[Matthias Rampke]

That goes the same way: just attach it at the end of the pipeline ( … | value | printf `%.2f`)

/MR

On Mon, Jul 23, 2018 at 11:43 AM 'Pete Leese' via Prometheus Users <promethe...@googlegroups.com> wrote:

Thanks for the pointers.

 I've had mixed results using humanize, how would one incorporate printf \"%.2f\" into the mix so that only the first 2 decimal places into the query? 

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/47b0a99d-f6fc-4498-860b-a215f0077e3b%40googlegroups.com.

[Pete Leese]

rules file failed to load with printf '%.2f' 

I tried printf \"%.2f\"} and this returns the following  lol 

3665317888.00

[Pete Leese]

* 1e-09 on my query did the trick :)

Thanks everyone for the help with this :) been a great learning experience.