Question on Promethues alert rules

[kiran]

Hello all

In Promethues alert rules file,  I am trying to understand what variables are available to use:

In the screenshot below, I see $value , $labels, $labels.instance

How do I know what variables are available to access within labels? Are these the keys(instance, job etc.) associated with a metric?

annotations:
      summary: "Host out of disk space (instance {{ $labels.instance }})"
      description: "Disk is almost full (< 10% left)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

[Brian Candler]

On Tuesday, 1 September 2020 05:08:27 UTC+1, kiran wrote:

In Promethues alert rules file,  I am trying to understand what variables are available to use:

In the screenshot below, I see $value , $labels, $labels.instance

How do I know what variables are available to access within labels? Are these the keys(instance, job etc.) associated with a metric?

Yes, these are labels of the timeseries.

https://prometheus.io/docs/concepts/data_model/

The "expr" in an alert rule is a PromQL query which acts as a filter - filtering down the universe of timeseries in your prometheus server to a selected subset.  If the subset has more than zero results, then an alert is fired.

For example:

expr: up == 0

returns all timeseries whose metric name is "up" and the value is zero.  Each instance of "up" in the result set has its own bag of labels.