alert labels with variables

2,054 views
Skip to first unread message

Meher Garda

unread,
Feb 17, 2017, 4:43:00 PM2/17/17
to Prometheus Users
HI

I have an alert 

IF (min((node_filesystem_size{device!=\"rootfs\"} - node_filesystem_free{device!=\"rootfs\"}) / node_filesystem_size{device!=\"rootfs\"})) *100 > 80

          FOR 10m

          LABELS { severity = "high" }

          ANNOTATIONS {

          summary = "High Disk Usage on {{ $labels.instance }}",

          description = "{{ $labels.instance }} has high Disk Usage for about 10 m (current value: {{ $value }}s)",

          }


Is it possible to use variable in the LABELS like so


LABELS { severity = "high", env = "{{ $labels.env }}" }

Brian Brazil

unread,
Feb 17, 2017, 5:00:15 PM2/17/17
to Meher Garda, Prometheus Users
You can, however that's redundant as all labels will be passed on by default.

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/caafeef3-24bf-421b-8c35-71be9f8b8a88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Meher Garda

unread,
Feb 17, 2017, 6:27:05 PM2/17/17
to Prometheus Users, meher...@gmail.com
Hi Brian

The reason I ask is because in the alertmanager we have routes which depend on combination of severity and environment labels. My understanding is that routing happens on the basis of the label key/value pairs in the alert definition.

Hence an alert with labels

LABELS { severity = "high", env = "{{ $labels.env }}" }

will get routed properly according to the following routing configuration.  is my understanding/expectation correct?

route:

 group_by: [job, cluster, alertname]

 group_wait: 30s

 group_interval: 2m

 repeat_interval: 3h

 # If an alert isn't caught by a route, send email.

 receiver: default


 routes:


  # Send severity=critical alerts to the pager.

  - match:

      severity: critical

      env: prod

    receiver: teamx-pager

  - match:

      severity: high

      env: prod

    receiver: teamx-email

  - match:

      severity: warning

      env: prod

    receiver: teamx-email

  - match:

      severity: critical

      env: qa

    receiver: teamx-pager

  - match:

      severity: high

      env: qa

    receiver: teamx-email

  - match:

      severity: warning

      env: prod

    receiver: teamx-email

  - match:

      severity: critical

      env: dev

    receiver: teamx-email

  - match:

      severity: high

      env: qa

    receiver: teamx-email

  - match:

      severity: warning

      env: prod

    receiver: teamx-email


inhibit_rules:

- source_match:

    severity: 'critical'

  target_match:

    severity: 'warning'

  # Apply inhibition if the alertname is the same.

  equal: ['alertname', 'cluster', 'job']


receivers:

- name: teamx-pager

  pagerduty_configs:

  - service_key: {{ pagerdutykey }}

    send_resolved: true

- name: devops-email

  email_configs:

  - to: 'teamx...@one.verizon.com'

    send_resolved: true

- name: default

  email_configs:

  - to: 'dev...@one.verizon.com'

    send_resolved: true


On Friday, February 17, 2017 at 2:00:15 PM UTC-8, Brian Brazil wrote:
On 17 February 2017 at 21:43, Meher Garda <meher...@gmail.com> wrote:
HI

I have an alert 

IF (min((node_filesystem_size{device!=\"rootfs\"} - node_filesystem_free{device!=\"rootfs\"}) / node_filesystem_size{device!=\"rootfs\"})) *100 > 80

          FOR 10m

          LABELS { severity = "high" }

          ANNOTATIONS {

          summary = "High Disk Usage on {{ $labels.instance }}",

          description = "{{ $labels.instance }} has high Disk Usage for about 10 m (current value: {{ $value }}s)",

          }


Is it possible to use variable in the LABELS like so


LABELS { severity = "high", env = "{{ $labels.env }}" }


You can, however that's redundant as all labels will be passed on by default.

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

Brian Brazil

unread,
Feb 17, 2017, 7:11:33 PM2/17/17
to Meher Garda, Prometheus Users
On 17 February 2017 at 23:27, Meher Garda <meher...@gmail.com> wrote:
Hi Brian

The reason I ask is because in the alertmanager we have routes which depend on combination of severity and environment labels. My understanding is that routing happens on the basis of the label key/value pairs in the alert definition.

Hence an alert with labels

LABELS { severity = "high", env = "{{ $labels.env }}" }

will get routed properly according to the following routing configuration.  is my understanding/expectation correct?

The LABELS are additional labels added to the alert before sending it to the alertmanager. If your alert expression doesn't already have an env label, the above won't add one.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f00d94ce-d456-4006-bbb2-a0c0ed79522d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Meher Garda

unread,
Feb 17, 2017, 7:56:34 PM2/17/17
to Prometheus Users, meher...@gmail.com
Ok. Thanx.



--
Reply all
Reply to author
Forward
0 new messages