Alert Rules: How to properly use labels vs annotations

[rlbol...@gmail.com]

Hi,

I would like to get a better understanding of how to effective use labels and annotations for alerts.

As per the documentation on the alerts page, annotations are for "a set of informational labels that can be used to store longer additional information such as alert descriptions or runbook links."

Labels are described as being labels, but I don't see any discussion with regards to application/use beyond that.

For example, with a generalized alert named LowDiskSpace, is it better to describe things like originating region and cluster as labels, as annotations, or both?

groups:
- name: RabbitMQ_Alerts
rules:
- alert: LowDiskSpace
expr: <expression-goes-here>
for: 10m
labels:
region: {$region} ??
cluster: {$cluster} ??
severity: warn
annotations:
region: {$region} ??
cluster: {$cluster} ??
summary: Low Disk Space
description: Low disk space was detected on {{ $labels.name }}

Here are what I would imagine some queries might look like:

ALERTS{severity="warning"} // Get all alerts with severity of warning

ALERTS{region="asia-pacific", role="rabbitmq"} // Get all alerts in asia-pacific for rabbitmq roles

Are these conceptually correct? Where would annotations fit into this?

[Brian Brazil]

Labels are for routing and grouping alerts, if you just want something to appear in the output of a notification it should be an annotation.

Usually labels like region and cluster would be external_labels rather than on every single alert. There's also no point duplicating labels into annotations, as both are available to notification templating. 

--

Brian Brazil

www.robustperception.io

[rlbol...@gmail.com]

If region and cluster are defined in external_labels, rather than piecemeal on each alert rule, will those be used for routing and grouping as well, or are they handled differently?

[Brian Brazil]

To the alertmanager it's just more labels, so they can be used for routing and grouping. 

--

Brian Brazil

www.robustperception.io

[rlbol...@gmail.com]

Thanks Brian, that clears up my understanding.