Enriching Alert Metadata - Labels vs Annotations

[Brian Gottfried]

We're forwarding our alerts out to an external system that requires some additional metadata and I'm trying to understand the effect of providing this metadata as annotations vs labels on each alert.

Context: We're utilizing both custom alerts and the default alerts that come with Openshift. The wrinkle is that those Openshift default alerts are Operator-managed, so we can't modify them directly to add the metadata as annotations (which from what I've found is the correct way to add metadata that you want to be the same across all instances of an alert). However, we can override the Openshift Prometheus' settings to add the values via the external labels field. Will this cause any issues with alerts being grouped together oddly or not being sent?

[Julius Volz]

Hi Brian,

An alert is identified by its unique set of labels, so the labels really serve as an identification, and changing / adding / removing any label will semantically mean that it's a different alert than before (with all the effects that has on routing and grouping in Alertmanager). Annotations are non-identifying pieces of extra information on alerts that do not affect routing or grouping at all, so those are indeed likely what you want. What you want to do might work with just labels if your metadata has a stable relationship to existing alerts that wouldn't cause frequent changes to their label set identities (e.g. if it's just a few extra attributes that rarely change). What is the metadata you want to attach?

External labels are applied to every outgoing alert of a Prometheus server equally (not just for some alerts), is that what you want?

Regards,

Julius

On Fri, Jan 15, 2021 at 4:38 PM Brian Gottfried <bgot...@redhat.com> wrote:

We're forwarding our alerts out to an external system that requires some additional metadata and I'm trying to understand the effect of providing this metadata as annotations vs labels on each alert.

Context: We're utilizing both custom alerts and the default alerts that come with Openshift. The wrinkle is that those Openshift default alerts are Operator-managed, so we can't modify them directly to add the metadata as annotations (which from what I've found is the correct way to add metadata that you want to be the same across all instances of an alert). However, we can override the Openshift Prometheus' settings to add the values via the external labels field. Will this cause any issues with alerts being grouped together oddly or not being sent?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/14d57178-e2ff-4902-a270-655f4cb5e04fn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Brian Gottfried]

For this scenario, I think we should be fine. The metadata we're applying is stable (it shouldn't change over the lifetime of the alerts unless we need to reconfigure the entire Prometheus instance) and it applying to every alert is actually desired in this case, because we can't access the alerts any other way. If we could apply labels/annotations more granularly, we would do so, but since we can't directly modify the alerts themselves, the external labels config is our best option currently for doing any sort of enrichment of the metadata on the labels, so we'll take what we can get!