A way to exclude incident_key from Alertmanager Payload

[Apollo L]

Hi,

Is there a way to exclude incident_key from the Alertmanager payload completely? I have one integration where I want to send repeat alerts even when the payload is exactly same, but the integration deduplicates events by default if the incident_key is same. If there is a way to exclude incident_key for certain alerts, this will solve a problem for us without requesting any changes to the integration.

[Apollo L]

Hi,

Is there a way to exclude incident_key from the Alertmanager payload completely? I have one integration where I want to send repeat alerts even when the payload is exactly same, but the integration deduplicates events by default if the incident_key is same. If there is a way to exclude incident_key for certain alerts, this will solve a problem for us without requesting any changes to the integration.

[Brian Candler]

You talk about "Alertmanager payload" and "the integration" without saying what this integration is (a webhook? something else?).

The string "incident_key" does not appear anywhere in Prometheus source code. It does appear in Alertmanager, but only in the context of Pagerduty. You didn't mention Pagerduty at all in your post.

If your issue *is* with alertmanager->pagerduty integration (which I'm just guessing), then the documentation for what you can configure is here:

https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config

and the logic for how incident_key is generated is in the source:

https://github.com/prometheus/alertmanager/blob/main/notify/pagerduty/pagerduty.go

It is hard-coded to key.Hash(), where key is an instance of notify.Key, which comes from notify.ExtractGroupKey

If you want to customise this, then I suggest you write your own webhook receiver, which accepts the (fixed) JSON webhook format, and then your code generates a new payload towards some other system, in whatever format you like.