grouping in AlertManager vs OpsGenie

[Laurent C.]

Hi,

AlertManager generates some UUID that it sets in field "alias" in OpsGenie alerts. This allows OpsGenie to determine, for all notifications received from AlertManager, if an OpsGenie alert is already opened for and it will update it, or it is a new alert.

This is working great when 1 alert in AlertManager matches 1 alert in OpsGenie.

But now, I am trying to deal with group of alerts. AlertManager can group alerts, and then it will have an UUID for the group of alerts, send it to OpsGenie.

Great, but... My concern is that, on AlertManager, a group of alert can evolve, meaning some alerts that were grouped could stop firing, some others could be added to the group.

The problem is that, on OpsGenie side, the alert description will be the one set at the alert creation. The alert count will be increased, and the logs will report an update was received for this alert, but new "sources" will be lost.

You can then use "notes", so each time the group changes, a new will be added, but the note will contain everything, and not only what changed.

However, the OpsGenie API seems to allow updating alert message and description. Would it be possible to leverage those features in AlertManager ?

I think it would be better to have alert description always being consistent with the group of alerts on AlertManager side (AlertManager -> range .Alerts -> OpsGenie description).

As an alternative, would it be possible to have notes contain only what changed (alert that stopping firing, alerts that have been added to the group)?

Thanks for your help.

Laurent C. 

[Laurent C.]

It seems https://github.com/prometheus/alertmanager/issues/2493 matches my concern updating the OpsGenie alert message and description, and PR https://github.com/prometheus/alertmanager/pull/2519 is waiting. I'll follow.