is it already a known issue with alertmanager integration to pagerduty

[ishu...@gmail.com]

Hallo,

I have an integration between alertmanager and pagerduty which has been since couple of years without any issue. 

On alertmanager, we are grouping by instance and when for the same instance, there are alerts with different severity (like critical and warning), the alertmanager is sending only one alert to pagerduty or may be pagerduty might also dropping it. 

Just wanted to know, how this integration works when alerts with the same instance name and two severity levels are triggered. 

Thanks

Eswar

[Julius Volz]

Hi Eswar,

If you are grouping only by "instance" in the Alertmanager, then all alerts with the same instance label in the same matched route will be grouped into a single notification. Usually you would have separate routes for different severities though, which would then also generate separate groups (even if both group only by instance). But I guess that's not the case in your case?

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/788f29e0-0f58-41cc-81c0-5cb80b375949n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[ishu...@gmail.com]

Hi,

Thanks for the reply.

I have only one route per environment and doesn't have separate routes per severity within environment. When, I have multiple alerts for the same route with same severity, all ended up in pagerduty as expected. But, when I have alerts triggered for the same instance with different severity, then I have only one (may be also the first alert) sent to pagerduty and the other alerts are although showing up on alertmanager but not reaching pagerduty.

Thanks

Eswar

[Brian Candler]

> When, I have multiple alerts for the same route with same severity, all ended up in pagerduty as expected

Yes, because these are separate alerts that are not grouped together.

> But, when I have alerts triggered for the same instance with different severity, then I have only one (may be also the first alert) sent to pagerduty

But you said you've told alertmanager to group on "instance", so that's what it will do: all alerts for the same instance (i.e. with the same value of the 'instance' label) will be delivered in a single message.  Whether the *body* of the pagerduty message shows those individual alerts, depends on how you've configured things.

You might want to show your alertmanager pagerduty_config, and also your alert grouping config.

[Stuart Clark]

On 26/09/2022 12:02, Brian Candler wrote:

> When, I have multiple alerts for the same route with same severity, all ended up in pagerduty as expected

Yes, because these are separate alerts that are not grouped together.

> But, when I have alerts triggered for the same instance with different severity, then I have only one (may be also the first alert) sent to pagerduty

But you said you've told alertmanager to group on "instance", so that's what it will do: all alerts for the same instance (i.e. with the same value of the 'instance' label) will be delivered in a single message.  Whether the *body* of the pagerduty message shows those individual alerts, depends on how you've configured things.

You might want to show your alertmanager pagerduty_config, and also your alert grouping config.

In general if you are sending alerts through to an external incident management system (PagerDuty, OpsGenie, JIRA, etc.) you are probably best not doing any grouping at all:

group_by: ['...']

-- 
Stuart Clark