Alertmanager not sending alert notification to opsgenie

326 views
Skip to first unread message

Abhi Rathod

unread,
Apr 6, 2022, 8:40:55 AM4/6/22
to Prometheus Users
Hi,
We are running 2 node alertmanager cluster. Its working fine in all the cases except for couple of alerts and instead I see this in the logs.

Apr 04 16:00:51 prom0012 alertmanager[231282]: level=error ts=2022-04-``` ```04T20:00:51.937Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="techops/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 422: {\"message\":\"Request body is not processable. Please check the errors.\",\"errors\":{\"message\":\"Message can not be empty.\"},\"took\":0.001,\"requestId\":\"0ae8692e-9d18-4ea1-855d-c6d4d6a3bde7\"}"```

expr: sum by(instance, device) (increase(node_network_receive_errs_total{device=~"eth"}[1m])) > 0
for: 0s

Evaluation interval on prometheus is set to 2s
Group interval on alert manager is set to 0s.

I dont see any errors on Opsgenie. 



Reply all
Reply to author
Forward
0 new messages