Alert manager failed to send webhook(sometimes) on my server

[Amol Nagotkar]

Hi all, 

i have my prometheus, alertmanager and webhook server working as expected.

but in few cases promethues is able to collect events (i saw this in graph as shown in image on 2nd june).

but it didnt fire the event. i checked prometheus , alertmanager and webhook logs too.

what could be possible solutions ?

[Brian Candler]

> i checked prometheus , alertmanager and webhook logs too.

And what did you see in those logs?

What does the complete alerting rule look like? (Not just the "expr:" but other things like "for:")

You need to identify:

- did prometheus identify an alert condition, and send it to alertmanager?

- did alertmanager receive the alert?

- did alertmanager send the webhook?

- was the webhook up and able to receive the notification?