Prometheus AlertManager not clearing alerts in AlertManager

[Prathap Dasari]

Hi folks ,

Prometheus AlertManager not clearing alerts in AlertManager even though The services are up and running

Is there any way to delete the data at alert manager side ?

Kr,

Prathap 

[Brian Candler]

You'll need to be more specific with your description of the problem. What do you mean by "clearing" alerts, and what do you want to "delete"?  What specifically do you see or not see, that is different to what you expect or want?

Can you describe exactly what you see:

- in the Prometheus web interface

- in the Alertmanager web interface

- in your notifications

When an alert stops firing in Prometheus, it will stop sending it to Alertmanager, and Alertmanager will expire it.  If the alert receiver has "send_resolved: true" then alertmanager will send an extra "resolved" notification at this point  (although it may contain a mixture of alerts which are still firing and which have been resolved, if they are in the same alert group; and in any case, sending alert resolved notifications is strongly discouraged).

Therefore, you need to first identify: what is the user-visible problem you're trying to debug?

- Is it that Prometheus keeps generating an active alert, even when you think the condition has gone away?

- Is it that Alertmanager still shows the alert as active, even though prometheus is no longer sending it?

- Is it that Alertmanager isn't sending a "resolved" notification when you think it should?

- Something else?

Then you can look into your configurations to see why it's not working in the way you expect; and share the relevant configs. That is: attack the problem is a stepwise, logical manner, and be explicit about what you actually see, not your guesses as to the the underlying cause.

It's always worth stating what versions of prometheus and alertmanager you are running - and also checking that your clocks are in sync, if these are running on different hosts.