Behavior of Prometheus wrt to alerts and alertmanager.

[akshay sharma]

Hi,

I've some queries wrt to alerts and alertmanager.

Queries as follows,

1)

After reading about prometheus behaviour, I understood that prometheus sends alerts notification/events to alertmanager and alertmanager will relay to channels, Do prometheus can send alerts event/notification to  any other entity like kafka or any application also?. If yes, please provide some information on how we can configure Prometheus to send alert notifications to other applications.

2)

In what format prometheus sends alert notifications/events to alertmanager? I mean in json or any other format? Can you please provide some examples of exact message events. 

3) 

What is the time interval of sending an event/notification to alertmanager? where all these configurations recite in prometheus.

Thanks,

Akshay

[Julius Volz]

On Mon, Feb 22, 2021 at 11:10 AM akshay sharma <sharmaa...@gmail.com> wrote:

Hi,

I've some queries wrt to alerts and alertmanager.

Queries as follows,

1)

After reading about prometheus behaviour, I understood that prometheus sends alerts notification/events to alertmanager and alertmanager will relay to channels, Do prometheus can send alerts event/notification to  any other entity like kafka or any application also?. If yes, please provide some information on how we can configure Prometheus to send alert notifications to other applications.

You could point Prometheus at anything that understands the Prometheus->Alertmanager protocol (documented at https://prometheus.io/docs/alerting/latest/clients/), but it's not recommended, as it is pretty low-level and sends *all* firing alerts periodically. Alertmanager then applies a lot of dimensional grouping and time aggregation logic to those alerts (even before routing) that you would have to replicate if you want to get from a continuous firing alert state to a notification event again.

So if you are more interested in notifications / events rather than current ongoing alert states, you probably want to do Prometheus->Alertmanager->YourSystem, and you can do that by having Alertmanager send notifications to a webhook receiver that you implement on the other end: https://prometheus.io/docs/alerting/latest/configuration/#webhook_config

2)

In what format prometheus sends alert notifications/events to alertmanager? I mean in json or any other format? Can you please provide some examples of exact message events. 

Also see https://prometheus.io/docs/alerting/latest/clients/ for that, it's JSON.

 

3) 

What is the time interval of sending an event/notification to alertmanager? where all these configurations recite in prometheus.

For any alerts that are currently firing in Prometheus, Prometheus sends those alerts every rule evaluation interval, so typically every 15s, 30s, 1m, or whatever your configured evaluation interval is.

Regards,

Julius

 

Thanks,

Akshay

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOrgXNK8tute9XN1FuqhkgZudhyZnM%2BJs9_A1HJer2dravz2mQ%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com