will prometheus send the alert recovery report to the AM

[Yitao Jiang]

Hi, there

currently if the alert rules triggered, the AM will be notified. That's fine.

But if the alerts recovered, such as the nodes alive after a certain period time,

will it send the alerts to the AM and AM send the notifications to corresponding receivers ?

IMO, this feature is useful in operation, sometimes what you get alerts actually will auto recover and no impacts on the system, besides

i need to know whether the alerts handled.

[Fabian Reinartz]

Yes, Prometheus will notify that the alert is now resolved. We send these resolved alerts for 15 minutes after they disappeared.

The Alertmanager in return, will also send resolved notifications. This will have to configured for each respective receiver.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yitao Jiang]

Cool. so fthe 15mins, is it configurable ?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[Julius Volz]

No, it's hardcoded currently:

https://github.com/prometheus/prometheus/blob/master/rules/alerting.go#L143-L145

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fabian Reinartz]

To be clear. There's not much point in making it configurable. If Alertmanager works properly, it will take effect on the first try, so within the first few seconds of these 15 minutes. The 15 minutes is just in case your Alertmanager just went down and you want to bring it back up.

On Fri, Aug 5, 2016 at 6:38 PM Julius Volz <juliu...@gmail.com> wrote:

No, it's hardcoded currently:

https://github.com/prometheus/prometheus/blob/master/rules/alerting.go#L143-L145

On Fri, Aug 5, 2016 at 10:45 AM, Yitao Jiang <jiang...@gmail.com> wrote:

Cool. so fthe 15mins, is it configurable ?

On Friday, August 5, 2016 at 4:39:37 PM UTC+8, Fabian Reinartz wrote:

Yes, Prometheus will notify that the alert is now resolved. We send these resolved alerts for 15 minutes after they disappeared.

The Alertmanager in return, will also send resolved notifications. This will have to configured for each respective receiver.

On Fri, Aug 5, 2016 at 10:35 AM Yitao Jiang <jiang...@gmail.com> wrote:

Hi, there

currently if the alert rules triggered, the AM will be notified. That's fine.

But if the alerts recovered, such as the nodes alive after a certain period time,

will it send the alerts to the AM and AM send the notifications to corresponding receivers ?

IMO, this feature is useful in operation, sometimes what you get alerts actually will auto recover and no impacts on the system, besides

i need to know whether the alerts handled.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

[Yitao Jiang]

Ok, got it. But what if the Alertmanager is running ? will the promethus send recover notification to the AM? Per the code @julius sent, the alerts recovered within 15 minutes will be removed from the vector which in memory. won't send the recovery notification, am i right ?

If so, where the code to handle the recover notification

On Sat, Aug 6, 2016 at 12:54 AM, Fabian Reinartz <fab.re...@gmail.com> wrote:

To be clear. There's not much point in making it configurable. If Alertmanager works properly, it will take effect on the first try, so within the first few seconds of these 15 minutes. The 15 minutes is just in case your Alertmanager just went down and you want to bring it back up.

On Fri, Aug 5, 2016 at 6:38 PM Julius Volz <juliu...@gmail.com> wrote:

No, it's hardcoded currently:

https://github.com/prometheus/prometheus/blob/master/rules/alerting.go#L143-L145

On Fri, Aug 5, 2016 at 10:45 AM, Yitao Jiang <jiang...@gmail.com> wrote:

Cool. so fthe 15mins, is it configurable ?

On Friday, August 5, 2016 at 4:39:37 PM UTC+8, Fabian Reinartz wrote:

Yes, Prometheus will notify that the alert is now resolved. We send these resolved alerts for 15 minutes after they disappeared.

The Alertmanager in return, will also send resolved notifications. This will have to configured for each respective receiver.

On Fri, Aug 5, 2016 at 10:35 AM Yitao Jiang <jiang...@gmail.com> wrote:

Hi, there

currently if the alert rules triggered, the AM will be notified. That's fine.

But if the alerts recovered, such as the nodes alive after a certain period time,

will it send the alerts to the AM and AM send the notifications to corresponding receivers ?

IMO, this feature is useful in operation, sometimes what you get alerts actually will auto recover and no impacts on the system, besides

i need to know whether the alerts handled.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Yitao

[Julius Volz]

15 minutes before it is removed, the alert entry's ResolvedAt timestamp is already filled out: https://github.com/prometheus/prometheus/blob/master/rules/alerting.go#L246

It then still gets sent to AM during that period, but since AM now sees that it already has a resolved timestamp in the past, AM treats the alert as resolved.

On Sun, Aug 7, 2016 at 12:25 AM, Yitao Jiang <jiang...@gmail.com> wrote:

Ok, got it. But what if the Alertmanager is running ? will the promethus send recover notification to the AM? Per the code @julius sent, the alerts recovered within 15 minutes will be removed from the vector which in memory. won't send the recovery notification, am i right ?

If so, where the code to handle the recover notification

On Sat, Aug 6, 2016 at 12:54 AM, Fabian Reinartz <fab.re...@gmail.com> wrote:

To be clear. There's not much point in making it configurable. If Alertmanager works properly, it will take effect on the first try, so within the first few seconds of these 15 minutes. The 15 minutes is just in case your Alertmanager just went down and you want to bring it back up.

On Fri, Aug 5, 2016 at 6:38 PM Julius Volz <juliu...@gmail.com> wrote:

No, it's hardcoded currently:

https://github.com/prometheus/prometheus/blob/master/rules/alerting.go#L143-L145

On Fri, Aug 5, 2016 at 10:45 AM, Yitao Jiang <jiang...@gmail.com> wrote:

Cool. so fthe 15mins, is it configurable ?

On Friday, August 5, 2016 at 4:39:37 PM UTC+8, Fabian Reinartz wrote:

Yes, Prometheus will notify that the alert is now resolved. We send these resolved alerts for 15 minutes after they disappeared.

The Alertmanager in return, will also send resolved notifications. This will have to configured for each respective receiver.

On Fri, Aug 5, 2016 at 10:35 AM Yitao Jiang <jiang...@gmail.com> wrote:

Hi, there

currently if the alert rules triggered, the AM will be notified. That's fine.

But if the alerts recovered, such as the nodes alive after a certain period time,

will it send the alerts to the AM and AM send the notifications to corresponding receivers ?

IMO, this feature is useful in operation, sometimes what you get alerts actually will auto recover and no impacts on the system, besides

i need to know whether the alerts handled.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Yitao

[Yitao Jiang]

yes, the AM will treat the alert as a resolved one and wipe out it, is that right ?

From what i tested, the AM won't send the recover notification to the receiver even the node online within or after 15 minutes.BTW, i'm using node-exporter for alert testing

--

Regards,

Yitao

[Julius Volz]

On Sun, Aug 7, 2016 at 12:44 AM, Yitao Jiang <jiang...@gmail.com> wrote:

yes, the AM will treat the alert as a resolved one and wipe out it, is that right ?

Yes.

 

From what i tested, the AM won't send the recover notification to the receiver even the node online within or after 15 minutes.BTW, i'm using node-exporter for alert testing

Did you enable the sending of recovery notifications for your receiver's notification mechanism? For example, for email, this defaults to "false" and you'll need to add:

send_resolved: true

...to your email_config.

[Yitao Jiang]

Thank you Julius, i don't noticed such config, let me have a try. 

--

Regards,

Yitao

[Yang shuting]

hi, Do you minding telling me whether you try it or not?

在 2016年8月7日星期日 UTC+8上午6:59:55，Yitao Jiang写道：

Thank you Julius, i don't noticed such config, let me have a try. 

On Sun, Aug 7, 2016 at 6:53 AM, Julius Volz <juliu...@gmail.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Yitao

--

Regards,

Yitao

--

Regards,

Yitao