Alertmanager webhook issues

[Kiran Ali]

I'm using alertmanager v0.19.0 and my alertmanager configs are:

global:

  resolve_timeout: 1d

route:

  receiver: webhook

receivers:

- name: "webhook"

  webhook_configs:

  - url: http://ip/v1/alerts

    send_resolved: true

Alerts are generated on time from Prometheus and forwarded to alertmaanger. As soon as alertmanager receives alerts it should forward it to webhook but there is no proper time when alertmaanger will forward alerts to webhook. I was facing same issue with old alertmaanger version as well. 

[Simon Pasquier]

By default Alertmanager will wait 30s before sending the notification
(that's the group_wait parameter).
Can you be more specific about the issue you're seeing?

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5896ebd5-4a86-4a3e-845d-dacaa5123122%40googlegroups.com.

[Kiran Ali]

I suspect this issue is generated when alerts are in bulk. Suppose I have applied rule: Alert me when interfaces are down on 4 to 5 devices. And these alerts are 40 to 50 in number then alertmanager is not able to forward those alerts to webhook. 

Later on I changed alert manager YAML file to:

global:
  resolve_timeout: 1d
route:
  receiver: webhook
  group_by: ['...']
  group_wait: 0s
  group_interval: 10s
receivers:
- name: "webhook"
  webhook_configs:
  - url: http://nginx:8082/v1/alerts

I was still facing the issue. I then added repeat_nterval to the 30s. But now I start to get 

level=warn ts=2020-02-15T05:43:17.019Z caller=delegate.go:272 component=cluster msg="dropping messages because too many are queued" current=4118 limit=4096

On Thursday, February 13, 2020 at 7:40:29 PM UTC+5, Simon Pasquier wrote:

By default Alertmanager will wait 30s before sending the notification
(that's the group_wait parameter).
Can you be more specific about the issue you're seeing?

On Tue, Feb 11, 2020 at 5:54 AM Kiran Ali <kira...@wanclouds.net> wrote:
>
> I'm using alertmanager v0.19.0 and my alertmanager configs are:
>
> global:
>   resolve_timeout: 1d
> route:
>   receiver: webhook
> receivers:
> - name: "webhook"
>   webhook_configs:
>   - url: http://ip/v1/alerts
>     send_resolved: true
>
> Alerts are generated on time from Prometheus and forwarded to alertmaanger. As soon as alertmanager receives alerts it should forward it to webhook but there is no proper time when alertmaanger will forward alerts to webhook. I was facing same issue with old alertmaanger version as well.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Kiran Ali]

I'm still blocked, what could be the issue? I can see from /metrics of alertmanager that alerts are not forwarded.

On Thursday, February 13, 2020 at 7:40:29 PM UTC+5, Simon Pasquier wrote:

By default Alertmanager will wait 30s before sending the notification
(that's the group_wait parameter).
Can you be more specific about the issue you're seeing?

On Tue, Feb 11, 2020 at 5:54 AM Kiran Ali <kira...@wanclouds.net> wrote:
>
> I'm using alertmanager v0.19.0 and my alertmanager configs are:
>
> global:
>   resolve_timeout: 1d
> route:
>   receiver: webhook
> receivers:
> - name: "webhook"
>   webhook_configs:
>   - url: http://ip/v1/alerts
>     send_resolved: true
>
> Alerts are generated on time from Prometheus and forwarded to alertmaanger. As soon as alertmanager receives alerts it should forward it to webhook but there is no proper time when alertmaanger will forward alerts to webhook. I was facing same issue with old alertmaanger version as well.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.