Alertmanager firing much notification on webhook ?!

[petar...@gmail.com]

Hi,

I have configured Prometheus and Alertmanager in HA.

Alertmanager get alerts with same label because de-duplication, but I think that doesn't work.

On both alertmanager I have this configuration:

I'm using webhook for Jira alert. This hook works, but I have problem if some alert is firing I can see that alert on both alertmanager, what is ok. Both alerts have same label and that is ok. 
But problem is in notification. 
Notification on slack and push works in interval as it set, but on webhooks firing constantly.

What can be problem here ?

route:
  receiver: jira
  group_by:
  - alertname
  routes:
  - receiver: jira
    match:
      severity: Highest
    continue: true
  - receiver: slack
    match:
      severity: Highest
    continue: true
  - receiver: push
    match:
      severity: Highest
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 1h
receivers:
- name: slack
  slack_configs:
  - send_resolved: false
    api_url: <secret>
    channel: '#prometheus'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ .CommonAnnotations.SUMMARY }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ .CommonAnnotations.DESCRIPTION }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: '{{ template "slack.default.iconurl" . }}'
- name: push
  pushover_configs:
  - send_resolved: true
    user_key: <secret>
    token: <secret>
    title: '{{ template "pushover.default.title" . }}'
    message: '{{ template "pushover.default.message" . }}'
    url: '{{ template "pushover.default.url" . }}'
    priority: '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}'
    retry: 1m0s
    expire: 1h0m0s
- name: jira
  webhook_configs:
  - send_resolved: true
    url: http://127.0.0.1:8000
templates:
- /etc/alertmanager/default.tmpl

[jdbj...@gmail.com]

Hey Petar,

I've tested the same configuration and receivers, and it worked as expected. 

Can you share more information? Alertmanager version, logs (with -log.level=debug), and the webhook payload would be helpful to understand what is happening.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a2c93e0d-8808-42a1-82fa-49183522dad5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Petar Kozic]

@jdbjunior

How you can test when you don’t have jira webhook ?

Alerts on slack and pushover works correct, but webhooks send notification constantly ?!

[Petar Kozic]

In the enabled debug log-level I find problem. Problem is because webhook return failed, but POST request is correct and return status 200. What I need to add or do in webhook to resolve this.

time="2017-10-30T09:59:41+01:00" level=debug msg="Notify attempt 1 for "webhook" failed: Post http://127.0.0.1:8000: EOF" source="notify.go:585"

time="2017-10-30T09:59:42+01:00" level=debug msg="Notify attempt 2 for "webhook" failed: Post http://127.0.0.1:8000: EOF" source="notify.go:585"

time="2017-10-30T09:59:44+01:00" level=debug msg="Notify attempt 3 for "webhook" failed: Post http://127.0.0.1:8000: EOF" source="notify.go:585"

time="2017-10-30T09:59:45+01:00" level=debug msg="Notify attempt 4 for "webhook" failed: Post http://127.0.0.1:8000: EOF" source="notify.go:585"

Response from webhook:

127.0.0.1 - - [30/Oct/2017 09:59:37] "POST / HTTP/1.1" 200 -

Response HTTP Status Code: 201

Response HTTP Response Body: My JSON body

127.0.0.1 - - [30/Oct/2017 09:59:41] "POST / HTTP/1.1" 200 -

Response HTTP Status Code: 201

Response HTTP Response Body: My JSON body

127.0.0.1 - - [30/Oct/2017 09:59:42] "POST / HTTP/1.1" 200 -

Response HTTP Status Code: 201

Response HTTP Response Body: My JSON body

[jdbj...@gmail.com]

Hey Petar,

I'm able to test without the jira webook, for me what's important is that alertmanager sends the jira notification only once, what happens with it (if it goes to jira or not, does not invalidate the test).

The error does not appear to be alertmanager faults, when it sends the notification it is getting an error 'EOF', and this is on the jira code. Because of this error, alertmanager is retrying to send the notification again, and again. Try to debug the jira webhook integration with 'curl' to understand what is happening. 

On the alertmanager repo there's a basic webhook implementation that works, maybe you can base the work from it if you using go. 

https://github.com/prometheus/alertmanager/blob/master/examples/webhook/echo.go

[Petar Kozic]

I found problems.

I forget to add in my webhoks:

self.end_headers()

Thank You !