Alertmanager cannot access webhook service

[Елена Добровольская]

Hello.

I'm wrinting to you bcz someone from the command said that I have to ask you, not in an issue, here https://github.com/prometheus/alertmanager/issues/989 

I have alertmanager version v0.15.0-rc.3  installed on ubuntu 14, flags as follows: 

/usr/local/bin/alertmanager-v0.15.0-rc.3 --log.level=debug --config.file=/usr/local/etc/alertmanager.yml --web.external-url=https://staging-mon-internal.megafon.tv/alertmanager --web.listen-address=172.17.48.4:9093 --cluster.listen-address=172.17.48.4:9094 --storage.path=/var/alertmanager

Config https://pastebin.com/kFnHjiHr 

And also I have telegram bot service listening on port 8081. 

I make  curl -X POST  http://172.17.48.4:8081/alert and all is ok. 

This endpoint is placed in alertmanager.yml config.  

And in alertmanager I see errors:

2018-08-29T09:26:28.82156 level=debug ts=2018-08-29T09:26:28.821439966Z caller=notify.go:668 component=dispatcher msg="Notify attempt failed" attempt=1 integration=webhook receiver=bot err="context deadline exceeded"

2018-08-29T09:26:28.82160 level=error ts=2018-08-29T09:26:28.821550389Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="context deadline exceeded"

2018-08-29T09:26:28.82162 level=error ts=2018-08-29T09:26:28.821587883Z caller=dispatch.go:279 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="context deadline exceeded"

2018-08-29T09:26:28.82176 level=debug ts=2018-08-29T09:26:28.821701932Z caller=dispatch.go:444 component=dispatcher aggrGroup="{}:{alertname=\"ReplicationLag\"}" msg=Flushing alerts=[ReplicationLag[273d8de][active]]

Alerts work with slack and do not work with webhook service. 

Would you be so kind to help me with this, it seems like it some kind of misconfiguration issue. 

Thanks. 

--

Helen.

[Simon Pasquier]

First I would upgrade to v0.15.2 although that's not likely going to fix your issue.

Are you running AlertManager in a container?

To clear out configuration mismatch, you can double check the configuration reported by the Status page of the UI.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CANq_Y20Muup5TTuNme0GfRQcnMKXhAkGdd8prLe2FrMM_qVhxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Pasquier]

On Wed, Aug 29, 2018 at 3:01 PM, Елена Добровольская <hel...@mc2soft.ru> wrote:

Hello, Simon! 

Thank you for fast reply!

I upgraded to v0.15.2  and I still receive same errors.

No, I don't use a container. 

> double check the configuration reported by the Status page of the UI.

Alert successfully come from prometheus to alertmanager, so Status page of the alertmanager UI shows that there are alert firing now.  And, furthermore, alertmanager successfully sent a message to slack.

The problem is, that message is not sent to webhook service. Alertmanager writes in log, that "contex deadline exceeds".

"Context deadline exceeds" means that AlertManager couldn't complete the request in the allowed time. I would try the following:

1. tcpdump/wireshark to check whether any traffic happens between AlertManager and the webhook receiver

2. strace alertmanager

[Елена Добровольская]

Hi, Simon.

Using strace -s 2048 on alertmanager with specified in config flags, I got the following result: https://pastebin.com/VV8nBSzt

You see,  "POST /alert HTTP/1.1\r\nHost: 172.17.48.4:8081 

As I understand, the url alertmanager is using correct url to achieve webhook service. 

— 
С уважением,
Добровольская Елена | Team Lead  @ mc² software

hel...@mc2soft.ru
Моб. +7 999 848 2080

[Simon Pasquier]

On Wed, Aug 29, 2018 at 3:58 PM, Елена Добровольская <hel...@mc2soft.ru> wrote:

Hi, Simon.

Using strace -s 2048 on alertmanager with specified in config flags, I got the following result: https://pastebin.com/VV8nBSzt

You see,  "POST /alert HTTP/1.1\r\nHost: 172.17.48.4:8081 

As I understand, the url alertmanager is using correct url to achieve webhook service. 

Yes somehow the webhook receiver doesn't send a reply leaving the request open.

[Елена Добровольская]

> Yes somehow the webhook receiver doesn't send a reply leaving the request open.

Yes, and the problem is when I make curl request, it sends.

— 
С уважением,
Добровольская Елена | Team Lead  @ mc² software

hel...@mc2soft.ru
Моб. +7 999 848 2080