Alertmanager failing to notify | Context Deadline Exceeded

[keshav19...@gmail.com]

Hi,

I am getting below error when I am trying to send alert via alert manager. 

level=error ts=2018-09-20T20:35:33.377247181Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="context deadline exceeded"

level=error ts=2018-09-20T20:35:33.377314472Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="context deadline exceeded"

Details:

Prometheus version 2.3.2 running in Kubernetes

Alertmanager version 0.15.2 running in kubernetes

URL Endpoint for receiver: json-server:3030/EspServiceRequest

Above Endpoint is a service which is also running in kubernetes and I can able to hit the endpoint from the alert manager host

sh-4.2# curl -v json-server:3030/EspServiceRequest

* About to connect() to json-server port 3030 (#0)

*   Trying 192.168.145.41...

* Connected to json-server (192.168.145.41) port 3030 (#0)

> GET /EspServiceRequest HTTP/1.1

> User-Agent: curl/7.29.0

> Host: json-server:3030

> Accept: */*

> 

< HTTP/1.1 200 OK

< X-Powered-By: Express

< Vary: Origin, Accept-Encoding

< Access-Control-Allow-Credentials: true

< Cache-Control: no-cache

< Pragma: no-cache

< Expires: -1

< X-Content-Type-Options: nosniff

< Content-Type: application/json; charset=utf-8

< Content-Length: 2317

< ETag: W/"90d-uFTpG0JU/P9t/SwjtDitnIPYSAM"

< Date: Thu, 20 Sep 2018 20:44:25 GMT

< Connection: keep-alive

Below is my alert.rule and alertmanager.config

alerts.rule

sh-4.2# cat /etc/prometheus/alert.rules      

groups:

- name: sample1

  rules:

  - alert: sample_alert

    expr: testmetric == 0

    labels:

      severity: warning

      instance: i1

    annotations:

      summary: this is sample summary for {{ $labels.instance }}

sh-4.2# cat /etc/prometheus/alertmanager.yml 

global:

  resolve_timeout: 1m

route:

  group_by: [instance]

  group_wait: 1m

  group_interval: 5m

  repeat_interval: 1h

  receiver: 'alert_recv' 

receivers:

- name: 'alert_recv'

  webhook_configs:

  - url: 'json-server:3030/EspServiceRequest'

In prometheus UI, I can able to see alert is in firing state

Snapshot:

Could you please let me know what I am doing wrong here.

Any help would be appreciated.

Regards,

Keshav Sharma

[Simon Pasquier]

Can you double-check that the "url:" parameter in your AlertManager configuration file starts with "http://"? Although if not, it should be caught by AlertManager on startup.

Other than that it might a DNS resolution issue so you may want to specify the full name of the webhook service.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b467f5bb-65e0-48b1-a8a6-edad6080b1a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.