Issues sending alerts through SendGrid

[Cristhian Rivera]

Hi I'm trying to send the alerts generated by Prometheus trough the SendGrid SMTP.

Following the SendGrid docs: https://sendgrid.com/docs/API_Reference/SMTP_API/integrating_with_the_smtp_api.html

I included the corresponding settings:

global:
  smtp_from: "AlertManager <alertm...@domain.com>"
  smtp_smarthost: smtp.sendgrid.net:587
  smtp_require_tls: true
  smtp_auth_username: apikey
  smtp_auth_identity: apikey
  smtp_auth_password: "<MY API KEY>"

route:
  repeat_interval: 1m
  receiver: default

receivers:
  - name: default
    email_configs:
      - to: email-acc@domain.com

Then with a current alert active in prometheus I'm not receiving any emails. And in the SendGrid dashboard there is no activity either.

Also in the logs of the container there is no indication that there is an issue:

level=info ts=2020-06-13T20:03:47.844Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)"

level=info ts=2020-06-13T20:03:47.844Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)"

level=info ts=2020-06-13T20:03:47.845Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.19.0.4 port=9094

level=info ts=2020-06-13T20:03:47.846Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s

level=info ts=2020-06-13T20:03:47.872Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alert-manager.yaml

level=info ts=2020-06-13T20:03:47.872Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alert-manager.yaml

level=info ts=2020-06-13T20:03:47.874Z caller=main.go:497 msg=Listening address=:9093

level=info ts=2020-06-13T20:03:49.848Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001599783s

level=info ts=2020-06-13T20:03:57.857Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.010722253s

I've tried with the other ports that SendGrid supports but the same result

Only when the port is 25 and the smtp_require_tls: is set to false I receive a warning of

level=error ts=2020-06-13T20:13:33.862Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="*smtp.plainAuth auth: unencrypted connection"

Any idea of what I'm missing here?

Thanks!

[Matthias Rampke]

Can you enable debug logging for Alertmanager? That should show whether it is even trying to send an email.

How does SendGrid behave when you submit something with invalid credentials? Do you expect that to explicitly fail?

From this doc, SendGrid seems to use LOGIN not PLAIN. Try leaving out `smtp_auth_identity` and only specifying user and password?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2d159d2b-a1fa-43fa-ade2-9e9db3be0952o%40googlegroups.com.

[Cristhian Rivera]

How can I enable the debug log? I tried looking in the configuration section and found no option for that:  prometheus.io/docs/alerting/latest/configuration/

Regarding SendGrid behaviour with incorrect credentials i expect for it to fail, but currently have no got the result. I tried the interaction with SendGrid using a PHP client, but that uses the API provided by SendGrid.

From this doc, SendGrid seems to use LOGIN not PLAIN. Try leaving out `smtp_auth_identity` and only specifying user and password?

I tried that but same result,

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Cristhian Rivera]

Ok, i changed the log level to debug and I'm seeing that its "sending" the notifications:

level=debug ts=2020-06-15T18:17:54.150Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[48afd0d][active]]

level=debug ts=2020-06-15T18:18:09.135Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=InstanceDown[48afd0d][active]

level=debug ts=2020-06-15T18:18:09.152Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[48afd0d][active]]

level=debug ts=2020-06-15T18:18:24.154Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[48afd0d][active]]

level=debug ts=2020-06-15T18:18:39.155Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[48afd0d][active]]

Also I tried checking the metrics in prometheus, and the notificiations are beeing recorded as sent without errors

prometheus_notifications_sent_total{alertmanager="http://prometheus-alert-manager:9093/api/v1/alerts", instance="localhost:9090", job="prometheus"}     8

prometheus_notifications_errors_total{alertmanager="http://prometheus-alert-manager:9093/api/v1/alerts", instance="localhost:9090", job="prometheus"}   0

But in SendGrid there is still no trace of those messages... I tested the API-KEY again using the php library and it sent the mails like normal...