Alertmanager doesn't send mail when require_tls is set to false and smarthost is a remote machine
190 views
Skip to first unread message
Emmanuel Livingstone
Nov 26, 2019, 4:37:47 AM11/26/19
to Prometheus Users
This is not really an issue with AlertManager. More of a suggestion on improving docs that could potentially reduce the debugging time for devs who want to configure.
When we try to configure email receiver with a remote smarthost and set require_tls to false, then alertmanager fails with a cryptic error. Here are the logs
level=info ts=2019-11-26T09:09:29.822Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yamllevel=info ts=2019-11-26T09:09:29.832Z caller=main.go:429 msg=Listening address=:9093level=info ts=2019-11-26T09:09:31.790Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000171097slevel=info ts=2019-11-26T09:09:39.791Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001056255slevel=error ts=2019-11-26T09:09:47.560Z caller=email.go:148 integration=email msg="failed to close SMTP connection" err="write tcp 10.33.95.134:49772->10.47.255.15:25: use of closed network connection"level=error ts=2019-11-26T09:09:48.251Z caller=email.go:148 integration=email msg="failed to close SMTP connection" err="write tcp 10.33.95.134:49776->10.47.255.15:25: use of closed network connection"
When I looked into the source code of alertmanager/notify/email/email.go (v0.18.0), I came across the following:
- email.go line 179: n.auth is called and smtp.PlainAuth object is constructed
- email.go line 184: c.Auth(plainAuth) is called
- smtp/smtp.go line 204: a.Start is called, i.e., plainAuth.Start is called
- smtp/auth.go line 68, 69: it returns an error if tls is set to false and the smart host is not a local address
- Coming back to smtp/smtp.go line 205 this error is caught and the connection is closed
This is the reason the client.Close that is called in the defer block Email.Notify is throwing the error seen in the logs.
But this took me sometime for me to go through the code to figure this out. Has anyone else faced this issue before? Does it make sense to add this explicitly in the documentation?
Brian Brazil
Nov 26, 2019, 6:19:17 AM11/26/19
to Emmanuel Livingstone, Prometheus Users
On Tue, 26 Nov 2019 at 09:37, Emmanuel Livingstone <livings...@gmail.com> wrote:
This is not really an issue with AlertManager. More of a suggestion on improving docs that could potentially reduce the debugging time for devs who want to configure.
This sounds like a bug, TLS should be disableable smarthost or not. Would you like to file an issue to get this fixed, or even a PR fixing it given you've already dug through the code?
Brian
Simon Pasquier
Nov 26, 2019, 9:45:55 AM11/26/19
to Emmanuel Livingstone, Prometheus Users
Can you try again with v0.19.0. Log messages for the email integration
have been improved with this version.
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e46f5ba2-2b00-4bee-8d15-17f083d8bdf6%40googlegroups.com.
have been improved with this version.
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e46f5ba2-2b00-4bee-8d15-17f083d8bdf6%40googlegroups.com.
Emmanuel Livingstone
Nov 26, 2019, 11:54:09 PM11/26/19
to Prometheus Users
@Brian: Alertmanager code is not responsible for disallowing remote unencrypted SMTP connection. It is the golang net/smtp standard library that has this restriction. Here is the screenshot of the relevant code from https://golang.org/src/net/smtp/auth.go. This looks more like a security consideration as per the comments.
@Simon: Yes the logging seems relevant in v0.19.0. Here are the logs:
level=info ts=2019-11-27T04:33:15.836Z caller=main.go:466 msg=Listening address=:9093level=info ts=2019-11-27T04:33:17.786Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000130871slevel=info ts=2019-11-27T04:33:25.787Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000790267slevel=error ts=2019-11-27T04:39:15.025Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="*smtp.plainAuth auth: unencrypted connection" context_err="context deadline exceeded"level=error ts=2019-11-27T04:39:15.025Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="*smtp.plainAuth auth: unencrypted connection"
> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Emmanuel Livingstone
Nov 26, 2019, 11:59:03 PM11/26/19
to Prometheus Users
IMHO the standard library could've had a more relevant error message like "unencrypted connection not allowed for remote SMTP endpoints". But at least the current error message helps us to identify where the problem is.
Brian Brazil
Nov 27, 2019, 2:17:49 AM11/27/19
to Emmanuel Livingstone, Prometheus Users
On Wed, 27 Nov 2019 at 04:54, Emmanuel Livingstone <livings...@gmail.com> wrote:
@Brian: Alertmanager code is not responsible for disallowing remote unencrypted SMTP connection. It is the golang net/smtp standard library that has this restriction. Here is the screenshot of the relevant code from https://golang.org/src/net/smtp/auth.go. This looks more like a security consideration as per the comments.
Ah, in that case on the presumption that the AM maintainers don't want to workaround this would you like to send a PR for the docs? They're in the docs repo in this case.
Brian
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6054d3e5-d769-4a0b-8e95-f15b0da8d406%40googlegroups.com.
Brian Brazil
Emmanuel Livingstone
Nov 27, 2019, 3:39:09 AM11/27/19
to Prometheus Users
On Wednesday, November 27, 2019 at 12:47:49 PM UTC+5:30, Brian Brazil wrote:
On Wed, 27 Nov 2019 at 04:54, Emmanuel Livingstone <living...@gmail.com> wrote:@Brian: Alertmanager code is not responsible for disallowing remote unencrypted SMTP connection. It is the golang net/smtp standard library that has this restriction. Here is the screenshot of the relevant code from https://golang.org/src/net/smtp/auth.go. This looks more like a security consideration as per the comments.
Ah, in that case on the presumption that the AM maintainers don't want to workaround this would you like to send a PR for the docs? They're in the docs repo in this case.Brian
Sure will do.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6054d3e5-d769-4a0b-8e95-f15b0da8d406%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages