TLS troubles with Alertmanager email notification and Microsoft Exchange

[powe]

Hi,

I'm trying to setup email notifications with Alertmanager using Microsoft Exchange, this is my alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_from: 'te...@test.com'


route:
  group_by['alertname']
  group_wait: 10s
  group_interval: 1h
  receiver: 'admins'


receivers:
- name: 'admins'
  email_configs:
  - smarthost: 'mailserver.domain.com:465'
    to: 'ad...@domain.com'
    send_resolved: true
    require_tls: true


inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Receiving server is an Windows Server 2012 R2 running Microsoft Exchange 2010. In exchange I've setup an Recieve Connector with the following properties (excerpt from Get-ReceiveConnector):

AuthMechanism: Tls, Integrated, BasicAuth
Bindings: 0.0.0.0:465
PermissionGroups: AnonymousUsers
RemoteIPRanges: {<ip of prometheus server>}
Fqdn: mailserver.domain.com
RequireTLS: True
RequireEHLODomain: False

When alertmanager tries to send an email, the following message is written to stderr:

level=error
 ts=<timestamp> caller=dispatch.go:301 component=dispatcher
msg="Notify for alerts failed" num_alerts=1 err="establish TLS
connection to server: tls: first record does not look like a TLS
handshake"

I can see traffic going to port 465 in Wireshark on both servers, one TLSv1 Client Hello and then a SYN sequence.

I've tried to connect to the port with openssl which reports:

no peer certificate available
-----
no client certificate CA names sent
-----
SSL handshake has read 5 bytes and written 194 bytes
Verification OK
-----
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol: TLSv1.2
    Cipher: 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: none
    PSK identity hint: none
    SRP username: none
    Start Time: <start time>
    Timeout: 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no

[powe]

Solved it! We had a pre-existing Receive connector on port 25, but i couldn't even see any traffic in Wireshark when using that port, hence using port 465 in my original post. Tried changing back to that port and it was still dead in Wireshark. Turns out, our Anti-Virus was blocking all attempts to communicate on port 25 by default. Added an exception to the AV and hey presto, email alerts started to work!