What could be causing the alert manager to fail sending emails?

[Zikou]

I'm testing to trigger alert manager to send email but It doesnt work logs that I got from the pod of alert manager

ts=2023-04-18T17:35:15.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="monitoring/main-rules-alert-config/email/email[0]: notify retry canceled after 4 attempts: establish connection to server: dial tcp 74.125.206.108:587: i/o timeout" 

and here is the content of my alertmanagerconfig

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: main-rules-alert-config
  namespace: monitoring
spec:
  route:
    receiver: 'email'
    repeatInterval: 30m
    routes:
    - matchers:
      - name: alertname
        value: HostHighCPULoad
    - matchers:
      - name: alertname
        value: KuberenetesPodCrashLooping
      repeatInterval: 30m
  receivers:
    - name: 'email'
      emailConfigs:
      - to: "zikou...@gmail.com"
        from: "zikou...@gmail.com"
        smarthost: 'smtp.gmail.com:587'
        authIdentity: "zikou...@gmail.com"
        authUsername: "zikou...@gmail.com"
        authPassword:
          name: gmail-auth
          key: password

I created a secret named gmail-auth (in the same namespace as alert) before apply the alert config the secret contain password of my gmail but since 2FA is enabled I put app password in the secret base64 of course but I dont know where is the issue why I dont receive any mail

[Zikou]

I created a secret named gmail-auth (in the same namespace as alert) before apply the alert config the secret contain password of my gmail but since 2FA is enabled I put app passwors in the secret base64 of course but I dont know where is the issue why I dont receive any mail

[Brian Candler]

The error "dial tcp 74.125.206.108:587: i/o timeout" suggests that your container can't even make an outbound network connection. Try logging into the container and running "telnet smtp.gmail.com 587" or "nc smtp.gmail.com:587".  You should get a banner back like this:

220 smtp.gmail.com ESMTP t26-2002.............1 - gsmtp

If you don't, then you have a networking/firewalling issue to investigate on your k8s cluster - it's nothing to do with alertmanager.

As for your config, I observe that your 'receivers' section doesn't look like what Alertmanager requires:

https://prometheus.io/docs/alerting/latest/configuration/#email_config

For example, you have "emailConfigs" whereas alertmanager requires "email_configs".

Presumably, therefore, you are using some middleware which is munging the config before giving it to alertmanager (I guess this is something provided by "monitoring.coreos.com"?)

If you're able to find the alertmanager config *as deployed* by this middleware, e.g. by entering the running container and printing it, then you can post that here - or it may be obvious what the problem is. Note that auth_password is *not* base64 encoded in the alertmanager config file - I don't know if your middleware base64-decodes it as part of deployment.