[AlertManager] Send alert using localhost smtp

[Billy Teves]

I've seen alot of functions sending alert via gmail, hipchat, slack, page duty etc.

I was trying to setup the smtp by not using the gmail in sending alert. Instead use my localhost via port 25.

my sample configs:

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'localhost:25'
  smtp_from: 'root@localhost'
  smtp_auth_username: 'root'
  smtp_auth_password: ''
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 1h
  receiver: team-X-mails

  routes:
  - match:
      job: dailytest
    receiver: team-X-mails
    repeat_interval: 1h

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: '<my-email>'

here's my sample alert rule:

#Alert Sample

ALERT DailyTest
  IF vector(1) > 0
  FOR 1m
  LABELS {
    job = "dailytest",
  }
  ANNOTATIONS {
    summary = "daily alert test",
    description = "daily alert test"
  }

In my prometheus, I have setup alert rules. the alert manager is also linked. Prometheus able to detect issue in one of my rules and it shows on /alerts. Status is pending. When I check the alert manager, nothing happens or nothing is being emailed. I then checked if the smtp not working on my localhost. I tried to email my-email and i was able to send/receive.

I would like to know if is there something in my config that is incorrect?

Regards,

[Fabian Reinartz]

The configuration looks correct in general. You said that the Prometheus web UI shows your alert to be "pending" though. The alert is only sent to Alertmanager though as soon as it starts saying "firing". This should happen after about 1 minute based on your configuration.

Can you verify whether things are still not working when the alert is marked as "firing" in Prometheus?

Otherwise, are there any logs in Alertmanager indicating the problem. Set the `-log.level=debug` flag for the maximum verbosity.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthias Rampke]

Also check your MTA logs for denied authentication. It could be that the `smtp_auth` actually throws things off – with Alertmanager trying to authenticate as "root" with an empty password, while in fact (if you have configured it that way) your MTA would accept email from localhost without auth. FWIW, we do the same and only set smtp_smarthost and smtp_from.

/MR

[Billy Teves]

The logging is set to debug. Upon checking the log file, there is no log regarding this.

I then check the /var/log/maillog

and this shows:

Jul 14 00:54:21 localhost postfix/pickup[29896]: 963CD1A10D0: uid=0 from=<root>
Jul 14 00:54:21 localhost postfix/cleanup[29897]: 963CD1A10D0: message-id=<20160714075421.963CD1A10D0@localhost>
Jul 14 00:54:21 localhost postfix/qmgr[16629]: 963CD1A10D0: from=<root@localhost>, size=17069, nrcpt=1 (queue active)
Jul 14 00:54:22 localhost postfix/local[29899]: 963CD1A10D0: to=<root@localhost>, orig_to=<root>, relay=local, delay=3226, delays=3226/0.01/0/0.32, dsn=2.0.0, status=sent (delivered to mailbox)
Jul 14 00:54:22 localhost postfix/qmgr[16629]: 963CD1A10D0: removed

based on the result, instead of sending the result to <my-email>, it is being sent to root@localhost.

[Matthias Rampke]

I don't see anything wrong in the alertmanager configuration that would cause this. How is your postfix configured? Can you send email out at all, or does it possibly only deliver locally? Try using `mail` as root to send email to the same recipient?

/MR

[Billy Teves]

here's the postfix config:

alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debug_peer_level = 2
html_directory = no
inet_interfaces = localhost
inet_protocols = all
mail_owner = postfix
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
mydestination = $myhostname, localhost.$mydomain, localhost
newaliases_path = /usr/bin/newaliases.postfix
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES
relayhost = <relay-host-value-here>
sample_directory = /usr/share/doc/postfix-2.6.6/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
unknown_local_recipient_reject_code = 550

I tried this in my terminal: echo "this is the body" | mail -s "this is the subject" "user...@mydomain.com", I was able to send the email to user...@mydomain.com (which is an outlook email). I use the same email in my `to` config. I think I need to use the value of relayhost in smtp_smarthost?

[Billy Teves]

After I changed the smtp_smarthost to my relayhost with its port, the prometheus db alert has a status of firing, unlike last time which is stuck on pending. Unfortunately, i am receiving this kind of warning on alertmanager:

DEBU[5421] Received alert                                alert=DailyTest[c1fbb74][active] component=dispatcher source=dispatch.go:160
. . . . . . . .
DEBU[5421] notify [DailyTest[c1fbb74][active]]             source=notify.go:400 step=retry
WARN[5421] Notify attempt 1 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5421] Notify attempt 2 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5422] Notify attempt 3 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5422] Notify attempt 4 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5424] Notify attempt 5 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5426] Notify attempt 6 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5429] Notify attempt 7 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5436] Notify attempt 8 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5443] Notify attempt 9 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5453] Notify attempt 10 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5467] Notify attempt 11 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5497] Notify attempt 12 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5544] Notify attempt 13 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5610] Notify attempt 14 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
WARN[5658] Notify attempt 15 failed: require_tls: true (default), but "<relay-host-value-here>" does not advertise the STARTTLS extension  source=notify.go:193
DEBU[5660] Received alert                                alert=DailyTest[c1fbb74][active] component=dispatcher source=dispatch.go:160
ERRO[5721] Error on notify: context deadline exceeded    source=notify.go:152
ERRO[5721] Notify for 1 alerts failed: context deadline exceeded  source=dispatch.go:238

I saw on this link: https://github.com/prometheus/alertmanager/issues/377 in the global part `require_tls: true` I put it on false (not sure if this works), reload the alertmanager, still same result above.

[Billy Teves]

Hi,

I think I was able to resolve this by recompiling the alertmanager and set the DefaultEmailConfig's RequiredTLS (config/notifiers.go line 36) to false. Seems it is enforcing to required tls even if you put it to false in the alertmanager config. With this changes, I was able to receive email from alert manager

[Brian Brazil]

On 19 July 2016 at 09:52, Billy Teves <billy...@gmail.com> wrote:

Hi,

I think I was able to resolve this by recompiling the alertmanager and set the DefaultEmailConfig's RequiredTLS (config/notifiers.go line 36) to false. Seems it is enforcing to required tls even if you put it to false in the alertmanager config. With this changes, I was able to receive email from alert manager

Can you file a bug for this with the config you used?

Brian

--

Brian Brazil

www.robustperception.io

[Billy Teves]

Here's the link : https://github.com/prometheus/alertmanager/issues/433