Not receiving email alerts from AlertManager on gmail

[Gabriel Balan]

The setup is the following: A stack of servers using Prometheus and AlertManager that are hosted on google cloud, we are receiving alerts on pagerduty and slack just fine and one of our customers need alerts via email.

The smtp is google. The yaml is correct by my judgement, format is definitely correct.

https://pastebin.com/BSGejXcR

I have religiously respected the following article: https://www.robustperception.io/sending-email-with-the-alertmanager-via-gmail/

Any help is appreciated.

[Simon Pasquier]

If Alertmanager can't send the emails, you should have something in the logs.

> This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/61e63ee5-e9ad-4b82-96ae-41bf5a739e41%40googlegroups.com.

[Gabriel Balan]

-- Logs begin at Fri 2019-09-06 04:14:01 UTC. --

Oct 02 11:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T11:00:03.027772514Z caller=compact.go:496 component=tsdb msg="write block" mint=1570003200000 maxt=1570010400000 ulid=01DP620WYQZBGSQ9EQGFZVJ9F1 duration=2.940232809s

Oct 02 11:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T11:00:03.410596834Z caller=head.go:536 component=tsdb msg="head GC completed" duration=103.791205ms

Oct 02 11:00:08-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T11:00:08.986385823Z caller=head.go:583 component=tsdb msg="WAL checkpoint complete" first=11654 last=11660 duration=5.575643408s

Oct 02 13:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T13:00:03.278748977Z caller=compact.go:496 component=tsdb msg="write block" mint=1570010400000 maxt=1570017600000 ulid=01DP68WM6XQW74F5A0C455EAFW duration=3.18561563s

Oct 02 13:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T13:00:03.651186104Z caller=head.go:536 component=tsdb msg="head GC completed" duration=107.065384ms

Oct 02 13:00:09 -prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T13:00:09.214028303Z caller=head.go:583 component=tsdb msg="WAL checkpoint complete" first=11661 last=11667 duration=5.562736854s

Oct 02 15:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T15:00:03.375180715Z caller=compact.go:496 component=tsdb msg="write block" mint=1570017600000 maxt=1570024800000 ulid=01DP6FRBEP7NCDCPAVK4NK2C82 duration=3.288658273s

Oct 02 15:00:03-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T15:00:03.754789228Z caller=head.go:536 component=tsdb msg="head GC completed" duration=106.577187ms

Oct 02 15:00:10-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T15:00:10.936123113Z caller=head.go:583 component=tsdb msg="WAL checkpoint complete" first=11668 last=11675 duration=7.181200921s

Oct 02 15:00:17-prod-monitor-01 prometheus[8237]: level=info ts=2019-10-02T15:00:17.587947802Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1569996000000 maxt=1570017600000 ulid=01DP6FRPBQ0SMT3EDBQ4NX4K9E sources="[01DP5V55PS3BH3W2D4NRBG3GZA 01DP620WYQZBGSQ9EQGFZVJ9F1 01DP68WM6XQW74F5A0C455EAFW]" duration=6.332094367s

Just what I am getting from Prometheus and on alertmanager the logs I receive from alertmanager.log are just the stream of alerts.

[Simon Pasquier]

I meant Alertmanager's logs.

> This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/efb46e3b-20a0-4448-9ba8-2dc900ee3c89%40googlegroups.com.

[Gabriel Balan]

./alertmanager --log.level=info

level=info ts=2019-10-02T16:30:24.240654095Z caller=main.go:140 msg="Starting Alertmanager" version="(version=0.17.0, branch=HEAD, revision=c7551cd75c414dc81df027f691e2eb21d4fd85b2)"

level=info ts=2019-10-02T16:30:24.240749285Z caller=main.go:141 build_context="(go=go1.12.4, user=root@932a86a52b76, date=20190503-09:10:07)"

level=info ts=2019-10-02T16:30:24.241617607Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=10.154.0.9 port=9094

level=info ts=2019-10-02T16:30:24.245328365Z caller=cluster.go:613 component=cluster msg="Waiting for gossip to settle..." interval=2s

level=info ts=2019-10-02T16:30:24.28172957Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=alertmanager.yml

level=info ts=2019-10-02T16:30:24.282426876Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=alertmanager.yml

level=info ts=2019-10-02T16:30:24.286025012Z caller=main.go:365 msg=Listening address=:9093

level=error ts=2019-10-02T16:30:24.286258066Z caller=main.go:367 msg="Listen error" err="listen tcp :9093: bind: address already in use"

level=info ts=2019-10-02T16:30:24.286443951Z caller=cluster.go:622 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=40.847907ms

level=error ts=2019-10-02T16:30:24.28655941Z caller=nflog.go:372 component=nflog msg="Creating shutdown snapshot failed" err="open data/nflog.4c84fdf671388436: permission denied"

level=info ts=2019-10-02T16:30:24.286608322Z caller=silence.go:386 component=silences msg="Creating shutdown snapshot failed" err="open data/silences.691cd2bf7bda49e0: permission denied"

This is all I have. Prometheus is not a easy tool if you just started using it as me, most probably I will need to need to buy the book.

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Benoit Dubois]

level=error ts=2019-10-02T16:30:24.286258066Z caller=main.go:367 msg="Listen error" err="listen tcp :9093: bind: address already in use"

Could you have another alertmanager process already running, but with older configs without the email part?

[Gabriel Balan]

We have a script that restarts it everytime we have a new config but I will check again to see if it's killing the process properly and come back here with a response.

[Christian Hoffmann]

On 10/2/19 10:01 PM, Gabriel Balan wrote:
> We have a script that restarts it everytime we have a new config but I
> will check again to see if it's killing the process properly and come
> back here with a response.

Couldn't you just reload via SIGHUP in this case?
Only command line parameter changes should need a full restart.

https://prometheus.io/docs/alerting/configuration/

Kind regards,
Christian

[Gabriel Balan]

I will be honest Christian, I just started working at this company for a month now and it's my first dev-opsy role and I am not far away from clueless in some days.

I understand what you mean though by SIGHUP and will check the script tomorrow to see if it's killing the process or restarting the whole service.

[Simon Pasquier]

As others pointed out, you need to find why another process is already
listening on the same 9093 port.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHvujoHg2M3FX7j8CYdiGo7kV8F%3DQahcKeqyR8atanD7yAvZMg%40mail.gmail.com.

[Gabriel Balan]

Good morning,

COMMAND     PID       USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

alertmana 23477 prometheus    3u  IPv6 121708925      0t0  TCP *:copycat (LISTEN)

sudo netstat -tulpn | grep 9093

tcp6       0      0 :::9093                 :::*                    LISTEN      23477/./alertmanage

Apparently Alertmanager is on that specific port.

The following is the port that Prometheus is using.

sudo netstat -tulpn | grep prometheus

tcp6       0      0 :::9090                 :::*                    LISTEN      8237/prometheus

 

There are not multiple AlertManger processes running apparently.

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Gabriel Balan]

Config from Prometheus.

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      - localhost:9093

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Simon Pasquier]

Looking again at your configuration, there's no route sending
notifications to the "email-me" receiver. This is why no email is
sent.

> This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6ccf9af7-bb5e-4a7a-b920-39759ade5ae5%40googlegroups.com.

[Peter Leese]

Beat me too it Simon.

https://prometheus.io/webtools/alerting/routing-tree-editor/

This will help when building routes

[Gabriel Balan]

I do not know how routing works and can't find a example that I can understand.

Could you have a look please? I am already working on it for two days now and I am not familiar at all with Prometheus.

https://pastebin.com/BSGejXcR

I just want to it to finally send emails.

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Pradeep Chetal]

On line 13 did you try changing

    receiver: 'pager-duty' 

 to

    receiver: 'email-me'

 

This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a4f8998b-b0e3-4de6-b7e9-9454d4332ac7%40googlegroups.com.

[Gabriel Balan]

But I need two receivers.

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Gabriel Balan]

It works now as Pradeep suggested but I need three receivers: slack, pagerduty and email.

On Wednesday, 2 October 2019 15:30:30 UTC+1, Gabriel Balan wrote:

[Harald Koch]

On Thu, Oct 3, 2019, at 08:53, Gabriel Balan wrote:

It works now as Pradeep suggested but I need three receivers: slack, pagerduty and email.

Receivers can have multiple destinations configured; that might be the simplest solution if all of your alerts need to go
multiple places. 

Here is a more complicated routing example from my ESB:

  # send any alerts with an 'email_to' label to that email address, as well as normal EMS processing

  - match_re:

      email_to: .*@.*

    group_by: [ 'alertname', 'email_to' ]

    receiver: email_router

    # also email interfacing

    continue: true

  - match_re:

      alertname: "(TooManyMessagesQueued|MessagesNotFlowing)"

    group_by: [ 'alertname', 'priority', 'timeperiod' ]

    routes:

      # priority='none' goes nowhere

      - match:

          priority: none

        receiver: devnull

      # priority='high' goes to the on-call pager

      - match:

          priority: high

        receiver: pager

        continue: true

      # priority high and medium go via email

      - match_re:

          priority: (high|medium)

        receiver: interfacing-users

        continue: true

      # all alerts to Slack

      - match_re:

          priority: .*

        receiver: slack

      # end TooManyMessagesQueued

This works because of the "continue: true" in the various rules.

The first rule sends any alert with an "email_to" label to that email address, and then continues on to the rest of the route. The main section sends "high" priority alerts to all three destinations; "medium" priority alerts to the team; and all alerts to Slack.

--

Harald

Harald Koch

c...@pobox.com

[Gabriel Balan]

Hi,

Well, I am really new to this and the configs have already been set up by someone else. This week I was introduced to Prometheus so consider me a baby still.

https://pastebin.com/BSGejXcR

Can you have a look and tell me how I can receive alerts to every receiver?

[Simon Pasquier]

You can combine all notifiers into one receiver like this:
https://pastebin.com/Y4s17nps

> This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d4a72a93-2d4f-44c1-beda-68ed94706fff%40googlegroups.com.

[Gabriel Balan]

Thank you, Simon. Evidently, that worked.

Is it possible for the email notifier to only send alerts about a specific server?

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Simon Pasquier]

On Thu, Oct 3, 2019 at 5:05 PM Gabriel Balan
<gabrie...@celer-tech.com> wrote:
>
> Thank you, Simon. Evidently, that worked.
>
> Is it possible for the email notifier to only send alerts about a specific server?

Sure, you need to configure a specific route and receiver for this
particular instance:

routes:
route:
group_by: ['title']
group_wait: 30s
group_interval: 2m
repeat_interval: 30s
receiver: send-to-all
routes:
- match:
alertname: InstanceDown
instance: "someinstance:9100"
receiver: 'email-me'
- ...
receivers:
- name: email-me
email_configs:
- ...

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee9bfd7c-c545-44ae-a83c-b2576fb5cba1%40googlegroups.com.

[Gabriel Balan]

Hi Simon,

Thank you for your help until now. Almost there.

https://pastebin.com/V72WNrem

Does this look correct? I am getting duplicated maping key at line 16.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee9bfd7c-c545-44ae-a83c-b2576fb5cba1%40googlegroups.com.

[Christian Hoffmann]

Hi Gabriel,

On 10/4/19 10:57 AM, Gabriel Balan wrote:
> Thank you for your help until now. Almost there.
>
> https://pastebin.com/V72WNrem
>
> Does this look correct? I am getting duplicated maping key at line 16.

Alertmanager is right, you've got a duplicate "receiver:" line there.
This needs to be removed. Instead, you probably want to keep your
default receiver.

Also, the order of the following configuration seems wrong -- your old
receiver was outside of the receivers: hash.

Try the following:
https://pastebin.com/BBzzFrxA

Kind regards,
Christian

[Gabriel Balan]

Hi Christian,

One last thought, please :)

Checking 'alertmanager.yml'  FAILED: yaml: unmarshal errors:

  line 4: field routes not found in type config.plain

amtool: error: failed to validate 1 file(s)

It's referring to routes

[Christian Hoffmann]

Hi Gabriel,

On 10/4/19 11:07 AM, Gabriel Balan wrote:
> One last thought, please :)
>
> Checking 'alertmanager.yml'  FAILED: yaml: unmarshal errors:
>   line 4: field routes not found in type config.plain
>
> amtool: error: failed to validate 1 file(s)
>
> It's referring to routes

Line 5 looks like it should be removed. It causes the whole block to be
assigned to the value "route:" which does not exist at this level. It
also causes the "routes:" in the line before to be empty (instead of
containing the following block).

Kind regards,
Christian

[Gabriel Balan]

How about now?

https://pastebin.com/btwYJcNu

It still has a problem with line 4 even though line 5 is removed.

[Gabriel Balan]

Hi Christian,
Didn't mean to rechristen you to Simon earlier. Busy day at the office.

[Julien Pivotto]

On 03 Oct 16:35, Simon Pasquier wrote:
> You can combine all notifiers into one receiver like this:
> https://pastebin.com/Y4s17nps

I wonder how this would react if one receiver is not working. Would
AM resend the nofitication to the working receivers multiple times?

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAM6RFu4rLVKm0JaBfLWZBV%3DLrFRmCbEuiaCJUquS_yEBzG6TQw%40mail.gmail.com.

--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Simon Pasquier]

On Fri, Oct 4, 2019 at 1:43 PM Julien Pivotto <roidel...@inuits.eu> wrote:
>
> On 03 Oct 16:35, Simon Pasquier wrote:
> > You can combine all notifiers into one receiver like this:
> > https://pastebin.com/Y4s17nps
>
> I wonder how this would react if one receiver is not working. Would
> AM resend the nofitication to the working receivers multiple times?

No the notification log is per integration, not per receiver.

[Simon Pasquier]

The second "receiver" entry (line 11) should be aligned with the
preceding "match" entry. You miss a space character after "to:" and
"auth_password:" too.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1c33ef9f-631f-495d-ba5e-64ea673ce916%40googlegroups.com.

[Gabriel Balan]

Hi Simon,

I listened to your advice regarding and it's now aligned. 

Checking 'alertmanager.yml'  FAILED: yaml: unmarshal errors:

  line 4: field routes not found in type config.plain

amtool: error: failed to validate 1 file(s)

On Friday, 4 October 2019 14:01:54 UTC+1, Simon Pasquier wrote:

The second "receiver" entry (line 11) should be aligned with the
preceding "match" entry. You miss a space character after "to:" and
"auth_password:" too.

On Fri, Oct 4, 2019 at 1:43 PM Gabriel Balan <bala...@gmail.com> wrote:
>
> Hi Christian,
> Didn't mean to rechristen you to Simon earlier. Busy day at the office.
>
> https://pastebin.com/btwYJcNu
>
> It still has a problem with line 4 even though line 5 is removed.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Simon Pasquier]

the top-level entry for the routes is "route:" (singular).

> This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cab6cfa3-c676-4263-b75a-bc480caaa3a5%40googlegroups.com.

[Gabriel Balan]

Unfortunately not receiving any emails.

When I checked the config is only tells me that it detected two receivers.

Also, regarding the name of the instance, we have ports opened for telegraf and node_exporter. Which one has to be used? Maybe the problem is on my side.

Log wise, I have the following:

level=info ts=2019-10-04T15:24:06.444987121Z caller=main.go:140 msg="Starting Alertmanager" version="(version=0.17.0, branch=HEAD, revision=c7551cd75c414dc81df027f691e2eb21d4fd85b2)"

level=info ts=2019-10-04T15:24:06.445087744Z caller=main.go:141 build_context="(go=go1.12.4, user=root@932a86a52b76, date=20190503-09:10:07)"

level=info ts=2019-10-04T15:24:06.446021979Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=10.154.0.9 port=9094

level=info ts=2019-10-04T15:24:06.450928507Z caller=cluster.go:613 component=cluster msg="Waiting for gossip to settle..." interval=2s

level=info ts=2019-10-04T15:24:06.483734913Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=alertmanager.yml

level=info ts=2019-10-04T15:24:06.484482794Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=alertmanager.yml

level=info ts=2019-10-04T15:24:06.487596183Z caller=main.go:365 msg=Listening address=:9093

level=error ts=2019-10-04T15:24:06.487796714Z caller=main.go:367 msg="Listen error" err="listen tcp :9093: bind: address already in use"

level=info ts=2019-10-04T15:24:06.487955565Z caller=cluster.go:622 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.786384ms

level=info ts=2019-10-04T15:24:06.48813983Z caller=silence.go:386 component=silences msg="Creating shutdown snapshot failed" err="open data/silences.53c5172fc25ca14f: permission denied"

level=error ts=2019-10-04T15:24:06.488213526Z caller=nflog.go:372 component=nflog msg="Creating shutdown snapshot failed" err="open data/nflog.31184b399a70a79: permission denied"

It says the same thing about the port being already in use but I do not have two alertmanagers' running.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cab6cfa3-c676-4263-b75a-bc480caaa3a5%40googlegroups.com.

[Simon Pasquier]

On Fri, Oct 4, 2019 at 5:25 PM Gabriel Balan
<gabrie...@celer-tech.com> wrote:
>
> Unfortunately not receiving any emails.
>
> When I checked the config is only tells me that it detected two receivers.
>
> Also, regarding the name of the instance, we have ports opened for telegraf and node_exporter. Which one has to be used? Maybe the problem is on my side.
>
> Log wise, I have the following:
>
> level=info ts=2019-10-04T15:24:06.444987121Z caller=main.go:140 msg="Starting Alertmanager" version="(version=0.17.0, branch=HEAD, revision=c7551cd75c414dc81df027f691e2eb21d4fd85b2)"
> level=info ts=2019-10-04T15:24:06.445087744Z caller=main.go:141 build_context="(go=go1.12.4, user=root@932a86a52b76, date=20190503-09:10:07)"
> level=info ts=2019-10-04T15:24:06.446021979Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=10.154.0.9 port=9094
> level=info ts=2019-10-04T15:24:06.450928507Z caller=cluster.go:613 component=cluster msg="Waiting for gossip to settle..." interval=2s
> level=info ts=2019-10-04T15:24:06.483734913Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=alertmanager.yml
> level=info ts=2019-10-04T15:24:06.484482794Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=alertmanager.yml
> level=info ts=2019-10-04T15:24:06.487596183Z caller=main.go:365 msg=Listening address=:9093
> level=error ts=2019-10-04T15:24:06.487796714Z caller=main.go:367 msg="Listen error" err="listen tcp :9093: bind: address already in use"
> level=info ts=2019-10-04T15:24:06.487955565Z caller=cluster.go:622 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.786384ms
> level=info ts=2019-10-04T15:24:06.48813983Z caller=silence.go:386 component=silences msg="Creating shutdown snapshot failed" err="open data/silences.53c5172fc25ca14f: permission denied"
> level=error ts=2019-10-04T15:24:06.488213526Z caller=nflog.go:372 component=nflog msg="Creating shutdown snapshot failed" err="open data/nflog.31184b399a70a79: permission denied"
>
> It says the same thing about the port being already in use but I do not have two alertmanagers' running.

Well if it tells that it can't bind to the port then you still need to
find out the reason because Alertmanager can't start until it is
fixed.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d84b43fc-51d9-4632-84e0-1504293d27e1%40googlegroups.com.