ALERTMANAGER NOT RUNNING

[Chinelo Ufondu]

I tried running alertmanager again and i came across this issue, here is the error

ts=2024-09-01T17:35:52.421Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)"
ts=2024-09-01T17:35:52.421Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)"
ts=2024-09-01T17:35:52.440Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=192.168.101.2 port=9094
ts=2024-09-01T17:35:52.441Z caller=main.go:221 level=error msg="unable to initialize gossip mesh" err="create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 9094: listen tcp 0.0.0.0:9094: bind: address already in use"

I have tried all i can to stop the processes that is currently running  on alert manager, but it didn't work out, i also tried adding an external command to run alertmanager --web.listen-address=localhost:9095 --config.file=alertmanager.yml, but it still isn't picking the new port number i would appreciate further assistance from you guys please, Thank you.

[Julius Volz]

Then you probably have something else already running on port 9094.

On Linux, you should be able to see it via either:

$ netstat -tlpen

or:

$ ss -tlnpe

Regarding the flag --web.listen-address, that is for the web interface. To change the port for the cluster mesh listener, you'll need to use --cluster.listen-address.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAL2E%2B826K8SbAM%3DA2q%3DUk7WRf853J%2B69R-B1zZCX%2BvQpXyQyiA%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

[Chinelo Ufondu]

Hello 

i have tried again by running this command like you suggested and specifying a port that the clusters should listen on  alertmanager --config.file=alertmanager.yml --cluster.listen-address=0.0.0.0:8081, and i got a different error saying port 9093 is already in use, and port 9093 is the default port alertmanager is currently listening on

ts=2024-09-10T13:10:36.310Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)"
ts=2024-09-10T13:10:36.310Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)"
ts=2024-09-10T13:10:36.325Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=192.168.101.2 port=8081
ts=2024-09-10T13:10:36.326Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2024-09-10T13:10:36.359Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=alertmanager.yml
ts=2024-09-10T13:10:36.359Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=alertmanager.yml
ts=2024-09-10T13:10:36.360Z caller=main.go:394 level=info component=configuration msg="skipping creation of receiver not referenced by any route" receiver=send_email2
ts=2024-09-10T13:10:36.363Z caller=main.go:517 level=error msg="Listen error" err="listen tcp :9093: bind: address already in use"
ts=2024-09-10T13:10:36.365Z caller=cluster.go:692 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=39.047022ms

[Brian Candler]

alertmanager listens on two ports. By default:

--web.listen-address=:9093

--cluster.listen-address=0.0.0.0:9094

[Chinelo Ufondu]

Yes, but i do not know why when trying to start Alertmanager it tells me the port is already in use and can’t start.

I was able to change the default port to 9095 on Alertmanager.service file, and specify this command on run

alertmanager --config.file=alertmanager.yml --cluster.listen-address=0.0.0.0:8081 - - web.listen-address= 0.0.0.0:9095 

It ran successfully, then I checked my log file and saw an http2 error, meaning Alertmanager is till using its default port. I also tried accessing Alertmanager via my web interface with the new port no attached, nothing showed up 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b8ff032c-990d-4870-9bec-2d3abecf843fn%40googlegroups.com.

[Brian Candler]

> Yes, but i do not know why when trying to start Alertmanager it tells me the port is already in use and can’t start.

It's because there's an instance of alertmanager already running. (*)

This is not really a question about prometheus or alertmanager; it's a general system administration question. It all depends on how alertmanager was originally installed on your system, and whether it's running under some sort of supervisor process, and if so what that supervisor is.  For example, it's possible to run alertmanager under systemd, in which case you'd use systemd commands to start and stop it. But that configuration is not supplied as part of alertmanager; it's something that a third party would have added, perhaps when packaging it up.

So the answer depends entirely on the details of your system.  You might want to find a local system administrator who can help you identify how alertmanager was originally installed and configured.

(*) Or possibly it could be some other software listening on ports 9093 and 9094. Either way, you need to identify what that process is. Julius gave you some commands as a starting point to help identify that process.

[Chinelo Ufondu]

Hello
I have been able to figure it out, the issue is that there was no issue😃, alertmanger was running on supervisor process which is systemctl, of which there was no need to manually start alertmanager. Since i was trying to troubleshoot an email issue, i just had to systemctl stop alertmanager,  then manually start it again by doing alertmanager  --config.file=/etc/alertmanager/alertmanager.yml and it worked out fine afterwards with no errors.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/370d776c-5fa1-47d7-8389-6c0bc26a4881n%40googlegroups.com.

[Brian Candler]

Thank you for coming back with the solution.