node exporter not listening externally Ubuntu 22.04

[Nicholas Reid]

Has anyone else had trouble getting the node exporter working on Ubuntu 22.04.

I've followed the instructions here: https://ourcodeworld.com/articles/read/1686/how-to-install-prometheus-node-exporter-on-ubuntu-2004

I have also tried installing the prometheus-node-exporter package from the ubuntu repo using apt-get

The service starts up OK and I can curl it locally (localhost:9100/metrics). However, when I try to access it remotely I'm not able to connect on 9100, or any port I set it to listen on . 

I've confirmed that ufw is inactive, and the service has bound to port 9100 with netstat .There is also no other firewall that could be blocking this port. I have tried connecting from the same subnet. I've confirmed the remote traffic is reaching the server with tcpdump.

I have tried installing multiple versions of the node exporter.

I'm able to ping and SSH to the server running the node exporter. I also have docker containers running that I can connect to from a remote host. It seems like this might be a GO / Linux issue

Is there something else I might have missed?

I haven't have this issue with Ubuntu18.04 .

If anyone has any insights I would greatly appreciate this as I've been stuck on this for a while and I'm sure this is supposed to work 

Nick

[Brian Candler]

On Wednesday, 1 March 2023 at 11:11:19 UTC Nicholas Reid wrote:

Has anyone else had trouble getting the node exporter working on Ubuntu 22.04.

Nope, it works perfectly for me using the release binaries from github.

There's another guide you can look at here, and see if you did anything differently: https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-node-exporter.html

 

The service starts up OK and I can curl it locally (localhost:9100/metrics). However, when I try to access it remotely I'm not able to connect on 9100, or any port I set it to listen on . 

What does "ps auxwww | grep node_exporter" show on the host where node_exporter is running?

What does "iptables -L -n -v" on the host where node_exporter is running show?

What does "telnet x.x.x.x 9100" from a remote host show? (where x.x.x.x is the host where node_exporter is running)

Also: try running "tcpdump -i eth0 -nn tcp port 9100 or icmp" on the node_exporter host (replace 'eth0' with the actual network interface), *then* run "telnet x.x.x.x 9100" from a remote host. What *exactly* does tcpdump show?  The details of this will help determine the problem.

(e.g. do you just see SYN packets arriving? Do you see SYNs followed by ICMP messages, if so what exactly is the ICMP type/code? Do you see SYNs followed by SYN ACK responses, but repeated SYNs?)

I've confirmed that ufw is inactive, and the service has bound to port 9100 with netstat .

It will need to be bound to :::9100 or 0.0.0.0:9100 to accept connections from outside on all interfaces.  If it's bound to a single interface like 127.0.0.1:9100 then it will only accept connections on that address.

[Brian Candler]

The OP has replied to me off-list with the solution (there seems to be some problem with Google auto-deleting replies)

> It turns out firewalld was enabled. 
> I ran  systemctl disable --now firewalld and this worked.

[Michael Kogelman]

Bites us all at some point or another. 

Get Outlook for iOS

---

From: promethe...@googlegroups.com <promethe...@googlegroups.com> on behalf of Brian Candler <b.ca...@pobox.com>
Sent: Wednesday, March 1, 2023 9:25:49 AM
To: Prometheus Users <promethe...@googlegroups.com>
Subject: [prometheus-users] Re: node exporter not listening externally Ubuntu 22.04

 

The OP has replied to me off-list with the solution (there seems to be some problem with Google auto-deleting replies)

> It turns out firewalld was enabled. 
> I ran  systemctl disable --now firewalld and this worked.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cd778eb4-60ad-482b-8764-e4026fae751cn%40googlegroups.com.

[Nicholas Reid]

I’m glad I’m in good company. Very lucky to have this user group dig me out. When I get stuck 

Sent from Outlook for iOS

---

From: promethe...@googlegroups.com <promethe...@googlegroups.com> on behalf of Michael Kogelman <michael....@azuleng.com>
Sent: Wednesday, March 1, 2023 8:51:10 PM
To: Brian Candler <b.ca...@pobox.com>; Prometheus Users <promethe...@googlegroups.com>
Subject: Re: [prometheus-users] Re: node exporter not listening externally Ubuntu 22.04

 

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/WK9GWrvfeDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/DM5PR07MB4133296EFF410D32BB69C83EF1AD9%40DM5PR07MB4133.namprd07.prod.outlook.com.