Run two node exporters on same server

[BHARATH KUMAR]

Hii all,

I want to run two node exporter on same server.. One is with kubernetes and another one with binary file. each node exporter has different port numbers. I am getting error like time out or context deadline exceeded. Will it be possible if so sould you please tell me the solution?

thanks 

Bharath 

[Brian Candler]

You need to be clearer about what you're doing and what errors you see.

Yes, you could run two node_exporters on the same server, bound to different ports.  However this is normally completely pointless, since they're both monitoring the same server, and exporting the same data.

It is not recommended that you run node_exporter inside a container at all.  But if you do, see the instructions here:

https://github.com/prometheus/node_exporter#docker

[BHARATH KUMAR]

Hiii

Thanks for your reply. we were using two different ports for two node exporters. Actually one node exporter is running in container/pod and another node exporter is running as a binary file. we are not getting any issue with container/pod one. But when we are running with binary one we are getting error like context deadline exceeded.

our prometheus config file is as follows:

scrape_interval: 2m
evaluation_interval: 15s

scrape_timeout: 2m

could anyone please help me out of this?

regards,

Bharath.

[Brian Candler]

> we are getting error like context deadline exceeded

You haven't show the actual error, nor where you saw it.

The most likely explanation I can see is simply that prometheus cannot communicate with node_exporter - for example, you've misconfigured the target or there is some sort of firewalling in between.

To prove this, login to the prometheus server (or container where prometheus is running), and do:

curl -h 'http://x.x.x.x:yyyy/metrics'

where x.x.x.x:yyyy is the IP address and port that you've configured as the target to scrape.

[BHARATH KUMAR]

If I do curl in the particular server where I installed n=binary node exporter I am able to see metrics but in browser and grafana and in prometheus UI I am not able to see metrics. It is showing CONTEXT DEADLINE EXCEEDED.

thanks 

Bharath

[Brian Candler]

"Context deadline exceeded" simply means "timeout waiting to connect or receive data"

It sounds to me like you have a network connectivity problem between the client (i.e. prometheus) and wherever the binary node exporter was installed.  Talk to a local network administrator or system administrator to help you find where the problem is.

The best way to reproduce this would be to run the same curl command on the prometheus server itself.

If prometheus is running inside a container, then run the curl command inside that container.  If prometheus is running inside a kubernetes pod, then see here - you may need to add an ephemeral debugging container to your pod to be able to use 'curl', if your container image doesn't already have it.

[BHARATH KUMAR]

thanks for your reply. I think we fixed some firewall issues and now working fine for most servers. But still we are facing new error like 

Get "http://some_ip:port_number/metrics": dial tcp some_ip:port_number: connect: connection refused

what could be the reason for this error?

thanks 

Bharath

[Stuart Clark]

On 2022-04-21 16:03, BHARATH KUMAR wrote:
> thanks for your reply. I think we fixed some firewall issues and now
> working fine for most servers. But still we are facing new error like
>
> Get "http://some_ip:port_number/metrics": dial tcp
> some_ip:port_number: connect: connection refused
>
> what could be the reason for this error?
>

That generally means that the connection is passing through the
firewalls ok but the end server is then rejecting it. Usually because
the port number is wrong or the service attached to that port isn't
running. For containers it could mean the port hasn't been exposed to
the outside host.

--
Stuart Clark

[Brian Candler]

Could mean many things:

- node_exporter isn't running on the target host

- node_exporter is listening on a different port than the one you're trying to connect to

- it is listening on the wrong IP address or interface (e.g. if bound to 127.0.0.1 then it won't accept connections from outside)

There are also some types of firewall which can block traffic in this way, making it look like connection refused.

[BHARATH KUMAR]

Thanks for your reply. I understand where it went wrong. Now its working fine. I will try to install on few more servers and we will monitor.

Thanks 

Bharath