Get "https://127.0.0.1:10001/metrics": dial tcp 127.0.0.1:10001: connect: connection refused,

[Bertalan Balázs]

HI, my prometheus endpoint always down, however the data exist on the endpoint . Can help me someone with this issue? I tried with IPv4 and localhost also..

[Brian Candler]

It's telling you very clearly: you have nothing listening on port 10001 on localhost.  Therefore, there is nothing for prometheus to connect to.

ssh into your prometheus server or container, and try:

curl -v https://127.0.0.1:10001/metrics

and you'll get the same answer.

Note: it is usually wrong to be using https for connections to 127.0.0.1, but that's not your problem here.

[Bertalan Balázs]

I get it, but if i click on the link its give me the expected data, 'curl -k -v https://127.0.0.1:10001/metrics' works properly.

[Brian Candler]

In that case, it seems that prometheus is running on a different machine (or in a different VM or container) than curl and the exporter.

On the prometheus machine, try:

tcpdump -i lo -nn -s0

and you'll see traffic to/from 127.0.0.1.  Capture the tcpdump output when prometheus tries to do its regular scrape (e.g. every minute)

Then try the curl command again, and again capture the tcpdump output.

Compare the two tcpdump results.

[Bertalan Balázs]

I can't catch the regualar scrape in the output but if i try curl i can catch it.

19:03:25.942567 IP 127.0.0.1.10001 > 127.0.0.1.47224: Flags [F.], seq 18714, ack 942, win 512, options [nop,nop,TS val 3544202431 ecr 3544202431], length 0
19:03:25.942585 IP 127.0.0.1.47224 > 127.0.0.1.10001: Flags [.], ack 18715, win 512, options [nop,nop,TS val 3544202431 ecr 3544202431], length 0

[Brian Candler]

That implies strongly to me that prometheus is actually running on a different server (or in a different VM or container) than where you are running tcpdump and curl.

127.0.0.1 is a special address for "localhost" - it can only be used to connect to something on the same device (or in the case of containers, in the same network namespace).