Prometheus is able to return metrics, but the host itself can't curl the endpoint

[Kristopher Kahn]

I'm able to use PromQL and return metrics in the dashboard for endpoint.example.com:9100 but when actually on the Prometheus host and running curl endpoint.example.com:9100/metrics nothing is returned and the request just times out.

What might be causing this behavior? I looked through the Prometheus config but could not see anything special about what DNS resolver it's set to use. I would imagine it's using the system-wide resolvers. This is a systemd host - Ubuntu 18.04.1.

[Kristopher Kahn]

Would anyone happen to know what is causing this behavior? up() PromQL statements return 0 for the hosts in question, but they are all scraping metrics successfully.

I don't believe I see any service discovery or pushes from the host, these were all set up traditionally to be scraped from Prometheus.

[Brian Candler]

That's a contradiction.  If up is 0, it means that scraping failed, so you can't have any metrics.

You can query the local Prometheus API to prove this one way or the other:

promtool query instant http://localhost:9090 'up{job="node"}'   # adjust job name as required to match your scrape job
promtool query instant http://localhost:9090 '{job="node"}'

Or if you don't have promtool, you can get something similar using curl:

curl -g 'http://localhost:9090/api/v1/query?query=up{job="node"}' | python3 -m json.tool

I find it highly unlikely that you'll find the same instance with up=0 but with metrics collected.

Also: if you see up=0, you should also see the scraping problem in the prometheus web UI.  Go to http://x.x.x.x:9090 and select Status > Targets.

I think that up=0 is consistent with the problem you originally reported, i.e. being unable to scrape the targets directly using curl from the prometheus server.  If you're seeing metrics somewhere else, then my only guess is that you're looking at a completely different prometheus server.

[Kristopher Kahn]

Yes I understand that if up returns 0, that means scraping failed. That was inconsistent with the behavior I was seeing on the Prometheus web UI, where it was still returning metrics from the server even though it was returning 0 for up(). It didn't make any sense.

After Prometheus was restarted yesterday afternoon the behavior is now as expected and the metrics accurately reflect that the host can't be reached and can't be scraped.

[Brian Candler]

So now the case is simply that the target exporter is not reachable from the prometheus server (as demonstrated by curl). 

This is not a prometheus problem, it's a network problem that you have to debug.  Maybe it involves network ACLs, firewall rules, iptables on the remote host etc.