Prometheus "debug" log shows me that I have some "Scrape fail", but don't tell me on which Pod / Container

26 views
Skip to first unread message

Jean-Michaël Cyr

unread,
Jun 4, 2020, 9:20:47 AM6/4/20
to Prometheus Users
Hello guys

So I can see a bunch of these  in my logs

level=debug ts=2020-06-04T13:05:21.739Z caller=scrape.go:962 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.4.15.232:8081/actuator msg="Scrape failed" err="Get http://10.4.15.232:8081/actuator: dial tcp 10.4.15.232:8081: connect: connection refused"
level=debug ts=2020-06-04T13:05:25.806Z caller=scrape.go:962 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.4.9.226:8081/actuator/prometheus msg="Scrape failed" err="server returned HTTP status 403 "

These are probably java applications that are not well configured, but prometheus doesn't tell me which one. 
I have to start a big research, base on the server IP, lookup which container reside on that node, investigate each on these projects (and can't even find the issue lol).

Is there a better way ? Can prometheus tell me which container is actually faulty ? 
How I can know exactly where to look ? 


Thanks for helping ! 

Reply all
Reply to author
Forward
0 new messages