k8 pod targets are not shown in UI

[Karan Chaudhary]

Hello folks,

I'm want to connect prometheus running locally on my machine to k8 cluster running somewhere in a data center. For prometheus to talk to the k8 cluster, i give it the kubeconfig that is used by local kubectl, via "kuernetes_sd_config.kubeconfig_file" config param.

This failed because oidc auth provider, that is required by kubeconfig, is not supported in prom. It is easily fixed by importing "_ "k8s.io/client-go/plugin/pkg/client/auth/oidc" in discovery pkg.

The problem I face now is that the targets are not visible in the ui even when prom has successfully discovered them and is also successfully scraping them. This is confirmed by logs and tcpdump.

Prom version = 2.51. Prom config is pretty minimal and is shared in message below.

It'd be great if someone could help here. Thanks!

[Karan Chaudhary]

It so happened that there was another prometheus instance, installed by brew, running on port 9090 that i was accidently inspecting while making changes to dev copy of promethes. Its odd that conflicting listening port error was not raised by custom built prometheus that also started on port 9090. But thats for another day.