Scrape config w/ Prometheus operator not upheld

[Bastien Moyet]

Hi,

I deployed a prometheus instance using prometheus operator (8.16.1) alongside a pushgateway (1.4.3) in a kubernetes cluster. But I cannot access my pushgateway metrics from prometheus.

My pushgateway is correctly deployed, accessible and its data can be viewed on its UI.

Prometheus is up and running too. However I cannot find the pushgateway in the listed target on http://prometheus:9090/targets . That seemed logic at first since the http://prometheus:9090/config page does not show any pushgateway either.

However if I go check the config file I can find in http://prometheus:9090/flags (/etc/prometheus/config_out/prometheus.env.yaml) it gives me differents values than the one I can find on the UI.

Notably in my pod the config file contains :

- honor_labels: true
  job_name: pushgateway
  static_configs:
  - targets:
    - prometheus-pushgateway:9091

So i'm wondering why the config files differs from the config displayed on the UI. Is it a normal behavior or does it explain my issue ?

Thanks

[Bastien Moyet]

Otherwise, are there any logs I can find regarding which endpoints are effectively scrapped, or if there is an error scrapping them ? Though I suspect there is no error since my prometheus gateway is not showing up in targets list.

[Ian Billett]

Hi Bastien,

If your prometheus instance is correctly configured to scrape metrics from push_gateway, you should be able to see this in the Prometheus > Status > Targets UI. It should look 'healthy' like the following from the demo prometheus:

Best,

Ian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com.

[Bastien Moyet]

Hi,

Thanks for your answer.

That's what I thought too, but I'm not seeing any push_gateway in the UI, neither healthy nor unhealthy. As if prometheus wasn't using my configuration.

That's why I'm asking my question here, and if there is a way to see why my configuration is not used.

[Ian Billett]

Bastien,

Hmm interesting. At this point it would be helpful if you could share some more information so that we can help you figure out what is wrong here - I would be interested to know what configuration is reported in the Configuration UI, and what is being reported in the Targets UI. The logs from Prometheus itself almost always contains useful information too.

Best,

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com.

[Bastien Moyet]

Sure,

You can find both config on the following gist : https://gist.github.com/bmoyet/a3438a881d12caa4b4b9e2b109cdeda8

I added config_ui.yaml which is what the UI is reporting to me. And prometheus.env.yaml which is what you can find in the prometheus config path (/etc/prometheus/config_out/prometheus.env.yaml). As you can see some job defined in the server config (push_gateway, istio) are absent from the UI config.

Targets UI is reporting only the jobs defined in the Config UI, which is htmltoimage-metrics, htmltoimage-metrics, htmltoimage-metrics and a whole lot of monitoring/xxx

In the logs I have from the docker container (kubectl logs...) I cannot find any relevant information... Mainly some rule manager warning and some info reports like :

level=info ts=2021-07-06T09:00:48.938Z caller=compact.go:495 component=tsdb msg="write block" mint=1625551200000 maxt=1625558400000 ulid=01F9XGQAQGF5GNVQYNXQEC3ASW duration=48.826038743s
level=info ts=2021-07-06T09:01:03.783Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=4.365682297s
level=info ts=2021-07-06T09:01:26.172Z caller=head.go:734 component=tsdb msg="WAL checkpoint complete" first=75464 last=75475 duration=22.389155689s
level=info ts=2021-07-06T09:02:14.296Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1625529600000 maxt=1625551200000 ulid=01F9XGT72PP4D2N8FHJZ2606VC sources="[01F9WW4519JMZAVQTXABY99TD5 01F9X2ZWA131TPVJBT6BNYQGR5 01F9X9VKJ1VFRNRWM87AY209PB]" duration=39.617840386s

If I can supply any other information that might help I will be glad to do so.

[Ian Billett]

Thanks for the comprehensive information!

> As you can see some job defined in the server config (push_gateway, istio) are absent from the UI config.

In general, the config UI should show exactly the same configuration as contained in your prometheus.env.yaml. Perhaps you just need to ask Prometheus to reload its configuration?

I am not an expert in the Prometheus Operator, but if you are making modifications to a file that the operator expects to control, then they may be reasonably overwritten. The operator expects to be configured by PodMonitor and ServiceMonitor-like kubernetes objects.

If reloading the config does not give you results you are expecting then it may be an idea to raise with #prometheus-operator, who will be able to give more specific advice.

Best,

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com.

[Bastien Moyet]

Thanks for your input.

I tried to reload the configuration to no avail. You may be right that's an issue with prometheus operator I'll ask them.