Change Prometheus initial conf

[Andy Andy]

Hello. I got some troubles with scrape config. Well, i got openshift cluster with Prometheus. When i use command oc edit prometheus to add some scrape config its okay next few minutes, but after that, something reset my prometheus conf and its still be like original, until my additions. What is that and how can i change configuration? 

 

[Ian Billett]

Hey Andy Andy,

Sounds like your issue lies somewhere between the prometheus-operator and OpenShift - I'm afraid that this mailing list is focused on the Prometheus server itself so you would be very lucky to find someone with the answer here :)

Best,

Ian

On Tue, Jul 27, 2021 at 3:44 PM Andy Andy <kreep...@gmail.com> wrote:

Hello. I got some troubles with scrape config. Well, i got openshift cluster with Prometheus. When i use command oc edit prometheus to add some scrape config its okay next few minutes, but after that, something reset my prometheus conf and its still be like original, until my additions. What is that and how can i change configuration? 

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/83dd19be-729d-4491-93f6-b7ee61490cf2n%40googlegroups.com.

[Brian Gottfried]

Ian, if you're referring to the default Prometheus that ships with Openshift (inside the openshift-monitoring namespace), the scrape config for that is controlled by the Cluster Monitoring Operator and only has limited customization options. The Openshift documentation details what options are available here: https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.html#preparing-to-configure-the-monitoring-stack. If you can clarify what you're trying to modify in the scrape config, I may be able to answer whether it's possible or not.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CADoMBNWHFsC7JM%2BOs2pig-rEiK%3Dn6a3OD%2Bn8MrtOB%2Bo%3DVgknBQ%40mail.gmail.com.

[Andy Andy]

Thank you for answer. I got CRD called prometheuses.monitoring.coreos.com , and i suppose what this item contains main config for prometheus. Because when i change it via oc edit prometheus, its show configuration. After that i can edit config and i add in spec section next part ( I setup blackbox exporter, i need to check my routes in openshift and i choose this way(i mean use blackbox) to do that. And for this reason i need to add additional scrape config to main prometheus config. ) : 

spec:

   additionalScrapeConfigs: 

      name: additional-scrape-configs 

      key: prometheus-additional.yaml
Well, after that all ok. Prometheus reload config with new parameters. I go to blackbox routes and i can see new targets for scrape, next i look at prometheus config, and i also can see what config was changed. I see new job called like "blackbox". Its seems like everything works, but after 1 or 2 minutes this changes are disappear. Prometheus rollback changes. Its looks like something looking at prometheus.env.yaml and if something change this thing is rollback config to template. I can suppose what its template located in secret maybe.  

четверг, 29 июля 2021 г. в 18:48:19 UTC+3, bgot...@redhat.com:

[Brian Gottfried]

Andy, you are correct that the Prometheus CRD is used to determine the configuration for the cluster Prometheus - specifically, the k8s CR instance is what defines the config for the cluster Prometheus. This CR is created and managed by the Cluster Monitoring Operator, so you cannot change it manually - the Cluster Monitoring Operator will overwrite any manual changes to keep the Prometheus instance in the correct state. There are ways to modify this CR, but it's done by modifying the cluster-monitoring-config ConfigMap and there are a limited number of modifications that can be made via this method currently.

I'm not familiar with the Blackbox Exporter, so I can't tell you off the top of my head whether the required changes to enable it are possible by modifying the cluster-monitoring-config ConfigMap - I would recommend you contact Openshift support if you have access to it, as they may be familiar with it and be able to tell you whether it's possible or not.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8222c7fa-27b4-46e4-a8f7-720a482af27fn%40googlegroups.com.