Customize Prometheus stack config in ClusterMonitoringOperator - inject custom config in ansible

[dominik....@gmail.com]

Hello,

I've installed ClusterMonitoringOperator (CMO) from openshift-ansible. It works great!

Now I have to add some custom configurations :

1. labels in PVCs to select labeled PVs

2. reconfigure Prometheus and Alertmanager statefullsets (antiaffinity to ensure placing in multiple AWS availibility zones)

3. reconfigure Prometheus CRD configuration (serviceMonitorSelector, ruleSelector, serviceMonitorNamespaceSelector, ruleNamespaceSelector, ...)

Currently I'm trying to achieve that by additional post-installation tasks in ansible playbook. I'm trying to patch all objects with my custom configuration and reload them one more time.

My questions are:

A. I need to stop reloading to default. I was trying to set "paused: true" parameter in PrometheusOperator and Prometheus itself, no success up to now. 

I'm confused with documentatiotion [1]:

"It is possible to pause this behavior, by setting the paused field in the AppVersion called openshift-monitoring"

Where (on which component of CMO) should "paused" filed be configured to stop reloading Prometheus?

B. currently I can change volumeClaimTempalte before starting playbook, by adding/modyfing particular lines before running default playbook to address configurations described  in points 1 and 2. is there more elegant way to do that?

C. how can I change configuration with configmap/secrets in scope of CMO ? It's mentioned in Prometheus-Operator docs [3]. I assume I should have paused:true parameter for entire CMO and then somehow replace secrets? It could be more elegant solution than patching objects, but default re-loader is blocking me from further testing.

D. CMO was officially released few days ago in Openshift 3.11 release, documentation [2] is still not so detailed in terms of custom configuration, it is only mentioned that:

"Beyond those explicit configuration options, it is possible to inject additional configuration into the stack. However this is unsupported, as configuration paradigms might change(..)" 

Any instructions how to "inject additional configurations" will be helpful. If you could provide links to examples with custom configurations similar to my requirements, they will be greatly appreciated as well.

I hope that is correct mailing group that I'm using currently to address my questions.

Regards,

Dominik

Links:

[1] https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/configuring_clusters/prometheus-cluster-monitoring#using-resources-created-by-openshift-container-platform-monitoring

[2]  https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html

[3] https://github.com/coreos/prometheus-operator/blob/master/Documentation/custom-configuration.md

[dominik....@gmail.com]

Hello,

further investigation showed that during developing process of Tectonic Prometheus Operator, AppVersion object was created in kubernetes. Example command related to this is mentioned in prometheus-operator GitHub issues section [1].

In Openshift, this Object is not visible (tested with openshift-ansible 3.10, 3.11). Operators control state of objects using immutable ControllerRevision objects, see below:

$ oc get AppVersion

the server doesn't have a resource type "AppVersion"

$ oc get ControllerRevision

NAME                          KIND

alertmanager-main-b99b6c78c   ControllerRevision.v1.apps

node-exporter-5fc4d87798      ControllerRevision.v1.apps

prometheus-k8s-6d4cd9b8fb     ControllerRevision.v1.apps

As for now I was able to stop resetting Prometheus to default state by manually deleting ClusterMonitoringOperator and PrometheusOperator instances (and then reconfigure CRDs with my configuration). This is probably not the most elegant way to provide custom configuration, another solutions (which will not require deletion of Operators) will be greatly appreciated.

Regards.

Dominik

[1] https://github.com/coreos/prometheus-operator/issues/483