Automating Prometheus config management

[Juergen Etzlstorfer]

Dear community,

we are working on an integration for our open-source project Keptn to automatically setup and configure scrape jobs and alerting rules for Prometheus based on SRE best-practices.

Our current implementation lacks a bit of flexibility in the sense that it can only configure it if it also manages the Prometheus deployment. As we have discovered that deploying Prometheus with the operator is the preferred way (right?) we are going to change this behavior.

We would appreciate it if there is someone from the community that can give some feedback and ideas on how to approach this. 

1. Assuming Prometheus is running in the same cluster as our control plane (Keptn) - is it safe to rewrite yaml files (or ServiceMonitor/PodMonitor) and store them in secrets to change the Prometheus configuration or is there another approach to update configs like scrape jobs and alerting rules?

2. How to make sure Prometheus picks up the new configuration? Is this done by the Operator?

Our implementation and the issues where we are working on this integration can be found here: https://github.com/keptn-contrib/prometheus-service/issues/53 

There core Keptn project that makes use of this integration can be found here: https://github.com/keptn/keptn  

Thanks for your help!