I run a large internal datacenter that hosts thousands of instances of OpenShift, each of which has two instances of Prometheus running. Looking into a performance issue that we have been experiencing, we noticed the Prometheus always writes new blocks/chunks exactly on the hour. When this is multiplied across thousands of potential instances we think it may have a performance impact to the underlying hosting environment.
Is there any method to time shift Prometheus' write time so that it does not always land exactly at the top of the hour?
Any help appreciated.