Prometheus Operator 0.38.1 not reconfiguring prometheus on new PodMonitors

[Roberto Camelk]

Hi. 

I'm using Apache Camel-K to run integration logics.

Camel-K is runing inside a Rancher 2.4.3 in k8s, in the namespace "platform".

This Rancher version has a embedded prometheus stack on it enabled. 

So, there is a Prometheus Operator 0.38.1 running...

This prometheus stack is at the namespace "cattle-prometheus".

So the first thing I done was configure the Prometheus resource for the operator, adding to it this:

    podMonitorNamespaceSelector: {}

    podMonitorSelector:

      matchExpressions:

      - key: camel.apache.org/integration

        operator: Exists

This tells to the operator that I want scan all namespaces for PodMontiors with the label key: camel.apache.org/integration.

So, when I start a Camel-K integration, a new pod is created, as so a PodMonitor resource to describe to the Prometheus Operator where Prometheus Server can scrape metrics of my integration pod.

Is expected, in a while, that the Prometheus Operator discover this PodMonitor and reconfigure the Prometheus Server to scrape it.

Here is where I'm getting in trouble...

The prometheus operator is discovering that a new PodMontior and logging this:

level=debug ts=2021-11-30T16:42:55.490845245Z caller=operator.go:734 component=prometheusoperator msg="PodMonitor added"

level=debug ts=2021-11-30T16:42:57.017240979Z caller=operator.go:748 component=prometheusoperator msg="PodMonitor updated"

level=debug ts=2021-11-30T16:43:13.978452732Z caller=operator.go:748 component=prometheusoperator msg="PodMonitor updated"

level=debug ts=2021-11-30T16:43:14.207084511Z caller=operator.go:748 component=prometheusoperator msg="PodMonitor updated"

But nothing occurs on the Prometheus Server. It does not start to scrape my pod metrics.

BUT, if I restart the Prometheus Operator, it start to scrape it (if my pod is already running). So, after restart the operator at got this log:

level=debug ts=2021-11-30T13:11:50.399104614Z caller=operator.go:1840 component=prometheusoperator msg="filtering namespaces to select PodMonitors from" namespaces=kube-node-lease,weave,cattle-prometheus,infrastructure,kube-system,security-scan,cattle-system,kube-public,platform,poc,default,aplicativos namespace=cattle-prometheus prometheus=cluster-monitoring

level=debug ts=2021-11-30T13:11:50.399141448Z caller=operator.go:1853 component=prometheusoperator msg="selected PodMonitors" podmonitors=platform/micro-profile-metrics2,platform/create-role-route namespace=cattle-prometheus prometheus=cluster-monitoring

I'm also reporting this at the prometheus operator project:

https://github.com/prometheus-operator/prometheus-operator/issues/4428

But I'm stuck, I need to solve this... Can someone help me?

Anyone knows why the operator isn't firing the regemeration of the prometheus config to scrape my new pods?

[Roberto Camelk]

Solved. 

I just upgraded my operator to versão 0.39.0.