Prometheus operator monitoring pods in different namespaces

[Tim Blackwell]

When running the examples in default namespace, all is good.
I tried to create the prometheus operator and prometheus stateful set in a monitoring namespace but then it was unable to monitor the example app.

When running the prometheus / operator in monitoring namespace, I updated the service account yaml accordingly.

Is it possible for prometheus / operator to monitor and collect metrics from different namespaces? If so what should I change to get this behaviour?

# Default Namespace

## Operator

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-service-account.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-cluster-role-binding.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus-operator/prometheus-operator-deployment.yaml

## Prometheus

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-service-account.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-cluster-role.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus-cluster-role-binding.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/rbac/prometheus/prometheus.yaml

## Example app

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-deployment.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-service-monitor.yaml

kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/user-guides/getting-started/example-app-service.yaml

# Monitoring Namespace

kubectl create namespace monitoring

## Operator

kubectl --namespace monitoring create -f example/rbac/prometheus-operator/prometheus-operator-service-account.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus-operator/prometheus-operator-cluster-role-binding.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus-operator/prometheus-operator-deployment.yaml

## Prometheus

kubectl --namespace monitoring create -f example/rbac/prometheus/prometheus-service-account.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus/prometheus-cluster-role.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus/prometheus-cluster-role-binding.yaml

kubectl --namespace monitoring create -f example/rbac/prometheus/prometheus.yaml

## Example app

kubectl create -f example/user-guides/getting-started/example-app-deployment.yaml

kubectl create -f example/user-guides/getting-started/example-app-service-monitor.yaml

kubectl create -f example/user-guides/getting-started/example-app-service.yaml

[Tim Blackwell]

I think I might have found my issue:

While ServiceMonitors must live in the same namespace as the Prometheus resource, discovered targets may come from any namespace. This allows cross-namespace monitoring use cases, for example, for meta-monitoring. Use the namespaceSelector of the ServiceMonitorSpec to restrict the namespaces from which Endpoints objects may be discovered.

From the generic service monitor guid https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md#namespace-limitsthings-to-keep-in-mind