Cross Namespace Monitoring using namespaceSelector with kube-prometheus manifests

[RakV1342]

Based on my query on GitHub: https://github.com/coreos/prometheus-operator/issues/1921#issue-362467141

What did you do?
I have a pod and a corresponding service running in the monitoring, default and trial namespaces. Using the "namespaceSelector.matchLabels" I am trying to use a serviceMonitor to detect this pod/service from all three namespaces.

What did you expect to see?
I expected prometheus to detect the pods from all three namespaces rak-app (monitoring ns), rak-app-def (default ns) and rak-app-trial (trial ns).

What did you see instead? Under which circumstances?
Prometheus is only able to detect pods from the monitoring and default namespaces. The pod running in the trial namespace goes undetected. Although it should not matter, I tried running the serviceMonitor in the monitoring, default, and trial namespace. The pod in the trial namespace was not detected in any of the three cases.

Environment
I have deployed prometheus operator using the kube-prometheus manifests. A minor change to serviceMonitorNamespaceSelector and serviceMonitorSelector fields of prometheus-prometheus.yaml was made (seen in yaml below)

• Prometheus Operator version: 0.22.2

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  Using kubeadm

• Manifests:
  prometheus-prometheus.yaml

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    beta.kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.3.1

rak-app-serviceMonitoring-default.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: rak-app
  name: rak-app
#  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: rakport
#  jobLabel: app
  selector:
    matchLabels:
      k8s-app: rak-app
  namespaceSelector:
    matchNames:
    - monitoring
    - default
    - trial

rak-app-service-trial.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: rak-app
  name: rak-app-trial
  namespace: trial
spec:
  clusterIP: None
  ports:
  - name: rakport
    port: 8080
    targetPort: 8080
  selector:
    app: rak-app

[Meier]

is prometheus.io/scrape: 'true' annotation set on all pods?