Monitor only specific namespace metrics - Prometheus-operator within Kubernetes

Ravi Teja Reddy

unread,

Apr 30, 2020, 6:43:35 AM4/30/20

to Prometheus Users

We have a dev k8s cluster with around 500 namespaces but we are interested only few namespaces, due to the memory and storage constraints we want the prometheus to scrape only those few namespaces instead of scraping the all namespaces.

Can someone help me how do i do that ?

Here are my prometheus config file.

Prometheus-operator.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

annotations:

deployment.kubernetes.io/revision: "7"

generation: 7

labels:

k8s-app: prometheus-operator

namespace: monitoring

resourceVersion: "69052986"

selfLink: /apis/apps/v1/namespaces/monitoring/deployments/prometheus-operator

uid: 7d7d07a6-ca5e-11e9-8b8f-069b6823c94e

spec:

progressDeadlineSeconds: 600

replicas: 1

revisionHistoryLimit: 2

selector:

matchLabels:

k8s-app: prometheus-operator

strategy:

rollingUpdate:

maxSurge: 25%

maxUnavailable: 25%

type: RollingUpdate

template:

metadata:

creationTimestamp: null

labels:

k8s-app: prometheus-operator

spec:

containers:

- args:

- --kubelet-service=kube-system/kubelet

- --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1

image: quay.io/coreos/prometheus-operator:v0.38.0

imagePullPolicy: IfNotPresent

ports:

- containerPort: 8080

protocol: TCP

resources:

limits:

cpu: 200m

memory: 500Mi

requests:

cpu: 100m

memory: 350Mi

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

dnsPolicy: ClusterFirst

restartPolicy: Always

schedulerName: default-scheduler

securityContext: {}

serviceAccount: prometheus-operator

serviceAccountName: prometheus-operator

terminationGracePeriodSeconds: 30

prometheus-k8s.yaml

apiVersion: v1

items:

- apiVersion: monitoring.coreos.com/v1

kind: Prometheus

metadata:

creationTimestamp: "2020-04-13T11:12:20Z"

generation: 6

labels:

prometheus: k8s

namespace: monitoring

resourceVersion: "69314335"

selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/k8s

uid: 931a9d67-9e3b-4fa2-809b-7bff136f414e

spec:

alerting:

alertmanagers:

- name: alertmanager-main

namespace: monitoring

port: web

externalUrl: http://prometheus:9090

replicas: 2

resources:

limits:

cpu: 300m

memory: 6500Mi

requests:

cpu: 200m

memory: 6000Mi

retention: 90d

ruleSelector:

matchLabels:

prometheus: k8s

role: prometheus-k8s-rules

securityContext:

fsGroup: 0

runAsNonRoot: false

runAsUser: 0

serviceAccountName: prometheus-k8s

serviceMonitorSelector:

matchExpressions:

- key: k8s-app

operator: Exists

storage:

volumeClaimTemplate:

metadata: {}

spec:

resources:

requests:

storage: 200Gi

storageClassName: ssd

version: v2.17.0

kind: List

metadata:

resourceVersion: ""

selfLink: ""

----------------------------------------------------------------------------

After login to one of the prometheus pod i see the below config.

prometheus.yaml

/etc/prometheus # cat prometheus.yml

# my global config

global:

scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

alertmanagers:

- static_configs:

- targets:

# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

# - "first_rules.yml"

# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: 'prometheus'

# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.

static_configs:

- targets: ['localhost:9090']
----------------------------------------------------------------------------------------------------------------

/etc/prometheus # ls

certs config_out console_libraries consoles prometheus.yml rules

/etc/prometheus # cd config_out/

/etc/prometheus/config_out # ls

prometheus.env.yaml

/etc/prometheus/config_out # cat prometheus.env.yaml

global:

evaluation_interval: 30s

scrape_interval: 30s

external_labels:

prometheus: monitoring/k8s

prometheus_replica: prometheus-k8s-0

rule_files:

- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml

scrape_configs:

- job_name: monitoring/alertmanager/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_alertmanager

regex: main

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: web

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- target_label: endpoint

replacement: web

- job_name: monitoring/kube-apiserver/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- default

scrape_interval: 30s

scheme: https

tls_config:

insecure_skip_verify: false

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

server_name: kubernetes

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_component

regex: apiserver

- action: keep

source_labels:

- __meta_kubernetes_service_label_provider

regex: kubernetes

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: https

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_component

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: https

- job_name: monitoring/kube-controller-manager/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- kube-system

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kube-controller-manager

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics

- job_name: monitoring/kube-dns/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- kube-system

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kube-dns

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics-skydns

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics-skydns

- job_name: monitoring/kube-dns/1

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- kube-system

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kube-dns

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics-dnsmasq

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics-dnsmasq

- job_name: monitoring/kube-scheduler/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- kube-system

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kube-scheduler

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics

- job_name: monitoring/kube-state-metrics/0

honor_labels: true

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kube-state-metrics

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics

- job_name: monitoring/kubelet/0

honor_labels: true

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- kube-system

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: kubelet

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: cadvisor

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: cadvisor

- job_name: monitoring/node-exporter/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: node-exporter

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http-metrics

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- source_labels:

- __meta_kubernetes_service_label_k8s_app

target_label: job

regex: (.+)

replacement: ${1}

- target_label: endpoint

replacement: http-metrics

- job_name: monitoring/prometheus/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

scrape_interval: 30s

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_prometheus

regex: k8s

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: web

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- target_label: endpoint

replacement: web

- job_name: monitoring/prometheus-operator/0

honor_labels: false

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_label_k8s_app

regex: prometheus-operator

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: http

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Node;(.*)

replacement: ${1}

target_label: node

- source_labels:

- __meta_kubernetes_endpoint_address_target_kind

- __meta_kubernetes_endpoint_address_target_name

separator: ;

regex: Pod;(.*)

replacement: ${1}

target_label: pod

- source_labels:

- __meta_kubernetes_namespace

target_label: namespace

- source_labels:

- __meta_kubernetes_service_name

target_label: service

- source_labels:

- __meta_kubernetes_pod_name

target_label: pod

- source_labels:

- __meta_kubernetes_service_name

target_label: job

replacement: ${1}

- target_label: endpoint

replacement: http

alerting:

alert_relabel_configs:

- action: labeldrop

regex: prometheus_replica

alertmanagers:

- path_prefix: /

scheme: http

kubernetes_sd_configs:

- role: endpoints

namespaces:

names:

- monitoring

relabel_configs:

- action: keep

source_labels:

- __meta_kubernetes_service_name

regex: alertmanager-main

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: web

Thanks and Regards
ravi

Brian Candler

unread,

Apr 30, 2020, 12:57:57 PM4/30/20

to Prometheus Users

It looks like you're already limiting to specific namespaces? Is that not working?

# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
  names:
    [ - <string> ]

Otherwise: you're already filtering your prometheus service discovery results: e.g.

- action: keep

source_labels:

- __meta_kubernetes_service_label_alertmanager

regex: main

- action: keep

source_labels:

- __meta_kubernetes_endpoint_port_name

regex: web

You can addd an additional relabeling keep rule to match on __meta_kubernetes_namespace with a regex that matches the values of interest. See:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

Ravi Teja Reddy Bonthu

unread,

May 4, 2020, 12:52:43 AM5/4/20

to Brian Candler, Prometheus Users

Hi Brian,

My existing prometheus config scrape for all the namespaces doesn't have any namespace restriction at this moment which means its not working.

I am need to scrape all the metrics for test and dev namespaces only rest namespaces can be ignored.

I am wondering how do i change the prometheus.yaml / prometheus.env.yaml i have installed prometheus using coreOS prometheus-operator, where id did not specify any config file either prometheus.yaml / prometheus.env.yaml while deploying the prometheus.

Any idea on how can i make changes so that my prometheus can scrape only test and dev namespaces only.

Thanks in advance.

Regards,

Ravi

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/59b38efc-806e-4f1a-a558-c3442bd12b79%40googlegroups.com.

Reply all

Reply to author

Forward