How to get Kubernetes pod metrics

[Yong Zhang]

Hi, all

I want to use prometheus to get Kubernetes metrics, but I'm totally can't understand how to config it, I can get kubernetes node metrics by below configs:

  - job_name: 'kubernetes-nodes'
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:10255'
        target_label: __address__

But I don't know how to get pods metrics, I tried the default config from official example below, seem nothing happend, what should I do?

Thanks for the help!

# Example scrape config for pods
	#
	# The relabeling allows the actual pod scrape endpoint to be configured via the
	# following annotations:
	#
	# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
	# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
	# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
	- job_name: 'kubernetes-pods'
	kubernetes_sd_configs:
	- role: pod
	relabel_configs:
	- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
	action: keep
	regex: true
	- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
	action: replace
	target_label: __metrics_path__
	regex: (.+)
	- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
	action: replace
	regex: ([^:]+)(?::\d+)?;(\d+)
	replacement: $1:$2
	target_label: __address__
	- action: labelmap
	regex: __meta_kubernetes_pod_label_(.+)
	- source_labels: [__meta_kubernetes_namespace]
	action: replace
	target_label: kubernetes_namespace
	- source_labels: [__meta_kubernetes_pod_name]
	action: replace
	target_label: kubernetes_pod_name

[Julius Volz]

Hi,

the example you cite has some relabeling rules which expect some Kubernetes annotations to be present on the pods to monitor:

- only pods with the annotation "prometheus.io.scrape=true" will be scraped (the dots get translated to underscores during the relabeling)

- the path (like /metrics) is taken from the "prometheus.io.path" annotation

- it expects a prometheus.io.port annotation containing the port to scrape

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/02af6493-a1fe-4788-88be-613ea6ccc798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yong Zhang]

Thank you Julius, and I have queries that, is this used only for discovering or we can get detail pod metrics without additional configuration?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.