kubernetes_sd_configs not targeting required pods

[Ansh Lehri]

I am trying to target and scrape node-exporter and Kube-state-metric pods running in the EKS cluster using Prometheus. For service discovery, I am using kubernetes_sd_configs but Prometheus is not able to target the required pods.

Node-exporter and Prometheus are running in the same namespace while Kube-state-metrics is running in another namespace.

I am using pod annotations to discover the pods.

Following is the snippet of node-exporter.yml, prometheus.yml

Node-exporter.yaml:

apiVersion: apps/v1

kind: DaemonSet

metadata:

  labels:

    app.kubernetes.io/component: exporter

    app.kubernetes.io/name: node-exporter

  name: node-exporter

  namespace: amazon-cloudwatch

spec:

  selector:

    matchLabels:

      app.kubernetes.io/component: exporter

      app.kubernetes.io/name: node-exporter

  template:

    metadata:

      annotations:

        prometheus.io/scrape: true

        prometheus.io/path: /metrics

        prometheus.io/port: "9100"

        prometheus.io/scheme: "http"

      labels:

        app.kubernetes.io/component: exporter

        app.kubernetes.io/name: node-exporter

The same goes for the Kube-state-metrics.yml file only with minor changes but not in the annotations part. As for Prometheus, my code snippet of Prometheus scrape configs is follow:

- job_name: "kubernetes-pods"

        kubernetes_sd_configs:

          - role: pod

        relabel_configs:

        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

          action: keep

          regex: true

        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

          action: replace

          target_label: __metrics_path__

          regex: (.+)

        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

          action: replace

          regex: (.+):(?:\d+);(\d+)

          replacement: ${1}:${2}

          target_label: __address__

        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]

          action: replace

          target_label: __scheme__

          regex: (.+)

Can anyone please point out any error I am doing because the above snippets are not working.  Also suggestions for using some other configurations are also appreciated.

Thanks 

Ansh Lehri