Kube-State-Metrics not being scraped by Prometheus

[Ranindu]

Hey guys my issue here is as stated, the kube-state-metrics are not being scraped by Prometheus, therefore my Grafana dashboards tell me that I have no data when displaying my pod metrics.
When I do:
curl stateMetricsIP:8080/metrics
it displays all the state-metrics. But these metrics are not present within the /metrics endpoint of Prometheus.
I suspect that the problem is within the prometheus.yaml kube-state-metrics job but I'm not sure.
My configuration on Prometheus side for the job is

      

- job_name: 'kube-state-metrics'

  honor_timestamps: true

  scrape_interval: 5s

  scrape_timeout: 5s

  metrics_path: /metrics

  scheme: http

  static_configs:

  - targets:

    - kube-state-metrics.ops.svc.cluster.local:8080

Am I missing something here? Both my services are working fine, prometheus gives me the 'OK' for kube-state-metrics target as well. 

[Brian Candler]

> When I do:
> curl stateMetricsIP:8080/metrics
> it displays all the state-metrics. But these metrics are not present within the /metrics endpoint of Prometheus.

That's correct and expected.  The /metrics endpoint of prometheus does *not* show all scraped metrics; it just returns metrics about prometheus itself, such as the status of the timeseries database.  The idea is so that you can get prometheus to scrape itself to get a history of the TSDB status.

If you want to check whether the kube-state-metrics have been ingested, you need to send a PromQL query to prometheus, either using the built-in PromQL query browser (the "Graph" tab in the web interface), or using the HTTP query API.

If you don't know what the metric names are, then you can do

{__name__=~".+",job="kube-state-metrics"}

although I would never recommend that on a production system because it could return many thousands of results.