Unable to get http_requests_total metric from latest version of the image

574 views
Skip to first unread message

arun...@gmail.com

unread,
Aug 27, 2018, 4:59:51 AM8/27/18
to Prometheus Users
Hi Guys,

The requirement is to configure the k8s HPA based on http_request which are captured in the Prometheus Metrics.

Issue:

I'm unable to get the http_request metric from the latest version of the Prometheus images (v2.3.2 and v2.3.1).

I can able to get the http_request metric from the version 2.1.0. 

Here are pod metrics which getting from image 2.1.0

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep pod

      "name": "pods/go_memstats_heap_objects",

      "name": "pods/memory_failcnt",

      "name": "pods/go_memstats_heap_alloc_bytes",

      "name": "pods/go_memstats_sys_bytes",

      "name": "pods/http_requests_bucket",

      "name": "pods/skydns_skydns_dns_response_size_bytes_sum",

      "name": "pods/cpu_cfs_throttled",

      "name": "pods/memory_working_set_bytes",

      "name": "pods/fs_writes_merged",

      "name": "pods/http_requests_count",

      "name": "pods/skydns_skydns_dns_response_size_bytes_bucket",

      "name": "pods/fs_reads_merged",

      "name": "pods/go_memstats_heap_sys_bytes",

      "name": "pods/go_memstats_stack_inuse_bytes",

      "name": "pods/process_resident_memory_bytes",

      "name": "pods/skydns_skydns_dns_response_size_bytes_count",

      "name": "pods/memory_cache",

      "name": "pods/spec_memory_swap_limit_bytes",

      "name": "pods/http_requests",

      "name": "pods/scrape_duration_seconds",

      "name": "pods/skydns_skydns_dns_request_count",

      "name": "pods/memory_rss",

      "name": "pods/cpu_cfs_periods",

      "name": "pods/fs_io_current",

      "name": "pods/go_memstats_heap_idle_bytes",

      "name": "pods/skydns_skydns_dns_request_duration_seconds_sum",

      "name": "pods/cpu_load_average_10s",

      "name": "pods/go_memstats_buck_hash_sys_bytes",

      "name": "pods/go_memstats_gc_sys_bytes",

      "name": "pods/fs_reads",

      "name": "pods/go_memstats_lookups",

      "name": "pods/go_memstats_mcache_sys_bytes",

      "name": "pods/go_memstats_next_gc_bytes",

      "name": "pods/http_request_duration_microseconds_count",

      "name": "pods/process_virtual_memory_bytes",

      "name": "pods/fs_inodes_free",

      "name": "pods/memory_max_usage_bytes",

      "name": "pods/go_memstats_alloc_bytes",

      "name": "pods/memory_swap",

      "name": "pods/kubernetes_build_info",

      "name": "pods/http_requests_sum",

      "name": "pods/fs_reads_bytes",

      "name": "pods/fs_sector_writes",

      "name": "pods/go_goroutines",

      "name": "pods/skydns_skydns_dns_request_duration_seconds_count",

      "name": "pods/go_memstats_mcache_inuse_bytes",

      "name": "pods/spec_memory_limit_bytes",

      "name": "pods/start_time_seconds",

      "name": "pods/go_memstats_other_sys_bytes",

      "name": "pods/http_request_size_bytes",

      "name": "pods/fs_io_time_weighted",

      "name": "pods/spec_cpu_shares",

      "name": "pods/http_request_duration_microseconds_sum",

      "name": "pods/skydns_skydns_dns_cachemiss_count",

      "name": "pods/fs_sector_reads",

      "name": "pods/http_request_duration_microseconds",

      "name": "pods/fs_write",

      "name": "pods/spec_memory_reservation_limit_bytes",

      "name": "pods/http_response_size_bytes_count",

      "name": "pods/process_start_time_seconds",

      "name": "pods/fs_limit_bytes",

      "name": "pods/cpu_user",

      "name": "pods/go_memstats_frees",

      "name": "pods/scrape_samples_scraped",

      "name": "pods/up",

      "name": "pods/go_memstats_mallocs",

      "name": "pods/memory_usage_bytes",

      "name": "pods/process_max_fds",

      "name": "pods/fs_writes",

      "name": "pods/memory_failures",

      "name": "pods/network_udp_usage",

      "name": "pods/http_request_size_bytes_count",

      "name": "pods/process_open_fds",

      "name": "pods/skydns_skydns_dns_error_count",

      "name": "pods/fs_writes_bytes",

      "name": "pods/go_memstats_heap_inuse_bytes",

      "name": "pods/fs_read",

      "name": "pods/process_cpu",

      "name": "pods/go_gc_duration_seconds_count",

      "name": "pods/go_gc_duration_seconds_sum",

      "name": "pods/go_memstats_stack_sys_bytes",

      "name": "pods/spec_cpu_period",

      "name": "pods/go_memstats_heap_released_bytes",

      "name": "pods/scrape_samples_post_metric_relabeling",

      "name": "pods/fs_inodes",

      "name": "pods/last_seen",

      "name": "pods/cpu_cfs_throttled_periods",

      "name": "pods/go_memstats_mspan_inuse_bytes",

      "name": "pods/network_tcp_usage",

      "name": "pods/http_request_size_bytes_sum",

      "name": "pods/skydns_skydns_dns_request_duration_seconds_bucket",

      "name": "pods/tasks_state",

      "name": "pods/cpu_usage",

      "name": "pods/fs_io_time",

      "name": "pods/go_memstats_last_gc_time_seconds",

      "name": "pods/go_gc_duration_seconds",

      "name": "pods/fs_usage_bytes",

      "name": "pods/spec_cpu_quota",

      "name": "pods/go_memstats_mspan_sys_bytes",

      "name": "pods/http_response_size_bytes_sum",

      "name": "pods/cpu_system",

      "name": "pods/http_response_size_bytes",


Here are pod metrics which getting from image 2.3.1.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep pod
     "name": "pods/memory_working_set_bytes",
     "name": "pods/spec_cpu_quota",
     "name": "pods/cpu_user",
     "name": "pods/fs_inodes_free",
     "name": "pods/fs_usage_bytes",
     "name": "pods/fs_limit_bytes",
     "name": "pods/spec_cpu_shares",
     "name": "pods/last_seen",
     "name": "pods/fs_sector_writes",
     "name": "pods/spec_memory_swap_limit_bytes",
     "name": "pods/fs_io_time_weighted",
     "name": "pods/start_time_seconds",
     "name": "pods/fs_reads",
     "name": "pods/fs_writes",
     "name": "pods/fs_io_time",
     "name": "pods/cpu_cfs_throttled",
     "name": "pods/network_tcp_usage",
     "name": "pods/memory_usage_bytes",
     "name": "pods/memory_cache",
     "name": "pods/memory_failures",
     "name": "pods/spec_memory_reservation_limit_bytes",
     "name": "pods/cpu_usage",
     "name": "pods/cpu_system",
     "name": "pods/fs_writes_bytes",
     "name": "pods/fs_reads_bytes",
     "name": "pods/fs_sector_reads",
     "name": "pods/spec_memory_limit_bytes",
     "name": "pods/cpu_load_average_10s",
     "name": "pods/memory_rss",
     "name": "pods/memory_swap",
     "name": "pods/spec_cpu_period",
     "name": "pods/fs_io_current",
     "name": "pods/tasks_state",
     "name": "pods/fs_write",
     "name": "pods/network_udp_usage",
     "name": "pods/memory_failcnt",
     "name": "pods/fs_writes_merged",
     "name": "pods/fs_read",
     "name": "pods/cpu_cfs_throttled_periods",
     "name": "pods/memory_max_usage_bytes",
     "name": "pods/cpu_cfs_periods",
     "name": "pods/fs_reads_merged",
     "name": "pods/fs_inodes",


Please help me to get the http_request metric from laster version of prometheus becasue downline we should use the latest version of image in the prod environment. 

Julius Volz

unread,
Aug 28, 2018, 7:55:30 AM8/28/18
to Arunkumar M, Prometheus Users
I'm not sure how the Kubernetes HPA translates/bridges Prometheus metrics, but Prometheus itself never exposed an "http_requests" metric. In Prometheus 2.1.0, it exposed a metric "http_request_duration_microseconds_count" giving you a counter of total HTTP requests, which got renamed to the better name "prometheus_http_request_duration_seconds_count" in the current Prometheus. Maybe one has to tell the HPA or custom metric API to expose that new metric name somehow?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cd0046c3-c3b7-4728-b351-91cad3908e1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages