Heapster: how to get all pod metrics (cpu/usage, memory/usage, etc) in API call

[lin.for...@gmail.com]

Hi,

I was trying to use Heapster to collect pod metrics and I did not specify sink for Heapster. The initial idea is to use Heapster metric model API get all metrics of a pod. But from the API specification and test, I did not see there is Heapster API which can return all the metrics, just like this

MetricSet: namespace:kube-system/pod:kubernetes-dashboard-v1.1.1-4cq9x
   Scrape time: 2016-08-17 22:53:57.057958401 -0400 EDT 1471488837057958401
   Create time: 2016-08-15 21:49:23.267586 -0400 EDT 1471312163267586000
   Labels:
      host_id = 9.21.60.67
      hostname = 9.21.60.67
      labels = k8s-app:kubernetes-dashboard,kubernetes.io/cluster-service:true,version:v1.1.1
      namespace_id = 0a6b2c6d-6353-11e6-b324-525400f5dfb2
      namespace_name = kube-system
      nodename = 9.21.60.67
      pod_id = a4f64ee8-6353-11e6-b324-525400f5dfb2
      pod_name = kubernetes-dashboard-v1.1.1-4cq9x
      pod_namespace = kube-system
      type = pod
   Metrics:
      cpu/limit = 100
      cpu/request = 100
      cpu/usage = 10298588
      cpu/usage_rate = 1
      memory/limit = 52428800
      memory/major_page_faults = 1
      memory/major_page_faults_rate = 0.000000
      memory/page_faults = 160
      memory/page_faults_rate = 0.000000
      memory/request = 52428800
      memory/usage = 12447744
      memory/working_set = 5976064
      network/rx = 8519194
      network/rx_errors = 0
      network/rx_errors_rate = 0.000000
      network/rx_rate = 46.852867
      network/tx = 18946258
      network/tx_errors = 0
      network/tx_errors_rate = 0.000000
      network/tx_rate = 102.357391
      uptime = 176681761
   Labeled Metrics:
      filesystem/usage = 40960
         resource_id = /dev/mapper/ubu14042tmp--vg-root
      filesystem/limit = 19145666560
         resource_id = /dev/mapper/ubu14042tmp--vg-root

While from the Heapster API, I only can get one metric at a time,

# curl http://k8s:8082/api/v1/model/namespaces/kube-system/pods/kubernetes-dashboard-v1.1.1-4cq9x/metrics/cpu/usage
{
  "metrics": [
   {
    "timestamp": "2016-08-17T22:51:00-04:00",
    "value": 10298588
   },
   {
    "timestamp": "2016-08-17T22:52:00-04:00",
    "value": 10298588
   },
   {
    "timestamp": "2016-08-17T22:53:00-04:00",
    "value": 10298588
   }
  ],
  "latestTimestamp": "2016-08-17T22:53:00-04:00"
 }

Is anyone know how to do?

Other question about Heapster:
I can not get filesystem metrics (filesystem/usage) from existing Heapster API. How can I get filesystem metrics of a pod?
What's exact meaning of the value of cpu/usage, memory/usage, filesystem/usage? Any reference?

Thanks
Liqiang Lin

[Solly Ross]

> I was trying to use Heapster to collect pod metrics and I did not specify
> sink for Heapster. The initial idea is to use Heapster metric model API get
> all metrics of a pod. But from the API specification and test, I did not see
> there is Heapster API which can return all the metrics, just like this

<snip>

> While from the Heapster API, I only can get one metric at a time,

<snip>

> Is anyone know how to do?

Unfortunately, this is not possible at the moment. You can get memory usage and
cpu usage rate together via the new API at
`/apis/metrics/v1alpha1/namespaces/$NS/pods?labelSelector=someSelector`, but other
than that you cannot get metrics together.

Can you elaborate on your usecase a bit? If you're just trying to collect the metrics
for later use, it's probably better to use a sink. If none of the sinks are applicable,
you could write your own sink adapter to use the sink of your choice.

> Other question about Heapster:
> I can not get filesystem metrics (filesystem/usage) from existing Heapster
> API. How can I get filesystem metrics of a pod?
> What's exact meaning of the value of cpu/usage, memory/usage,
> filesystem/usage? Any reference?

IIRC, you should be able to get filesystem metrics from Heapster.
Are they just not appearing?

The meanings of the various metrics are listed here:
https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md

Best Regards,
Solly Ross

[lin.for...@gmail.com]

On Thursday, August 18, 2016 at 11:05:08 PM UTC+8, Solly Ross wrote:

> I was trying to use Heapster to collect pod metrics and I did not specify
> sink for Heapster. The initial idea is to use Heapster metric model API get
> all metrics of a pod. But from the API specification and test, I did not see
> there is Heapster API which can return all the metrics, just like this

<snip>

> While from the Heapster API, I only can get one metric at a time,

<snip>

> Is anyone know how to do?

Unfortunately, this is not possible at the moment.  You can get memory usage and
cpu usage rate together via the new API at
`/apis/metrics/v1alpha1/namespaces/$NS/pods?labelSelector=someSelector`, but other
than that you cannot get metrics together.

Can you elaborate on your usecase a bit?  If you're just trying to collect the metrics
for later use, it's probably better to use a sink.  If none of the sinks are applicable,
you could write your own sink adapter to use the sink of your choice.

[Liqlang] Currently I was investigating whether we can get some basic metrics of pod from Heapster model API. Eventually we will use influxdb or ElasticSearch as the sink.

> Other question about Heapster:
> I can not get filesystem metrics (filesystem/usage) from existing Heapster
> API. How can I get filesystem metrics of a pod?
> What's exact meaning of the value of cpu/usage, memory/usage,
> filesystem/usage? Any reference?

IIRC, you should be able to get filesystem metrics from Heapster.
Are they just not appearing?

[Liqiang Lin] Well, from the API response, I did not see filesystem metrics

# curl http://10.1.23.3:8082/api/v1/model/namespaces/default/pods/busybox/metrics
[
  "memory/usage",
  "network/tx_rate",
  "network/tx",
  "cpu/usage_rate",
  "cpu/limit",
  "memory/major_page_faults",
  "network/rx_errors",
  "memory/request",
  "memory/limit",
  "network/rx_errors_rate",
  "memory/working_set",
  "uptime",
  "network/rx_rate",
  "memory/page_faults",
  "network/rx",
  "memory/page_faults_rate",
  "memory/major_page_faults_rate",
  "network/tx_errors",
  "cpu/request",
  "cpu/usage",
  "network/tx_errors_rate"
 ]

And it seems the filesystem metrics appears as

   Labeled Metrics:
      filesystem/usage = 4126711808
         resource_id = /dev/mapper/ubu14042tmp--vg-root
      filesystem/usage = 41402368
         resource_id = /dev/vda1

      filesystem/limit = 19145666560
         resource_id = /dev/mapper/ubu14042tmp--vg-root

      filesystem/limit = 246755328
         resource_id = /dev/vda1

I did not know what's the exact meaning of Labeled Metrics here.

[Solly Ross]

> And it seems the filesystem metrics appears as

> Labeled Metrics:
> filesystem/usage = 4126711808
> resource_id = /dev/mapper/ubu14042tmp--vg-root
> filesystem/usage = 41402368
> resource_id = /dev/vda1
> filesystem/limit = 19145666560
> resource_id = /dev/mapper/ubu14042tmp--vg-root
> filesystem/limit = 246755328
> resource_id = /dev/vda1

> I did not know what's the exact meaning of Labeled Metrics here.

Ah, so, labeled metrics are metrics that have the same name, but different "subjects".
For instance, here we have multiple labeled metrics, all named "filesystem/usage"
(since they're all fundamentally describing filesystem usage), but broken down by device.

As of recently, you can now fetch labeled metrics through the Heapster model API (it wasn't
possible previously) by using something like
`curl $HEAPSTER/api/v1/model/namespaces/$NS/pods/$POD/metrics/filesystem/usage?labels=resource_id:/dev/vda1`
You'll need a build of Heapster that includes https://github.com/kubernetes/heapster/pull/1229.

Hopefully that helps.