GKE - HPA scaling based on custom metrics.

[bibinw...@gmail.com]

I am working on a use case where I have to scale my pods based on custom metrics.

Kubernetes Env: GKE 1.8.4 with alpha features enabled

I am getting error in HPA. Here is output of HPA.

Name: sample-tomcat-app-hpa
Namespace: default
Labels:
Annotations:
CreationTimestamp: Wed, 13 Dec 2017 20:06:22 +0530
Reference: Deployment/sample-tomcat-app
Metrics: ( current / target )
"tomcat_threadpool_connectioncount" on pods: unknown / 2
"tomcat_threadpool_connectioncount" on Service/sample-tomcat-app: unknown / 2
resource cpu on pods (as a percentage of request): unknown / 50%
Min replicas: 2
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: did not receive metrics for any ready pods
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetPodsMetric 14s (x8 over 3m) horizontal-pod-autoscaler did not receive metrics for any ready pods
Warning FailedComputeMetricsReplicas 14s (x8 over 3m) horizontal-pod-autoscaler failed to get pods metric value: did not receive metrics for any ready pods

Here is the list of enabled API's

apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1

I am not sure if --horizontal-pod-autoscaler-use-rest-clients=true is set in the controller manager. As GKE is a managed kubernetes cluster, we do not have an option to edit the master configurations. Is this option enabled in latest clusters?

[Beata Skiba]

Custom Metrics are unfortunately not available on GKE 1.8.4, they will be available with 1.9.0

[Bibin Wilson]

Thanks Beata for the reply!

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/3hHvmK_5AjE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[dz...@spotify.com]

Are they available now on alpha clusters with 1.9.1?

> To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
>
> To post to this group, send email to kubernet...@googlegroups.com.

[parsi...@gmail.com]

I tried to use this API today, it looks like `--horizontal-pod-autoscaler-use-rest-clients` is set to false because I'm getting the following error: `object metrics are not yet supported`. It means that I fall into legacy code here:

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/metrics/legacy_metrics_client.go#L176

and REST interface is not used.

If I use "Pods" instead of "Objects" it starts to work, but it is not what I'm looking for.

My question is:
How to set `--horizontal-pod-autoscaler-use-rest-clients=true` on GKE?

[Beata Skiba]

This should be available on alpha clusters starting from 1.9.2 release.

[Raju Dawadi]

HPA to scale Deployment based on the custom metrics is now available according to release note and tut: https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling But still I couldn't figure out way to scale with metric like http_requests. Any suggestion? Thanks.

[Aleksandra Malinowska]

> But still I couldn't figure out way to scale with metric like http_requests.

Can you say a bit more about this? For example, how do you export this metric, at what point in the tutorial you run into problems etc. It would make it easier to help :)

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.