Is it possible to scale down gracefully?

[Nathan Smith]

I am using Kubernetes 1.4.6 on GKE with autoscaling enabled on my node-pool and --min-nodes set to 2. I have an HPA setup to autoscale the cluster size based on what is needed. This setup works fine for us except when the nodes/pods are scaled down and is most noteable when the scaling is manually triggered by editing minReplicas in the HPA by large amounts. The problem is that the user will have made a request that was sent to a pod, the container that handles the request is then either talking to another container or making network requests, but while it is processing the request, the pod/node is killed by the autoscaler and the request fails with network errors. This will happen many times when the pods scale down automatically or manually. 

This is a problem whether it is the pods scaling or if I am changing the node-pool size. How can I make Kubernetes properly drain the pods/nodes before ripping them out? Is this even possible?

[Timo Reimann]

I have zero experience with HPA yet so far but would hope that scaling down is just implemented in terms of regular pod termination, which means that these pods should be subject to the usual shutdown process. Consequently, you should be able to set up a SIGTERM handler or preStop hook making sure that pending requests are given sufficient time to finish.

References:

http://kubernetes.io/docs/user-guide/production-pods/#lifecycle-hooks-and-termination-notice

http://kubernetes.io/docs/user-guide/pods/#termination-of-pods

HTH,
Timo

[Nathan Smith]

Thanks for the suggestion Timo! I will look into this. These hooks are fired per container, and because my issue seems to be that the SQL proxy container is being terminated before my web server container, I'm not sure how I would go about implenting this. I'll investigate this as an option but any suggestions for implementation would be appreciated.

[Clayton Coleman]

Just remember to set a longer gracePeriod.  gracePeriod bounds both preStop and when SIGKILL is sent, so keep that in mind.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/656cb50a-b26e-46e4-976e-03f85997161a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Filip Grzadkowski]

This should be the case. HPA is using /scale subresource that only changes number of replicas of replication controller or deployment. All the rest is handled down the stack.

--

Filip