In-Place Vertical Autoscaling on Kind Cluster Running k8s 1.27.3

[tanmay modak]

Hello, 

I am currently trying to implement in place vertical autoscaling (memory and cpu) on a Kind cluster that is running Kubernetes 1.27.3 and has Autoscalerversion 1.21.  I am not able to get the desired behavior by using both the "Auto" and "InPlaceOnly" UpdateModes.

1. Observation when UpdateMode=InPlaceOnly
I get the below error when I set UpdateMode=InPlaceOnly
Error from server: error when creating "vpa.yaml": admission webhook "vpa.k8s.io" denied the request: unexpected UpdateMode value InPlaceOnly.

Upon taking a closer look at the code, it seems that the Vpa only supports 4 UpdateModes - Auto, Off, Recreate and Initial, and throws a validation error for other values. Does this mean that we are supposed to implement  in-place scaling  by setting UpdateMode:Auto ?

2. Observation when UpdateMode=Auto
When I set the UpdateMode to Auto, the new pod config gets applied BUT the pod gets evicted instead of scaling in-place.

Some additional debugging info below:

1. Scaling the pod manually in place

When I manually try to scale the pod using the below command, I can observe in-place scaling which means that there is no issue with my cluster.
kubectl -n example patch pod demo-1 --patch '{"spec":{"containers":[{"name":"demo-ctr-1", "resources":{"requests":{"cpu":"800m"}, "limits":{"cpu":"800m"}}}]}}'

2. Kind Cluster Configuration

I have enabled InPlacePodVerticalScaling and the alpha api in my cluster using the below block in my Kind config

runtimeConfig:
  "api/alpha": "true"
featureGates:
  InPlacePodVerticalScaling: true

The "Command" section of the kube-api-server pod for my kind node looks like this

    Command:
      kube-apiserver
      --advertise-address=172.18.0.2
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --feature-gates=InPlacePodVerticalScaling=true
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --runtime-config=api/alpha=true
      --secure-port=6443
      --service-account-issuer=https://kubernetes.default.svc.cluster.local
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
      --service-cluster-ip-range=10.96.0.0/16
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

3. VPA Config used

apiVersion: "autoscaling.k8s.io/v1"

kind: VerticalPodAutoscaler
metadata:
  name: quarkus-vpa
  namespace: quarkus
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: quarkus-service
  updatePolicy:
   updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 10m
          memory: 15Mi
        maxAllowed:
          cpu: 50m
          memory: 35Mi
        controlledResources: ["cpu", "memory"]

 

Can someone please point me in the right direction ?

[Adam Bernot]

Did you enable the InPlaceVerticalScaling feature gate?

Adam Bernot Software Engineer ber...@google.com

--
You received this message because you are subscribed to the Google Groups "Autoscaling Kubernetes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-auto...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-autoscaling/e9e99ea6-cae4-43cf-8c27-41f78ed56f72n%40googlegroups.com.

[Ant Weiss]

The OP says:

2. Kind Cluster Configuration

I have enabled InPlacePodVerticalScaling and the alpha api in my cluster using the below block in my Kind config

runtimeConfig:
  "api/alpha": "true"
featureGates:
  InPlacePodVerticalScaling: true

Searching through the repo I see the only file that the "InPlace..." update modes are mentioned in - is the AEP: https://github.com/kubernetes/autoscaler/blob/444e3295dfe3228ad2ca7e45c003598454c260f2/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md

So it looks like this hasn't been implemented yet?

[Adam Bernot]

Not sure if it's just a typo, but the docs reference two similar (but not identical) names. InPlaceVerticalScaling and InPlacePodVerticalScaling.

Adam Bernot Software Engineer ber...@google.com

On Tue, Dec 10, 2024 at 8:56 AM tanmay modak <tanmay...@gmail.com> wrote:

Yes I believe did. 

1. This is from my Kind cluster config

runtimeConfig:
  "api/alpha": "true"
featureGates:
  InPlacePodVerticalScaling: true

2. Also, the "Command" section of the kube-api-server pod for my Kind node looks like this. It shows both the InPodVerticalScaling featuregate, and also the Kubernetes 1.27 alpha API enabled (highlighted).

--

Best

Tanmay

[Ray Wainman]

Hey all,

This functionality is not yet implemented so the VPA will continue to evict the pod rather than update it.

https://github.com/kubernetes/autoscaler/pull/6652 is a draft PR for this but it likely needs some updates with the recent changes to in-place updates that just went into 1.32.

Would be great to look at this with some fresh eyes and see if we can pick this up again :)