Hot plugin cpu and memory for vmi(virt-launcher pod) in burstable or besteffort Qos group

[杨晨君]

Hi all, i was wondering if we are able to add hot plugin feature for vmi(virt-launcher pod) which in burstable or besteffort Qos group, lately i been working on a project use kubevirt i did some code modification to implement cpu and memory hot plugin for  burstable or besteffort Qos group,  I`m thinking about to contribute this feature to community but I`m not sure this feature is necessary

[Fabian Deutsch]

Hey!

That's interesting-

In general I think that we are waiting on https://github.com/kubernetes/kubernetes/pull/102884 in order to implement cpu +mem hotplug.

Your idea to do this when a VM is burstable … is interesting.

How do you decide when to expand cpu/memory and when to shrink it again?

Greetings

- fabian

On Thu, Jul 14, 2022 at 8:57 AM 杨晨君 <simo...@gmail.com> wrote:

Hi all, i was wondering if we are able to add hot plugin feature for vmi(virt-launcher pod) which in burstable or besteffort Qos group, lately i been working on a project use kubevirt i did some code modification to implement cpu and memory hot plugin for  burstable or besteffort Qos group,  I`m thinking about to contribute this feature to community but I`m not sure this feature is necessary

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/c84ece97-d4d0-435a-9df9-a20aa4496b8en%40googlegroups.com.

[杨晨君]

hi， fdeutsch, thank you for the reply ：）
For Expand cpu/memory on fly, i think few step need to be done

1. virt-launcher has to be adjust to allow cpu and memory hot plugin by doing so we need ensure cpu tag of libvirt xml should be something like "<vcpu placement='static' current='2'>128</vcpu>" and some cpu numa cell setting needs to be add into xml to support memory hot plugin like "<cell id='0' cpus='0-127' memory='4194304' unit='KiB'/>" also add a new tag "<maxMemory slots='16' unit='KiB'>134217728</maxMemory>"
2. Add a new socket file in virt-launcher and setup with restful api service over it for hot plugin cpu and memory
3. A crd called LiveResizer should be created with property of TargetVMI,CPU,Memory
4. (option 1)Add crd LiveResizer controller to virt-handler to watch crd LiveResizer life cycle and interfacing with virt-launcher new created socket and update vm and vmi object after resource modification
5. (option 2)Create a new project to control LiveResizer object and interfacing with virt-launcher new created socket and update vm and vmi object after resource modification

For resource shrink i think it`s to risk to shrink resource on fly, so i think may be we don't have to support shrink resource on fly.

[Fabian Deutsch]

On Thu, Jul 14, 2022 at 10:26 AM 杨晨君 <simo...@gmail.com> wrote:

hi， fdeutsch, thank you for the reply ：）
For Expand cpu/memory on fly, i think few step need to be done

1. virt-launcher has to be adjust to allow cpu and memory hot plugin by doing so we need ensure cpu tag of libvirt xml should be something like "<vcpu placement='static' current='2'>128</vcpu>" and some cpu numa cell setting needs to be add into xml to support memory hot plugin like "<cell id='0' cpus='0-127' memory='4194304' unit='KiB'/>" also add a new tag "<maxMemory slots='16' unit='KiB'>134217728</maxMemory>"
2. Add a new socket file in virt-launcher and setup with restful api service over it for hot plugin cpu and memory
3. A crd called LiveResizer should be created with property of TargetVMI,CPU,Memory
4. (option 1)Add crd LiveResizer controller to virt-handler to watch crd LiveResizer life cycle and interfacing with virt-launcher new created socket and update vm and vmi object after resource modification
5. (option 2)Create a new project to control LiveResizer object and interfacing with virt-launcher new created socket and update vm and vmi object after resource modification

 

For resource shrink i think it`s to risk to shrink resource on fly, so i think may be we don't have to support shrink resource on fly.

Yeah, but this would mean that VMs would be killed if they spiked, don't give memory back, and the node is getting under pressure, right?

What use-case are you seeing for this?

 

On Thursday, July 14, 2022 at 3:03:53 PM UTC+8 fdeu...@redhat.com wrote:

Hey!

That's interesting-

In general I think that we are waiting on https://github.com/kubernetes/kubernetes/pull/102884 in order to implement cpu +mem hotplug.

Your idea to do this when a VM is burstable … is interesting.

How do you decide when to expand cpu/memory and when to shrink it again?

Greetings

- fabian

On Thu, Jul 14, 2022 at 8:57 AM 杨晨君 <simo...@gmail.com> wrote:

Hi all, i was wondering if we are able to add hot plugin feature for vmi(virt-launcher pod) which in burstable or besteffort Qos group, lately i been working on a project use kubevirt i did some code modification to implement cpu and memory hot plugin for  burstable or besteffort Qos group,  I`m thinking about to contribute this feature to community but I`m not sure this feature is necessary

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/c84ece97-d4d0-435a-9df9-a20aa4496b8en%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/2ff42cf5-9004-4361-9cb8-3beb6a84a717n%40googlegroups.com.

[杨晨君]

This use-case more like openstack feature resource overcommit,we give an overcommit rate for cpu and memory then administrator is responsible for handling this rate.

 I totally agree with you, oom might be occur if vm don't return memory back, but we can add a feature like shrink the resource when vmi is shutoff.

For sure with https://github.com/kubernetes/kubernetes/pull/102884 feature vmi resource grow and shrink will be more safe, I think waiting for this feature cloud be best choice without much to risk, 

still thank you for updating me fdeutsch, it`s good to know you guys got a plan on this feature, please let me know if anything I can do for the community :)

[Fabian Deutsch]

Hey Simon,

On Fri, Jul 15, 2022 at 2:57 AM 杨晨君 <simo...@gmail.com> wrote:

This use-case more like openstack feature resource overcommit,we give an overcommit rate for cpu and memory then administrator is responsible for handling this rate.

 I totally agree with you, oom might be occur if vm don't return memory back, but we can add a feature like shrink the resource when vmi is shutoff.

For sure with https://github.com/kubernetes/kubernetes/pull/102884 feature vmi resource grow and shrink will be more safe, I think waiting for this feature cloud be best choice without much to risk, 

Yep, I'm really looking forward to this to land - and patches are welcome

 

still thank you for updating me fdeutsch,

Just call me Fabian ;)

 

it`s good to know you guys got a plan on this feature, please let me know if anything I can do for the community :)

Oh - So, while there is this feature that can - I think - already be designed, we also do regular bug scrubs after the weekly meeting on Wednesdays, and we have CI flakes, and the user-guide always needs love, thus - pick one to get started :)

- fabian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/9c6852ac-323d-4cd0-bdc2-949e1666431an%40googlegroups.com.

[杨晨君]

Copy that Fabian, I will dig into docs and bugs to see anything I can do :)

[Andrei Kvapil (kvaps)]

Hey folks,

I was also thinking on implementation of this.

I think that cpu/memory hotplug can be done the same way we doing for volumes hotplug. By running additional empty pod (for resources reservation on the node) and updating cgroup resources of vm container to hotplug them into vm?

What do you think about this, will it work?

Regards

[Fabian Deutsch]

Hey Andrei,

On Fri, Feb 24, 2023 at 4:15 PM Andrei Kvapil (kvaps) <kva...@gmail.com> wrote:

Hey folks,

I was also thinking on implementation of this.
I think that cpu/memory hotplug can be done the same way we doing for
volumes hotplug. By running additional empty pod (for resources reservation
on the node) and updating cgroup resources of vm container to hotplug them
into vm?

What do you think about this, will it work?

I'm not sure how well this would work, but - I know that some folks have been discussing to explore to use live migration for i.e. cpu hot plug

[kvaps]

Good news,

Today https://github.com/kubernetes/kubernetes/pull/102884 is get merged 🎉🎉🎉

Now we can think about implementation of this the propper way.

Best Regards,

Andrei Kvapil

[Fabian Deutsch]

On Tue, Feb 28, 2023 at 11:00 AM kvaps <kva...@gmail.com> wrote:

Good news,

Today https://github.com/kubernetes/kubernetes/pull/102884 is get merged 🎉🎉🎉

Now we can think about implementation of this the propper way.

That's awesome! FINALLY! :)

[Itamar Holder]

Very cool!

Thanks for sharing this!

--

You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAGO-sgMPOJFpXe1KkHrdbUQyFpeoeCqAw7_z4PB%2BewC6Cd3m4w%40mail.gmail.com.

[Roman Mohr]

On Tue, Feb 28, 2023 at 11:00 AM kvaps <kva...@gmail.com> wrote:

Good news,

Today https://github.com/kubernetes/kubernetes/pull/102884 is get merged 🎉🎉🎉

Now we can think about implementation of this the propper way.

Unbelievable. Awesome :)

 

--

You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAGO-sgMPOJFpXe1KkHrdbUQyFpeoeCqAw7_z4PB%2BewC6Cd3m4w%40mail.gmail.com.

[Andrei Kvapil (kvaps)]

Hey folks,

I just prepared a new design proposal for this feature, I invite you to discuss https://github.com/kubevirt/community/pull/212
Any suggestions are welcome 🙂