Disk resize for running virtual machines

[Maya Rashish]

Hi all,

I have an implementation for utilizing disk resize by a running ("online") virtual machine, which I hope is valuable.

I'm presenting it slightly ahead of having a clean pull request, which will hopefully be available in the next few days.

Rather than share a Google doc document, it is embedded in this HTML email.

Hoping to hear your feedback in today's meeting, or by email, or in the coming pull request.

Maya

(Online) Disk Resize for VMs

Background:

An optimization for the cloning speed of DataVolumes required an expansion step (for Filesystem disk.img).

Why not make this expansion generally usable?

Why not do it with a running VirtualMachine?

Kubernetes has a couple of relevant feature gates, now in beta (widely available):

• ExpandPersistentVolumes

• ExpandInUsePersistentVolumes

These allow us to expand a PVC, but what do we need in order to take advantage of it in a Virtual Machine?

• If the storage is a disk.img file on a Filesystem, increase the size of this file.

• Notify the VirtualMachine about this, via libvirt block-resize.

	virt-handler		virt-launcher
Filesystem	Writes down expected disk.img sizes	VirtualMachineOptions→	Compares to disk.img virtual size, expand if needed.
			↓
Block	Tracks sizes in VolumeStatus, notes changed disks	VirtualMachineOptions→	Sends block-resize to VM

Caveats to current implementation:

• Incomplete feature, no WIP PR to show yet. Details may change.

• Only implemented for RAW disk.img files (e.g. those created by DataVolumes) - qemu-img didn't like operating on an in-use disk.img, used fallocate/ftruncate instead.

• Only tried rook-ceph, which can do online resize.

If non-online expansion is common, might want to add a condition “filesystem resize pending” instructing the user that if they want the change to affect the Virtual Machine, they need to stop and then start it.

[Andrea Bolognani]

On Wed, Jun 23, 2021 at 03:45:17PM +0300, Maya Rashish wrote:
> Hi all,
>
> I have an implementation for utilizing disk resize by a running ("online")
> virtual machine, which I hope is valuable.
>

> Caveats to current implementation:
>
> - Only implemented for RAW disk.img files (e.g. those created by

> DataVolumes) - qemu-img didn't like operating on an in-use disk.img, used
> fallocate/ftruncate instead.

You should be able to just call libvirt.Domain.BlockResize() and have
libvirt perform the image resizing for you. This should work for both
raw and qcow2 images sitting on a filesystem.

Using qemu-img to perform modifications on a disk image that's being
used by a running VM is not a good idea, which is precisely why the
tool will refuse that sort of request :) Needless to say, going to an
even lower level and using raw file operations directly has an even
higher potential of things going sideways.

Did you run into issues with the pure libvirt approach? Perhaps there
are bugs that need tracking and squashing.

--
Andrea Bolognani / Red Hat / Virtualization

[Alice Frosi]

On Wed, Jun 23, 2021 at 2:45 PM Maya Rashish <mras...@redhat.com> wrote:

Hi all,

I have an implementation for utilizing disk resize by a running ("online") virtual machine, which I hope is valuable.

I'm presenting it slightly ahead of having a clean pull request, which will hopefully be available in the next few days.

Rather than share a Google doc document, it is embedded in this HTML email.

Hoping to hear your feedback in today's meeting, or by email, or in the coming pull request.

Maya

(Online) Disk Resize for VMs

Background:

An optimization for the cloning speed of DataVolumes required an expansion step (for Filesystem disk.img).

Why not make this expansion generally usable?

Why not do it with a running VirtualMachine?

Could you please clarify how does the communication between CDI/DV and the VM works? 

Kubernetes has a couple of relevant feature gates, now in beta (widely available):

• ExpandPersistentVolumes

• ExpandInUsePersistentVolumes

These allow us to expand a PVC, but what do we need in order to take advantage of it in a Virtual Machine?

• If the storage is a disk.img file on a Filesystem, increase the size of this file.

• Notify the VirtualMachine about this, via libvirt block-resize.

	virt-handler		virt-launcher
Filesystem	Writes down expected disk.img sizes	VirtualMachineOptions→	Compares to disk.img virtual size, expand if needed.
			↓
Block	Tracks sizes in VolumeStatus, notes changed disks	VirtualMachineOptions→	Sends block-resize to VM

Caveats to current implementation:

• Incomplete feature, no WIP PR to show yet. Details may change.

• Only implemented for RAW disk.img files (e.g. those created by DataVolumes) - qemu-img didn't like operating on an in-use disk.img, used fallocate/ftruncate instead.

• Only tried rook-ceph, which can do online resize.

If non-online expansion is common, might want to add a condition “filesystem resize pending” instructing the user that if they want the change to affect the Virtual Machine, they need to stop and then start it.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAH6i1q9waGqB%3DxKwsCvvzLDUMy67TThQNfaY_-6ZcPcYy-BuZw%40mail.gmail.com.

[Maya Rashish]

Hi Alice,

On 24/06/21 9:30 am, Alice Frosi wrote:

>
>
> On Wed, Jun 23, 2021 at 2:45 PM Maya Rashish <mras...@redhat.com <mailto:mras...@redhat.com>> wrote:
>
> Hi all,
>
>
> I have an implementation for utilizing disk resize by a running ("online") virtual machine, which I hope is valuable.
>
> I'm presenting it slightly ahead of having a clean pull request, which will hopefully be available in the next few days.
>
> Rather than share a Google doc document, it is embedded in this HTML email.
>
>
> Hoping to hear your feedback in today's meeting, or by email, or in the coming pull request.
>
> Maya
>
>
>
> (Online) Disk Resize for VMs
>
>
> Background:
>
> An optimization for the cloning speed of DataVolumes required an expansion step (for Filesystem disk.img).
>
> Why not make this expansion generally usable?
>
> Why not do it with a running VirtualMachine?
>
> Could you please clarify how does the communication between CDI/DV and the VM works?

No communication between CDI/DV and the VM right now.
The communication is:
PVC size is changed -> when kubevirt runs a VM, it will resize for this change.

(I was thinking of adding the option of
DV size change -> PVC size change
as well.)

[Alice Frosi]

Thanks! It makes sense  :)

From the Community meeting discussion, I think that is the point where you need a PVC watcher in KubeVirt right? 

[Maya Rashish]

Thanks Andrea. I did this in the PR submitted now.
https://github.com/kubevirt/kubevirt/pull/5981

I assume that there's no way to specify whether the additional space is
preallocated?