Proposal - add <iotune> support for DataVolume and PVC source

115 views
Skip to first unread message

Zulhilmi Zainudin

unread,
May 3, 2021, 3:48:13 AM5/3/21
to kubevirt-dev

Hi all,

I'm opening this conversation/thread in relation to my PR here:

Some more background about the PR can be found in this issue:

In short, we want to have this <iotune> fields in the VM XML so we can stop noisy tenants from slamming the real host's disk and affecting other tenants.

Please do let me know if you have any questions.

Thanks.

Fabian Deutsch

unread,
May 6, 2021, 8:58:52 AM5/6/21
to Zulhilmi Zainudin, kubevirt-dev
Heya!

So - It was a couple of times considered to expose these options. The reason why this was not done, is because we were expecting Kubernetes to provide a standard to do IO throttling.
IOW doesn't this feature also make sense to containers?

If so: Does a KEP exist to bring this feature to Kubernetes? Did you consider to include it the relevant CSI driver?

In general in KubeVirt we try to implement features that could also be useful to containers.

- fabian
 

Thanks.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/c8f31885-761c-4fbf-860b-d32c217fc12bn%40googlegroups.com.

Zulhilmi Zainudin

unread,
May 6, 2021, 9:41:54 AM5/6/21
to kubevirt-dev
Hey Fabian,

The closest issue and KEP that I found related to this are:


But, looked like there is less update/activity on both of them. I assume it will take quite some time for the KEP to get finalized and implemented. That's why I'm submitting this proposal/PR so we can have something while waiting for the official IO throttling support in Kubernetes.

I'm not sure if I understand what you meant by "doesn't this feature also make sense to containers?". Could you elaborate further?

As for the "Did you consider to include it the relevant CSI driver?" question, I don't have any plan for that. I think this feature should be in Kubevirt and not in CSI. I'm curious why you said so.

Thanks.

Marcus S

unread,
May 7, 2021, 11:45:08 AM5/7/21
to kubevirt-dev
Yes, I think it makes sense in Kubernetes, not Kubevirt.  Containers would also benefit from having QoS on storage, and I imagine this could be implemented at CSI by perhaps proprietary means from each provider, or leverage cgroups at the node. 

The question around whether or not this will happen soon is another thing. There seems to be a fair number of requests and #1902 is current, though it sounds like they would target Kubernetes 1.22 or 1.23 after cgroups v2.  The temptation is that we have control over what KubeVirt can do and it would be faster to shoehorn the feature in, even if it ultimately is redundant or conflicts with the Kubernetes version. I also agree somewhat with comments on the Kubernetes discussion, that putting these controls into the hands of the users via the VM APIs may not be helpful if there is no quota/limit for what the user can set.

Have you considered making a simple sidecar hook to inject the iotune into your VMs? That would be an even faster solution that you'd have complete control of.

Alice Frosi

unread,
May 10, 2021, 2:54:23 AM5/10/21
to Marcus S, kubevirt-dev
Hi,

I generally agree that this feature should be implemented at k8s/CSI/runtime level. However, in Kubevirt for filesystem-based PVC we create a disk image on top. For block devices, we can restrict IO at cgroup level. However for the images on filesystem based PVC that is not possible, and only Kubevirt knows how to handle this kind of PVCs.

Alice

Fabian Deutsch

unread,
May 10, 2021, 5:38:52 AM5/10/21
to Marcus S, kubevirt-dev
On Fri, May 7, 2021 at 5:45 PM Marcus S <shad...@gmail.com> wrote:
Yes, I think it makes sense in Kubernetes, not Kubevirt.  Containers would also benefit from having QoS on storage, and I imagine this could be implemented at CSI by perhaps proprietary means from each provider, or leverage cgroups at the node. 

The question around whether or not this will happen soon is another thing. There seems to be a fair number of requests and #1902 is current, though it sounds like they would target Kubernetes 1.22 or 1.23 after cgroups v2.  The temptation is that we have control over what KubeVirt can do and it would be faster to shoehorn the feature in, even if it ultimately is redundant or conflicts with the Kubernetes version. I also agree somewhat with comments on the Kubernetes discussion, that putting these controls into the hands of the users via the VM APIs may not be helpful if there is no quota/limit for what the user can set.

Have you considered making a simple sidecar hook to inject the iotune into your VMs? That would be an even faster solution that you'd have complete control of.

+1

At the same time I wonder if we can find a volunteer to update our sidecar mechanism. Today the boilerplate to be created is quite large.

Wouldn't it be possible if we can write a meta sidecar with a binary that is simply executing a bash script (ADD'ed to the container) against the domxml?
The meta side car would effectively need to receive and send the original and modifed domxml via grpc ...

If we had this, then writing sidecars should be way way simpler than today.
 

Roman Mohr

unread,
May 10, 2021, 6:19:25 AM5/10/21
to Fabian Deutsch, Marcus S, kubevirt-dev
On Mon, May 10, 2021 at 11:39 AM Fabian Deutsch <fdeu...@redhat.com> wrote:


On Fri, May 7, 2021 at 5:45 PM Marcus S <shad...@gmail.com> wrote:
Yes, I think it makes sense in Kubernetes, not Kubevirt.  Containers would also benefit from having QoS on storage, and I imagine this could be implemented at CSI by perhaps proprietary means from each provider, or leverage cgroups at the node. 

The question around whether or not this will happen soon is another thing. There seems to be a fair number of requests and #1902 is current, though it sounds like they would target Kubernetes 1.22 or 1.23 after cgroups v2.  The temptation is that we have control over what KubeVirt can do and it would be faster to shoehorn the feature in, even if it ultimately is redundant or conflicts with the Kubernetes version. I also agree somewhat with comments on the Kubernetes discussion, that putting these controls into the hands of the users via the VM APIs may not be helpful if there is no quota/limit for what the user can set.

Have you considered making a simple sidecar hook to inject the iotune into your VMs? That would be an even faster solution that you'd have complete control of.

+1

At the same time I wonder if we can find a volunteer to update our sidecar mechanism. Today the boilerplate to be created is quite large.

Wouldn't it be possible if we can write a meta sidecar with a binary that is simply executing a bash script (ADD'ed to the container) against the domxml?
The meta side car would effectively need to receive and send the original and modifed domxml via grpc ...

If we had this, then writing sidecars should be way way simpler than today.

+1 that would creating ad-hoc sidecars really much simpler.

Best regards,
Roman
 
Reply all
Reply to author
Forward
0 new messages