Vhost support in Kubevirt (old PR resurrection)

[Filip Gschwandtner]

Hi,
i'm interested in vhost support in Kubevirt (https://github.com/kubevirt/kubevirt/pull/3208 , cut into multiple PR tracked in https://github.com/kubevirt/kubevirt/issues/3405#issuecomment-659373177). You can see the high level use case referred in that PR in attached draw.io document (sheet "old PR - ovs-dpdk"). The use case that i'm interested in replaces OVS-DPDK with fd.io VPP(https://wiki.fd.io/view/VPP), see sheet "vpp replace". From Kubevirt perspective this presents no change to original PR as usespace-cni attached to OVS-DPDK also supports VPP.  For more use case information, see further below.
I not interested only to have some working implementation, but also to merge this functionality. For this reason I checked the PRs and it seems that these things stands between current state and merging:

  1.finish functionality implementation:
    1.1 resolve hugetblfs or other way how qemu and VPP can have access to vhost user socket
    1.2 i started from 4 virt launcher domain spec #5662 that contains also previous implementation tasks, but got some compilation problems -> rebase only?
    1.3 test it locally that it actually works
  2.do e2e test of this functionality (some pinging between VPP and VM through vhost, using VPP in test can maybe simplify the test setup of test host)
  3.write documentation
  4.help with user guide (this is done after merging, but i guess that you guys as developers of kubevirt need enough info about this functionality for writing user guide before merge)

Could you please confirm that after doing these things you don't see reason why not to merge the functionality? Before i consider investing effort i want to know whether there are no other known (but not written down) problems preventing merge (i.e. some qemu incompatibility, developer decision not to have it,...).

My use case:
It is basically a VNF service chaining use case. I have multiple VNFs, some as containers, but some as VMs. I need to provide high speed connections between them to make high performant VNF chaining. See sheet "hybrid vnf use case".  I want to use VPP as the router between them. I don't want to "hardcode" direct connection/tunnels between containers, therefore VPP routing. I want to use high performant connections like memifs(https://docs.fd.io/vpp/21.06/dc/dea/libmemif_doc.html) and vhost that use memory sharing for packet forwarding.

Why not to use SRVIO? Well, for use cases when most data traffic is within one node is vhost/memif more performant solution, see https://telcocloudbridge.com/blog/dpdk-vs-sr-iov-for-nfv-why-a-wrong-decision-can-impact-performance/ . Also, one kind of VNF service chaining optimization is to bring VNFs that are frequently called in one chain into one k8s node.
One of the benefits of vhost vs SR-IOV is also the performance on lower end, lower cost white box, edge/cCPE hardware ( e.g. CPU Celeron J3455 , or Atom C2000 , with I210 NICs). Utilising DPDK and vhost-user port, provides near wire speed bandwidth where VMs are deployed. With Vhost support, Kubevirt can offer competitive performance that already exist on other virtualization projects. SR-IOV makes more sense on higher end systems, e.g. Datacenter, but on the Edge the DPDK/VPP (or OVS) is widely used.

Also there were some concerns about usefulness(how many kubevirt users will find it usefull) of vhost kubevirt feature. From SRVIO reasoning, i can say that it can increase performance of many one node (or mostly on one node) solutions that will adopt vhost support (adoptiong vhost in VM is not for free, but for example DPDK can do the heavy lifting). One of such cases is service chaining, but i don't know how many other use cases from Kubevirt users can it influence.

Sincerely,

Filip Gschwandtner

Software developer,

Pantheon.tech (https://pantheon.tech/)

[Miguel Duarte de Mora Barroso]

On Tue, Nov 16, 2021 at 12:10 PM Filip Gschwandtner <fgschwan...@gmail.com> wrote:

Hi,
i'm interested in vhost support in Kubevirt (https://github.com/kubevirt/kubevirt/pull/3208 , cut into multiple PR tracked in https://github.com/kubevirt/kubevirt/issues/3405#issuecomment-659373177). You can see the high level use case referred in that PR in attached draw.io document (sheet "old PR - ovs-dpdk"). The use case that i'm interested in replaces OVS-DPDK with fd.io VPP(https://wiki.fd.io/view/VPP), see sheet "vpp replace". From Kubevirt perspective this presents no change to original PR as usespace-cni attached to OVS-DPDK also supports VPP.  For more use case information, see further below.
I not interested only to have some working implementation, but also to merge this functionality. For this reason I checked the PRs and it seems that these things stands between current state and merging:

  1.finish functionality implementation:
    1.1 resolve hugetblfs or other way how qemu and VPP can have access to vhost user socket
    1.2 i started from 4 virt launcher domain spec #5662 that contains also previous implementation tasks, but got some compilation problems -> rebase only?
    1.3 test it locally that it actually works
  2.do e2e test of this functionality (some pinging between VPP and VM through vhost, using VPP in test can maybe simplify the test setup of test host)
  3.write documentation
  4.help with user guide (this is done after merging, but i guess that you guys as developers of kubevirt need enough info about this functionality for writing user guide before merge)

I am not a KubeVirt maintainer, but I think you got the correct status here. 

 

Could you please confirm that after doing these things you don't see reason why not to merge the functionality? Before i consider investing effort i want to know whether there are no other known (but not written down) problems preventing merge (i.e. some qemu incompatibility, developer decision not to have it,...).

*I* don't see one. I would actually love to see this feature out in the wild.

 

My use case:
It is basically a VNF service chaining use case. I have multiple VNFs, some as containers, but some as VMs. I need to provide high speed connections between them to make high performant VNF chaining. See sheet "hybrid vnf use case".  I want to use VPP as the router between them. I don't want to "hardcode" direct connection/tunnels between containers, therefore VPP routing. I want to use high performant connections like memifs(https://docs.fd.io/vpp/21.06/dc/dea/libmemif_doc.html) and vhost that use memory sharing for packet forwarding.

Why not to use SRVIO? Well, for use cases when most data traffic is within one node is vhost/memif more performant solution, see https://telcocloudbridge.com/blog/dpdk-vs-sr-iov-for-nfv-why-a-wrong-decision-can-impact-performance/ . Also, one kind of VNF service chaining optimization is to bring VNFs that are frequently called in one chain into one k8s node.
One of the benefits of vhost vs SR-IOV is also the performance on lower end, lower cost white box, edge/cCPE hardware ( e.g. CPU Celeron J3455 , or Atom C2000 , with I210 NICs). Utilising DPDK and vhost-user port, provides near wire speed bandwidth where VMs are deployed. With Vhost support, Kubevirt can offer competitive performance that already exist on other virtualization projects. SR-IOV makes more sense on higher end systems, e.g. Datacenter, but on the Edge the DPDK/VPP (or OVS) is widely used.

Thanks for detailing your use case; I think you make a very valid point: you list relevant strategic scenarios where this proposal's solution outperforms SRIOV.

I also think KubeVirt should have a highly performant networking solution for general purpose hardware.

Also there were some concerns about usefulness(how many kubevirt users will find it usefull) of vhost kubevirt feature. From SRVIO reasoning, i can say that it can increase performance of many one node (or mostly on one node) solutions that will adopt vhost support (adoptiong vhost in VM is not for free, but for example DPDK can do the heavy lifting). One of such cases is service chaining, but i don't know how many other use cases from Kubevirt users can it influence.

Sincerely,

Filip Gschwandtner

Software developer,

Pantheon.tech (https://pantheon.tech/)

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/d4f3fc01-1a25-4c97-bfa2-5a864b0a5dfcn%40googlegroups.com.

[Petr Horacek]

pá 19. 11. 2021 v 12:54 odesílatel Miguel Duarte de Mora Barroso <mdba...@redhat.com> napsal:

On Tue, Nov 16, 2021 at 12:10 PM Filip Gschwandtner <fgschwan...@gmail.com> wrote:

Hi,
i'm interested in vhost support in Kubevirt (https://github.com/kubevirt/kubevirt/pull/3208 , cut into multiple PR tracked in https://github.com/kubevirt/kubevirt/issues/3405#issuecomment-659373177). You can see the high level use case referred in that PR in attached draw.io document (sheet "old PR - ovs-dpdk"). The use case that i'm interested in replaces OVS-DPDK with fd.io VPP(https://wiki.fd.io/view/VPP), see sheet "vpp replace". From Kubevirt perspective this presents no change to original PR as usespace-cni attached to OVS-DPDK also supports VPP.  For more use case information, see further below.
I not interested only to have some working implementation, but also to merge this functionality. For this reason I checked the PRs and it seems that these things stands between current state and merging:

  1.finish functionality implementation:
    1.1 resolve hugetblfs or other way how qemu and VPP can have access to vhost user socket
    1.2 i started from 4 virt launcher domain spec #5662 that contains also previous implementation tasks, but got some compilation problems -> rebase only?
    1.3 test it locally that it actually works
  2.do e2e test of this functionality (some pinging between VPP and VM through vhost, using VPP in test can maybe simplify the test setup of test host)
  3.write documentation
  4.help with user guide (this is done after merging, but i guess that you guys as developers of kubevirt need enough info about this functionality for writing user guide before merge)

I am not a KubeVirt maintainer, but I think you got the correct status here. 

 

Could you please confirm that after doing these things you don't see reason why not to merge the functionality? Before i consider investing effort i want to know whether there are no other known (but not written down) problems preventing merge (i.e. some qemu incompatibility, developer decision not to have it,...).

*I* don't see one. I would actually love to see this feature out in the wild.

Thank you for the justification you provided below, Filip.

I checked offline with some of the KubeVirt stakeholders and there seems to be no issue with adding support for vhostuser to KubeVirt. We would welcome this feature in KubeVirt and help get it reviewed. I'm not aware of any compatibility issue neither.

The only problem we may have is the maintenance of this. Since we haven't seen a great demand to have this from the user side and since we don't have experience with this technology, we cannot commit to maintaining it ourselves (speaking for the Red Hat part of contributors). So, if this feature gets merged to KubeVirt, there will need to be a volunteer maintaining it in case there is a degradation on CI. If the feature gets broken for too long, it will have to be removed from the codebase. I'm raising this, to set the expectations for the "cost" of this clearly, as it probably won't be a one time investment. If there is a volunteer to do this, I think it would be an awesome addition to the project.

 

 

My use case:
It is basically a VNF service chaining use case. I have multiple VNFs, some as containers, but some as VMs. I need to provide high speed connections between them to make high performant VNF chaining. See sheet "hybrid vnf use case".  I want to use VPP as the router between them. I don't want to "hardcode" direct connection/tunnels between containers, therefore VPP routing. I want to use high performant connections like memifs(https://docs.fd.io/vpp/21.06/dc/dea/libmemif_doc.html) and vhost that use memory sharing for packet forwarding.

Why not to use SRVIO? Well, for use cases when most data traffic is within one node is vhost/memif more performant solution, see https://telcocloudbridge.com/blog/dpdk-vs-sr-iov-for-nfv-why-a-wrong-decision-can-impact-performance/ . Also, one kind of VNF service chaining optimization is to bring VNFs that are frequently called in one chain into one k8s node.
One of the benefits of vhost vs SR-IOV is also the performance on lower end, lower cost white box, edge/cCPE hardware ( e.g. CPU Celeron J3455 , or Atom C2000 , with I210 NICs). Utilising DPDK and vhost-user port, provides near wire speed bandwidth where VMs are deployed. With Vhost support, Kubevirt can offer competitive performance that already exist on other virtualization projects. SR-IOV makes more sense on higher end systems, e.g. Datacenter, but on the Edge the DPDK/VPP (or OVS) is widely used.

Thanks for detailing your use case; I think you make a very valid point: you list relevant strategic scenarios where this proposal's solution outperforms SRIOV.

I also think KubeVirt should have a highly performant networking solution for general purpose hardware.

Also there were some concerns about usefulness(how many kubevirt users will find it usefull) of vhost kubevirt feature. From SRVIO reasoning, i can say that it can increase performance of many one node (or mostly on one node) solutions that will adopt vhost support (adoptiong vhost in VM is not for free, but for example DPDK can do the heavy lifting). One of such cases is service chaining, but i don't know how many other use cases from Kubevirt users can it influence.

Sincerely,

Filip Gschwandtner

Software developer,

Pantheon.tech (https://pantheon.tech/)

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/d4f3fc01-1a25-4c97-bfa2-5a864b0a5dfcn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAAx_ZVWcbDw2q8TEJR_y4LgPu0NuMT7G-juywrY5DTCHh7RPaA%40mail.gmail.com.

[Pooja Ghumre]

Hi,

We have also been looking to enable OVS-DPDK for VNF workloads running inside Kubevirt VMs. Using the changes from PR #3208, now able to create VMs using vhostuser interfaces. Can we please help get this PR reviewed and merged upstream? 

Also one issue right now is that the vhostuser ethernet interface exposed inside the guest is not seen from the virt-launcher pod and so DHCP IP assignment is not working using whereabouts plugin for IPAM. Any thoughts as to why the interface is not available inside the pod as well?

Thanks,

Pooja

[Miguel Duarte de Mora Barroso]

On Mon, Dec 13, 2021 at 8:27 PM Pooja Ghumre <po...@platform9.com> wrote:

Hi,

We have also been looking to enable OVS-DPDK for VNF workloads running inside Kubevirt VMs. Using the changes from PR #3208, now able to create VMs using vhostuser interfaces. Can we please help get this PR reviewed and merged upstream? 

Also one issue right now is that the vhostuser ethernet interface exposed inside the guest is not seen from the virt-launcher pod and so DHCP IP assignment is not working using whereabouts plugin for IPAM. Any thoughts as to why the interface is not available inside the pod as well?

Could you share the error messages you got ? 

I would not expect the vhostuser interface to be *visible* by the kernel (or iproute) within the pod. 

I would - on the other hand - expect userspace-cni to be able to set whatever IP was chosen by whereabouts on the vhostuser interface.

Finally, can you confirm if this works for pods ?... I.e. Can you have a pod with vhostuser interface configured via whereabouts ? 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/96866fae-bed2-4a40-974a-891bc8ee266cn%40googlegroups.com.

[Filip Gschwandtner]

Hi,

I just wanted to give you guys a update that i'm currently working on creating setup for vhost using Kubevirt PRs (more in https://github.com/kubevirt/kubevirt/pull/5662#issuecomment-1048888403).

Also thinking as stated in that comment, to make such setup public as nothing similar exists (or i'm not aware of). This could help others to kickstart their projects. However, my setup is just hacky changes to Kubevirt development environment setup (Kind + Kubevirt) and probably can't be used as nice example inside Kubevirt. However it could help later with CI test setup for example.

[Miguel Duarte de Mora Barroso]

On Wed, Feb 23, 2022 at 4:38 PM Filip Gschwandtner <fgschwan...@gmail.com> wrote:

Hi,

I just wanted to give you guys a update that i'm currently working on creating setup for vhost using Kubevirt PRs (more in https://github.com/kubevirt/kubevirt/pull/5662#issuecomment-1048888403).

Also thinking as stated in that comment, to make such setup public as nothing similar exists (or i'm not aware of). This could help others to kickstart their projects. However, my setup is just hacky changes to Kubevirt development environment setup (Kind + Kubevirt) and probably can't be used as nice example inside Kubevirt. However it could help later with CI test setup for example.

Thanks for the heads up. There have been kind based test lanes in the past - ipv6 for instance - and right now (unless it recently changed) SR-IOV.

Meaning, your approach isn't unheard of. 

It could help you dispel your doubts (and build up your confidence for the solution) if you'd present your approach / thoughts / challenges on one of the community calls. I'm personally very happy that someone is working on this, and I'd love to hear more about it.