Drop support for rootful VMs

[Orel Misan]

Hello everyone,

For the past year, non-root VMs have been the default in KubeVirt [1].

Rootful VMs are less secure, pose a maintenance burden and are not tested since March 2023 [2].

We plan to drop support for rootful VMs in v1.2.0.

Does anyone have any objections or concerns?

Your feedback is much appreciated,

Orel

[1] https://github.com/kubevirt/kubevirt/pull/8563

[2] https://github.com/kubevirt/project-infra/pull/2646

[Edward Haas]

On Mon, Oct 16, 2023 at 7:26 PM Orel Misan <omi...@redhat.com> wrote:

Hello everyone,

For the past year, non-root VMs have been the default in KubeVirt [1].

Rootful VMs are less secure, pose a maintenance burden and are not tested since March 2023 [2].

We plan to drop support for rootful VMs in v1.2.0.

Does anyone have any objections or concerns?

Nice, hope to see this happening.

- Are there any known cons for not being able to run rootful VMs?

- Could you please clarify what are the implications on existing running root VMs?

- Is there a plan on what needs to be done to drop the support? (e.g. remove the FG)

Your feedback is much appreciated,

Orel

[1] https://github.com/kubevirt/kubevirt/pull/8563

[2] https://github.com/kubevirt/project-infra/pull/2646

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALzEmMcCqZGc1cq3tPv9HLz2kzMjC2biaYz62iKfHveNEEkkTg%40mail.gmail.com.

[Orel Misan]

On Tue, Oct 17, 2023 at 10:19 AM Edward Haas <edw...@redhat.com> wrote:

On Mon, Oct 16, 2023 at 7:26 PM Orel Misan <omi...@redhat.com> wrote:

Hello everyone,

For the past year, non-root VMs have been the default in KubeVirt [1].

Rootful VMs are less secure, pose a maintenance burden and are not tested since March 2023 [2].

We plan to drop support for rootful VMs in v1.2.0.

Does anyone have any objections or concerns?

Nice, hope to see this happening.

- Are there any known cons for not being able to run rootful VMs?

Not that we are aware of at the moment. 

- Could you please clarify what are the implications on existing running root VMs?

Currently, there is a feature to migrate rootful VMs to be non-root (and vice versa).

The plan is to remove at least the non-root to root path.

The root to non-root could also potentially be removed, and it will require users to migrate the rootful VMs to non root before upgrading to v1.2.0.

 

- Is there a plan on what needs to be done to drop the support? (e.g. remove the FG)

There is a plan in the making, I will publish it soon.

The "root" feature gate will be removed. 

[Edward Haas]

On Tue, Oct 17, 2023 at 12:55 PM Orel Misan <omi...@redhat.com> wrote:

On Tue, Oct 17, 2023 at 10:19 AM Edward Haas <edw...@redhat.com> wrote:

On Mon, Oct 16, 2023 at 7:26 PM Orel Misan <omi...@redhat.com> wrote:

Hello everyone,

For the past year, non-root VMs have been the default in KubeVirt [1].

Rootful VMs are less secure, pose a maintenance burden and are not tested since March 2023 [2].

We plan to drop support for rootful VMs in v1.2.0.

Does anyone have any objections or concerns?

Nice, hope to see this happening.

- Are there any known cons for not being able to run rootful VMs?

Not that we are aware of at the moment. 

It would be useful to identify scenarios in which there is an effect.

For example, would hook sidecars that depend on some "root" access be affected?

- Could you please clarify what are the implications on existing running root VMs?

Currently, there is a feature to migrate rootful VMs to be non-root (and vice versa).

The plan is to remove at least the non-root to root path.

The root to non-root could also potentially be removed, and it will require users to migrate the rootful VMs to non root before upgrading to v1.2.0.

I guess this can be elaborated in the plan.

Just note that we may need to keep existing (old) VMs running without disruption,
automate their migration somehow or simply block an upgrade.

[Stu Gott]

On Tue, Oct 17, 2023 at 6:28 AM Edward Haas <edw...@redhat.com> wrote:

On Tue, Oct 17, 2023 at 12:55 PM Orel Misan <omi...@redhat.com> wrote:

On Tue, Oct 17, 2023 at 10:19 AM Edward Haas <edw...@redhat.com> wrote:

On Mon, Oct 16, 2023 at 7:26 PM Orel Misan <omi...@redhat.com> wrote:

Hello everyone,

For the past year, non-root VMs have been the default in KubeVirt [1].

Rootful VMs are less secure, pose a maintenance burden and are not tested since March 2023 [2].

We plan to drop support for rootful VMs in v1.2.0.

Does anyone have any objections or concerns?

Nice, hope to see this happening.

- Are there any known cons for not being able to run rootful VMs?

Not that we are aware of at the moment. 

It would be useful to identify scenarios in which there is an effect.

For example, would hook sidecars that depend on some "root" access be affected?

While I'm not aware of a use case requiring rootful VMs, there are plenty of non-public projects where we can't know if this is done. That's why we're asking in this forum. In the interests of acting on real data instead of guessing: please chime in if losing the ability to run a VM as root affects you in some way.

Feature Freeze for 1.2.0 is approximately 4 months from today (since today is 1.1.0 FF). I wonder if we might be being too aggressive with that timeline. Maybe it might be a good idea to follow a deprecation period where KubeVirt warns users that are using root VMs that they're going away--thus giving users time to react in case they're not paying attention to this email list.

Whether or not we continue to allow root-enabled VMs to be created, a point Orel raised up front will apply: our automated testing does not cover this case anymore.

 

- Could you please clarify what are the implications on existing running root VMs?

Currently, there is a feature to migrate rootful VMs to be non-root (and vice versa).

The plan is to remove at least the non-root to root path.

The root to non-root could also potentially be removed, and it will require users to migrate the rootful VMs to non root before upgrading to v1.2.0.

I guess this can be elaborated in the plan.

Just note that we may need to keep existing (old) VMs running without disruption,
automate their migration somehow or simply block an upgrade.

 

- Is there a plan on what needs to be done to drop the support? (e.g. remove the FG)

There is a plan in the making, I will publish it soon.

The "root" feature gate will be removed. 

Your feedback is much appreciated,

Orel

[1] https://github.com/kubevirt/kubevirt/pull/8563

[2] https://github.com/kubevirt/project-infra/pull/2646

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALzEmMcCqZGc1cq3tPv9HLz2kzMjC2biaYz62iKfHveNEEkkTg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALmkdFTG1kF4cdp9-%2BmzfFdAcDD8GB59%3DbG9x_VnQPHYSJTeLQ%40mail.gmail.com.

[Zang Li]

Well I understand that rootless is more secure, in reality we do have cases that rootless doesn't work, for example due to storage limitations, or some other factors. In CDI, we recently had a thread and request to make root as a feature gate (https://github.com/kubevirt/containerized-data-importer/issues/2919). So can we keep this feature gate a bit longer and give customers an escape hatch when rootless doesn't work for them? What is the downside of keeping it?

Thanks,

Zang

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAH7ySct3dgppKbHey0q1Vg5sucEjqVM8L9U4C-LzCnwwnLU2Uw%40mail.gmail.com.

[Mark Lavi]

This was discussed in today's community meeting: it was acknowledged where rootful is still needed; however it is not tested, therefore its functionality as well as support of it is uncertain and currently unsustainable. Hence the proposal to drop it and remove the burden of maintaining the code associated as well as future design, test, and support considerations. Further discussion is encouraged for adopting or dropping (with KubeVirt releases warning of deprecation), a final decision has not been made.

[Luboslav Pivarc]

Hi Zang,

Can you please elaborate on which cases rootless VMs don't work in order for us to tackle them in the next release?

-Lubo

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAO_S94gmMkBZQqsebguHDRGzZ7uYicpUrG2aBYx4cc-0%3DC-HLg%40mail.gmail.com.

[Zang Li]

Mark, thanks for the update!

Lubo, in my case, rootless didn't work when I used the macvtap plugin. The trace just showed that the domain is not ready as below (this is on v1.59.0)

{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:504","timestamp":"2023-06-05T22:23:45.736904Z"}

{"component":"virt-launcher","level":"info","msg":"Starting metrics server","pos":"virt-launcher.go:510","timestamp":"2023-06-05T22:23:45.739317Z"}

{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:537","timestamp":"2023-06-05T22:23:45.740266Z"}

{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:80","timestamp":"2023-06-05T22:23:45.740771Z"}

panic: timed out waiting for domain to be defined

goroutine 1 [running]:

main.waitForDomainUUID(0xc00010e5a0?, 0xc0003fea20, 0xc00033c3c0, {0x1cdf9a8, 0xc0003f40f0})

/workspace/cmd/virt-launcher/virt-launcher.go:251 +0x434

main.main()

/workspace/cmd/virt-launcher/virt-launcher.go:474 +0x117a

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 11 with status 512","pos":"virt-launcher-monitor.go:124","timestamp":"2023-06-05T22:29:17.748168Z"}

{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:142","timestamp":"2023-06-05T22:29:17.748343Z"}

I didn't have time to dig further on why so I used the root mode to bypass it. We do have some customized changes on macvtap, so it is not clear to me if OSS will have the same issue. It may be worth testing it out. 

I also didn't check if all other tests would pass with the rootless mode either.

Thanks,

Zang   

[Luboslav Pivarc]

Thanks for the update,

Please don't hesitate to open an issue with [rootless] in the title and logs attached (we will have a look during the next release). This is the pure reason for this mail, collect any last missing pieces. Of course, we appreciate hunting down these issues.

For now, we can settle on "Let's file issues from users, and let's revisit this during the next release cycle".

How does that sound?

-Lubo

[Zang Li]

Hi Lubo, 

Because we have some customized changes on macvtap, it is not clear to me if OSS would hit the same issue, so I didn't file the bug. I wanted to revisit this but don't have bandwidth for it. 

I understand the lack of test courage if we keep the root feature gate, but my intuition is that if things work with less permission, it should just work with more permission (as long as we don't treat more permission as a violation). If things got broken accidently, users who use root mode shall find it and file issues to fix it. So we can leave it as is for now and revisit it later. 

Thanks,

Zang

  

[Orel Misan]

Hi,

Rootful lanes are still being executed in KubeVirt's CI system:

Periodic rootful lanes:

- periodic-kubevirt-e2e-k8s-1.26-sig-storage-root

- periodic-kubevirt-e2e-k8s-1.26-sig-compute-root

- periodic-kubevirt-e2e-k8s-1.26-sig-operator-root

Two optional presubmit lanes were recently added by kubevirt/project-infra#3011 [1]:

- pull-kubevirt-e2e-k8s-1.26-sig-storage-root
- pull-kubevirt-e2e-k8s-1.26-sig-compute-root

[1] https://github.com/kubevirt/project-infra/pull/3011

Orel

[Zang Li]

Hi Orel,

Good to know that!  Would this automatically carry over when the k8s version updates? 

Thanks,

Zang

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALzEmMccYJoAbE0uTL%2Bi__Eczht80xzbw024fi_cUCMPL58Bzg%40mail.gmail.com.

[Stu Gott]

On Mon, Oct 23, 2023 at 12:11 PM Zang Li <zan...@google.com> wrote:

Hi Orel,

Good to know that!  Would this automatically carry over when the k8s version updates? 

We usually only target compatibility with the last three k8s releases. We would have to intentionally re-create new test lanes when 1.26 is retired.

[Edward Haas]

If the intention is to go forward with this initiative, I suggest composing a suggestion with timelines.

I think we raised the option to prolong the time until this is fully deprecated (e.g. 2 or 3 versions).

Having in the middle some warnings raised about the targeted deprecation.

We could always backoff if needed per community feedback.

On Mon, Oct 23, 2023 at 7:41 PM Stu Gott <sg...@redhat.com> wrote:

On Mon, Oct 23, 2023 at 12:11 PM Zang Li <zan...@google.com> wrote:

Hi Orel,

Good to know that!  Would this automatically carry over when the k8s version updates? 

We usually only target compatibility with the last three k8s releases. We would have to intentionally re-create new test lanes when 1.26 is retired.

 

Thanks,

Zang

On Sun, Oct 22, 2023 at 3:52 AM Orel Misan <omi...@redhat.com> wrote:

Hi,

Rootful lanes are still being executed in KubeVirt's CI system:

Periodic rootful lanes:

- periodic-kubevirt-e2e-k8s-1.26-sig-storage-root

- periodic-kubevirt-e2e-k8s-1.26-sig-compute-root

- periodic-kubevirt-e2e-k8s-1.26-sig-operator-root

Two optional presubmit lanes were recently added by kubevirt/project-infra#3011 [1]:

- pull-kubevirt-e2e-k8s-1.26-sig-storage-root
- pull-kubevirt-e2e-k8s-1.26-sig-compute-root

[1] https://github.com/kubevirt/project-infra/pull/3011

While these exist and run, they are not gating.

It is not clear if they got left behind intentionally or are just leftovers.
The fact that they have not been bumped to use the latest k8s release is a warning sign.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAH7yScs403U_NTX2BmdmYmON01PiK0LXmOb7CW6ROP%3Dy6h%3D5Hg%40mail.gmail.com.

[Luboslav Pivarc]

On Tue, Oct 24, 2023 at 7:00 AM Edward Haas <edw...@redhat.com> wrote:

If the intention is to go forward with this initiative, I suggest composing a suggestion with timelines.

I think we raised the option to prolong the time until this is fully deprecated (e.g. 2 or 3 versions).

Having in the middle some warnings raised about the targeted deprecation.

I fully agree. I think it is reasonable to raise deprecation now and preliminary set the removal to 1.3 release. During the 1.2 release, we can gather feedback and let users see if they can run with default (non root). If this timeline is found too aggressive during the 1.2 release we can push the removal even further.

The deprecation will also help us in gathering feedback as users will be slowly switching to default.

What do others think?

@Zang Li  Can you check if https://github.com/kubevirt/kubevirt/pull/9686 fixes your reported issue?

-Lubo

[Miguel Duarte de Mora Barroso]

On Tue, Oct 24, 2023 at 12:08 PM Luboslav Pivarc <lpi...@redhat.com> wrote:

On Tue, Oct 24, 2023 at 7:00 AM Edward Haas <edw...@redhat.com> wrote:

If the intention is to go forward with this initiative, I suggest composing a suggestion with timelines.

I think we raised the option to prolong the time until this is fully deprecated (e.g. 2 or 3 versions).

Having in the middle some warnings raised about the targeted deprecation.

I fully agree. I think it is reasonable to raise deprecation now and preliminary set the removal to 1.3 release. During the 1.2 release, we can gather feedback and let users see if they can run with default (non root). If this timeline is found too aggressive during the 1.2 release we can push the removal even further.

The deprecation will also help us in gathering feedback as users will be slowly switching to default.

What do others think?

I think it's reasonable and fair to both the community and the maintainers. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAFTR0GXPpGOO%2BDo8Csodxn1VKCZDrhbKVd_QKAKJYZZ1YnTfFw%40mail.gmail.com.

[Orel Misan]

Hello,

I posted a proposal for the removal of the rootful VMs:
https://github.com/kubevirt/community/pull/248

Your feedback is much appreciated.

Orel

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAAx_ZVVudLAvpKUCQ_KO4hSzW5tuwUMpeh4AZGHdYbq3yo4Dig%40mail.gmail.com.

[Zang Li]

Hi Lubos,

Thank you for the patch! A quick try shows that it fixed the issue we hit earlier. 

Thanks,

Zang

[Or Mergi]

Hey, 

I think its good opportunity to start adding deprecation notes in our release notes.

How are release notes generated?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAO_S94ig_sqMT4k1_fohB3WVgpH2dVCWiZPpqwwZBNMnqCCkCQ%40mail.gmail.com.

--

Or Mergi

[Igor Bezukh]

Hi,

On Sun, Oct 29, 2023 at 5:01 PM Or Mergi <ome...@redhat.com> wrote:

Hey, 

I think its good opportunity to start adding deprecation notes in our release notes.

How are release notes generated?

 

They are generated by the release tool, whose code is in project-infra. We can add a deprecation section there.

I can't find any documentation about rootful and non-rootful VMs, maybe it is worth creating one, and note about rootful being deprecated there as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJ38DpDXPeMazoJDFaSfaRGAOKLhE0gubPP5T23sy3hh5cJpxA%40mail.gmail.com.

[Andrew Burden]

I'll likely manually sort the release notes by type in the tag and generate the same for the user guide. You can send me the copy and link to relevant PR and I can make sure it's included.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CABUdoDdU%2BpaeTRpkd_%2BGq3t8G%3D0aU5Gkw5WS007g017iRBQOKg%40mail.gmail.com.

[Orel Misan]

Hello everyone,

Following the proposal in PR [1] I would like to suggest the deprecation of rootful VMs starting v1.3.0.
Before proceeding, I wanted to ask if there are any objections or concerns from the community.
I wanted to begin by posting a PR to warn users when the `Root` feature gate is set on the KubeVirt CR.

Please feel free to share your thoughts by replying to this email.

Thank you.
Orel

[1] https://github.com/kubevirt/community/pull/248

[Ryan Hallisey US]

Can we time this with the beta release of DRA in Kubernetes?  Some of the limitations in the device-plugin framework can be worked around by allowing the launcher to run as root and I don't think beta for DRA is far away.

[Fabian Deutsch]

Ryan,

sounds reasonable. Would you be able to work on DRA support for KubeVirt?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/54edaf7d-51a4-48e8-bb5b-357294125bcen%40googlegroups.com.

[Ryan Hallisey US]

Yes - Nvidia is going to build and open source a kubevirt dra driver which we'll be supporting in addition to the current one: https://github.com/NVIDIA/kubevirt-gpu-device-plugin.

-Ryan

[Fabian Deutsch]

Ryan, will you also take care of adding DRA support ot kubevirt itself?

@Alice Frosi isn't there a related GSOC project?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/e6d33bb2-9905-452a-8135-b10ce35a142fn%40googlegroups.com.

[Alice Frosi]

Hi Fabian, hi Ryan,

On Fri, May 17, 2024 at 2:00 PM Fabian Deutsch <fdeu...@redhat.com> wrote:

Ryan, will you also take care of adding DRA support ot kubevirt itself?

@Alice Frosi isn't there a related GSOC project?

Yes, we have a GSoC project [1] focusing on DRA. However, we'll be using NVMe devices not GPUs. It is easier since we can use emulated NVMes from kubevirtci.

@Sibasish Behera has just started to work on it. Right now, we are looking at an independent DRA drivers, once this is prototyped then, we can think how to integrate this with KubeVIrt.

@Ryan Hallisey US, Do you already have some public repository with your DRA driver?

[1] https://github.com/kubevirt/community/issues/254

[Ryan Hallisey]

The kubevirt gpu driver is still in its early stages so we don't have anything to show yet.

Drivers are vendor specific and there will be many. We need to align in the community on how KubeVirt can consume external drivers.
I can start a proposal.

-Ryan

________________________________________
From: kubevi...@googlegroups.com <kubevi...@googlegroups.com> on behalf of Alice Frosi <afr...@redhat.com>
Sent: Friday, May 17, 2024 8:25 AM
To: Fabian Deutsch
Cc: Ryan Hallisey; kubevirt-dev; Sibasish Behera; Victor Toso de Carvalho
Subject: Re: [kubevirt-dev] Drop support for rootful VMs

External email: Use caution opening links or attachments

Hi Fabian, hi Ryan,

On Fri, May 17, 2024 at 2:00 PM Fabian Deutsch <fdeu...@redhat.com<mailto:fdeu...@redhat.com>> wrote:
Ryan, will you also take care of adding DRA support ot kubevirt itself?

@Alice Frosi<mailto:afr...@redhat.com> isn't there a related GSOC project?

Yes, we have a GSoC project [1] focusing on DRA. However, we'll be using NVMe devices not GPUs. It is easier since we can use emulated NVMes from kubevirtci.

@Sibasish Behera<mailto:beherasi...@gmail.com> has just started to work on it. Right now, we are looking at an independent DRA drivers, once this is prototyped then, we can think how to integrate this with KubeVIrt.

@Ryan Hallisey US<mailto:rhal...@nvidia.com>, Do you already have some public repository with your DRA driver?

[1] https://github.com/kubevirt/community/issues/254

On Mon, Apr 29, 2024 at 9:05 PM 'Ryan Hallisey US' via kubevirt-dev <kubevi...@googlegroups.com<mailto:kubevi...@googlegroups.com>> wrote:
Yes - Nvidia is going to build and open source a kubevirt dra driver which we'll be supporting in addition to the current one: https://github.com/NVIDIA/kubevirt-gpu-device-plugin.

-Ryan

On Monday, April 29, 2024 at 2:57:01 PM UTC-4 Fabian Deutsch wrote:
Ryan,

sounds reasonable. Would you be able to work on DRA support for KubeVirt?

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com<mailto:kubevirt-dev...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CABBoX7OGttNPawytuZ-4XQJzOf8Y130TkeACF-JNJTN6YyRYaw%40mail.gmail.com<https://groups.google.com/d/msgid/kubevirt-dev/CABBoX7OGttNPawytuZ-4XQJzOf8Y130TkeACF-JNJTN6YyRYaw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Alice Frosi]

On Fri, May 17, 2024 at 3:14 PM Ryan Hallisey <rhal...@nvidia.com> wrote:

The kubevirt gpu driver is still in its early stages so we don't have anything to show yet.

Drivers are vendor specific and there will be many.  We need to align in the community on how KubeVirt can consume external drivers.
I can start a proposal.

Yes, I agree. We would like to have a basic understanding and a PoC of how a DRA plugin would look like before thinking of external plugins. 

I think, in kubevirt we could either have something integrated in virt-handler or maybe optionally installable. However, for basic PCI passthrough, we should, at least, guarantee that the device isn't released at every VM restarts. For more fancy device discovery and allocation, this would require, for sure, an external driver as it is the case for CSI plugins.

Alice

[Fabian Deutsch]

On Fri, May 17, 2024 at 3:27 PM Alice Frosi <afr...@redhat.com> wrote:

On Fri, May 17, 2024 at 3:14 PM Ryan Hallisey <rhal...@nvidia.com> wrote:

The kubevirt gpu driver is still in its early stages so we don't have anything to show yet.

Drivers are vendor specific and there will be many.  We need to align in the community on how KubeVirt can consume external drivers.
I can start a proposal.

Yes, I agree. We would like to have a basic understanding and a PoC of how a DRA plugin would look like before thinking of external plugins. 

I think, in kubevirt we could either have something integrated in virt-handler or maybe optionally installable. However, for basic PCI passthrough, we should, at least, guarantee that the device isn't released at every VM restarts. For more fancy device discovery and allocation, this would require, for sure, an external driver as it is the case for CSI plugins.

What I was thinking of

1. GSOC for kubevirt in-tree driver for NVME

2. NV for out-of-tree driver for GOU

3. DRA API support in VM API.

Who is taking care of #3?

I'm expecting to - in the end:

apiVersion: v1
kind: VM
metadata:
  name: vm-fast-w-gpu
spec:

  # folloowing in template or on spec level?

  resourceClaims:
  - name: gpu-a100
    source:
      resourceClaimTemplateName: gpu-a100
  - name: nvme-fast
    source:
      resourceClaimTemplateName: nvme-fast

AKA integration of the DRA API stanza on the VM level.

Do we plan something like this, if yes, who is doing it?

[Ryan Hallisey]

I started a design proposal. Let's discuss in the community repo: https://github.com/kubevirt/community/pull/293.

________________________________________
From: Fabian Deutsch <fdeu...@redhat.com>
Sent: Friday, May 17, 2024 10:13 AM
To: Alice Frosi

Cc: Ryan Hallisey; kubevirt-dev; Sibasish Behera; Victor Toso de Carvalho
Subject: Re: [kubevirt-dev] Drop support for rootful VMs

External email: Use caution opening links or attachments

On Fri, May 17, 2024 at 3:27 PM Alice Frosi <afr...@redhat.com<mailto:afr...@redhat.com>> wrote:

From: kubevi...@googlegroups.com<mailto:kubevi...@googlegroups.com> <kubevi...@googlegroups.com<mailto:kubevi...@googlegroups.com>> on behalf of Alice Frosi <afr...@redhat.com<mailto:afr...@redhat.com>>

Sent: Friday, May 17, 2024 8:25 AM
To: Fabian Deutsch
Cc: Ryan Hallisey; kubevirt-dev; Sibasish Behera; Victor Toso de Carvalho
Subject: Re: [kubevirt-dev] Drop support for rootful VMs

External email: Use caution opening links or attachments

Hi Fabian, hi Ryan,

On Fri, May 17, 2024 at 2:00 PM Fabian Deutsch <fdeu...@redhat.com<mailto:fdeu...@redhat.com><mailto:fdeu...@redhat.com<mailto:fdeu...@redhat.com>>> wrote:
Ryan, will you also take care of adding DRA support ot kubevirt itself?

@Alice Frosi<mailto:afr...@redhat.com<mailto:afr...@redhat.com>> isn't there a related GSOC project?

Yes, we have a GSoC project [1] focusing on DRA. However, we'll be using NVMe devices not GPUs. It is easier since we can use emulated NVMes from kubevirtci.

@Sibasish Behera<mailto:beherasi...@gmail.com<mailto:beherasi...@gmail.com>> has just started to work on it. Right now, we are looking at an independent DRA drivers, once this is prototyped then, we can think how to integrate this with KubeVIrt.

@Ryan Hallisey US<mailto:rhal...@nvidia.com<mailto:rhal...@nvidia.com>>, Do you already have some public repository with your DRA driver?

[1] https://github.com/kubevirt/community/issues/254

On Mon, Apr 29, 2024 at 9:05 PM 'Ryan Hallisey US' via kubevirt-dev <kubevi...@googlegroups.com<mailto:kubevi...@googlegroups.com><mailto:kubevi...@googlegroups.com<mailto:kubevi...@googlegroups.com>>> wrote:
Yes - Nvidia is going to build and open source a kubevirt dra driver which we'll be supporting in addition to the current one: https://github.com/NVIDIA/kubevirt-gpu-device-plugin.

-Ryan

On Monday, April 29, 2024 at 2:57:01 PM UTC-4 Fabian Deutsch wrote:
Ryan,

sounds reasonable. Would you be able to work on DRA support for KubeVirt?

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com<mailto:kubevirt-dev%2Bunsu...@googlegroups.com><mailto:kubevirt-dev...@googlegroups.com<mailto:kubevirt-dev%2Bunsu...@googlegroups.com>>.