Support for KVM when using nvproxy

[Colin Weld]

Hey all,

I'm curious if there has been any movement on supporting the KVM platform with nvproxy (or perhaps it is supported, and I'm not aware). We're interested in this for performance reasons.

According to the nvproxy design doc, the blocker seems to be that, for KVM, the Sentry would have to add the right caching-related flags to its page table for mapped device memory; whereas this is not a problem with systrap and ptrace because there is no gvisor-managed page table. Is this still the primary issue, and if so, is fixing it anywhere on the gvisor roadmap, or is it something you'd accept contributions for?

Thanks!

[Colin Weld]

Also, happy to move this to a github issue instead!

[Colin Weld]

Thanks so much! It's good to hear that this might be possible. I'm likely to test your suggested changes sometime this month - I'll let you know how it goes.

On Wed, Nov 6, 2024 at 6:54 PM Jamie Liu <jami...@google.com> wrote:

Forgot to mention that you'd also need to add MAP_SHARED|MAP_FIXED_NOREPLACE to syscall filters: https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/runsc/boot/filter/config/config_main.go#L164

On Wed, Nov 6, 2024 at 11:42 AM Jamie Liu <jami...@google.com> wrote:

I'm less confident now that caching-related page table flags will be an issue, at least on x86; IIUC both Intel VT-x and AMD SVM effectively take the least caching specified by both guest page tables and EPT/NPT respectively, and I think KVM should configure memory type in EPT/NPT consistently with host page tables. The major known problem is mmap of /dev/nvidia-uvm, as described by the third paragraph of section "Unified Virtual Memory (UVM)"; in short, /dev/nvidia-uvm must be mapped into the sentry at specific addresses, which might conflict with existing mappings in the sentry.

How much of a problem this is is unclear. Apps might in practice consistently map /dev/nvidia-uvm at a fixed address that happens not to conflict with any existing sentry mappings, in which case simply enabling nvproxy and nvidia-uvm mmap on platform/kvm might just work. I think this could be attempted fairly easily:

- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/platform/kvm/kvm.go#L70 to platform.DoesNotOwnPageTables
- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/devices/nvproxy/uvm_mmap.go#L80 to use fsutil.PreciseHostFileMapper, as in https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/devices/tpuproxy/vfio/pci_device_fd_mmap.go#L78
- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/fsutil/precise_host_file_mapper.go#L189 to pass address hint fr.Start (instead of 0) and pass flag MAP_FIXED_NOREPLACE (in addition to MAP_SHARED)

If you do try this, please let us know how it goes. If apps instead consistently map /dev/nvidia-uvm at a fixed address that *does* conflict with sentry mappings, it might still be possible to work around the issue by messing around with build flags to place runsc executable mappings at specific (non-conflicting) addresses, but I have little experience with this.

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gvisor-users/fe3a6bd0-715b-458e-a4ef-f14e24d8cf3cn%40googlegroups.com.

[Jamie Liu]

I'm less confident now that caching-related page table flags will be an issue, at least on x86; IIUC both Intel VT-x and AMD SVM effectively take the least caching specified by both guest page tables and EPT/NPT respectively, and I think KVM should configure memory type in EPT/NPT consistently with host page tables. The major known problem is mmap of /dev/nvidia-uvm, as described by the third paragraph of section "Unified Virtual Memory (UVM)"; in short, /dev/nvidia-uvm must be mapped into the sentry at specific addresses, which might conflict with existing mappings in the sentry.

How much of a problem this is is unclear. Apps might in practice consistently map /dev/nvidia-uvm at a fixed address that happens not to conflict with any existing sentry mappings, in which case simply enabling nvproxy and nvidia-uvm mmap on platform/kvm might just work. I think this could be attempted fairly easily:

- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/platform/kvm/kvm.go#L70 to platform.DoesNotOwnPageTables
- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/devices/nvproxy/uvm_mmap.go#L80 to use fsutil.PreciseHostFileMapper, as in https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/devices/tpuproxy/vfio/pci_device_fd_mmap.go#L78
- Change https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/pkg/sentry/fsutil/precise_host_file_mapper.go#L189 to pass address hint fr.Start (instead of 0) and pass flag MAP_FIXED_NOREPLACE (in addition to MAP_SHARED)

If you do try this, please let us know how it goes. If apps instead consistently map /dev/nvidia-uvm at a fixed address that *does* conflict with sentry mappings, it might still be possible to work around the issue by messing around with build flags to place runsc executable mappings at specific (non-conflicting) addresses, but I have little experience with this.

On Wed, Nov 6, 2024 at 11:19 AM 'Colin Weld' via gVisor Users [Public] <gvisor...@googlegroups.com> wrote:

--

[Jamie Liu]

Forgot to mention that you'd also need to add MAP_SHARED|MAP_FIXED_NOREPLACE to syscall filters: https://github.com/google/gvisor/blob/30c58b6a6bbdf9c6fa7759fa1b54cd96145940c9/runsc/boot/filter/config/config_main.go#L164

On Wed, Nov 6, 2024 at 11:42 AM Jamie Liu <jami...@google.com> wrote: