# lspci
....
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman XT [Radeon HD 6970]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
GRUB_CMDLINE_LINUX=".... rd.qubes.hide_pci=03:00.0,03:00.1 modprobe=xen-pciback.passthrough=1 xen-pciback.permissive"
GRUB_CMDLINE_XEN_DEFAULT="... dom0_mem=min:1024M dom0_mem=max:4096M"
GRUB_CMDLINE_XEN_DEFAULT="... apic_verbosity=debug loglvl=all guest_loglvl=all iommu=verbose"
# grub2-mkconfig -o /boot/grub2/grub.cfg
# xl dmesg
...
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
# xl pci-assignable list
0000:03:00.0
0000:03:00.1
# dd if=/dev/zero of=win8.img bs=1M count=30000
# dd if=/dev/zero of=win8-user.img bs=1M count=30000
# qubes-dom0-update vnc
# xl create win8.hvm -V
# xl vncviewer win8
# losetup (Check for free loop device)
# losetup -P /dev/loop10 win8-user.img (Setup loop device and scan partition. Assuming loop10 is free)
# mount /dev/loop10p1 /mnt/removable ( Mount the first partition )
fwcfg.sh:
#!/bin/bash
vmip=$1
iptables -A FORWARD -s $vmip -p udp -d 10.137.1.1 --dport 53 -j ACCEPT
iptables -A FORWARD -s $vmip -p udp -d 10.137.1.254 --dport 53 -j ACCEPT
iptables -A FORWARD -s $vmip -p icmp -j ACCEPT
iptables -A FORWARD -s $vmip -p tcp -d 10.137.255.254 --dport 8082 -j DROP
iptables -A FORWARD -s $vmip -j ACCEPT
# sudo ./fwcfg.sh 10.137.2.50 # substitute with the win8.1 VM ip address
lbxl: error: .... The kernel doesn't support reset from sysfs for PCI device ...
wow cool.
Would this mean, I can in some way (extra manual work) use the full GPU power in a WindowsVM or a LinuxVM, without security issues for the hole QubesOS System?
(Or should I first use this setup on a separate machine or some Qubes-Qubes Dual boot machine).
Kind Regards
pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled
3877 /* 3878 * Many Intel PCH root ports do provide ACS-like features to disable peer 3879 * transactions and validate bus numbers in requests, but do not provide an 3880 * actual PCIe ACS capability. This is the list of device IDs known to fall 3881 * into that category as provided by Intel in Red Hat bugzilla 1037684. 3882 */This relates to this patch
Lack of ACS may not be a big deal to many. But it may limit isolation in
some cases, most notably having storage on PCIe slot connected SSDs and
GPU passthrough. And passing through more than a single GPU to different
VMs might have some isolation related hazards too because of the usual
PCIe slot arrangement. But one likely needs deep pockets to have such
arrangements anyway, so going to server or high-end platform may be less
of a issue to begin with :-).
> One thing about FLReset (Function Level Reset): There's quite general
> misconception about FLR being a requirement in order to do GPU passthrough,
> but this isn't true. As a matter of fact, not even the NVidia Quadros have
> FLR+ in PCI DevCaps, and not many non-GPU PCI devices do either. So even
> though the how-to here (http://wiki.xen.org/wiki/VTd_HowTo) states
> otherwise, the missing FLR capability will not necessarily mean that device
> can't be used in VM, but could only make it harder to survive DomU boot.
> I've seen in my tests that both Win 7 and Win8 VMs can be in fact booted
> several times without a requirement to boot Dom0 (but hopping BETWEEN the
> two Windows versions will result in either BSOD or Code 43). But again, this
> may wary a lot with GPU models and driver versions. But anyway, if you see
> this message during VM startup:
> lbxl: error: .... The kernel doesn't support reset from sysfs for PCI
> device ...
> ... you can safely ignore it
FLR is "needed" for reseting the device "safely" (after first init, if a
reset is needed), not for the passthrough.
Add this file to /etc/X11/xorg.conf.d/20-nouveau.conf
```
Section "Device"
Identifier "NVidia Card"
Driver "modesetting"
BusID "PCI:0:5:0"
EndSection
```
Note that the PCI address is the address that the GPU has inside the VM (use lspci IN the vm to find out that). Also "pci_msitranslate=0" has to be set in VM configuration, otherwise VM will hang when X is started.
This is tested with Arch linux (up to date 8.7.2016), with Linux 4.6.3-1-ARCH, modesetting and X versions 1.18.3.
-----
Ok, now that it's proven that newer Nvidia cards CAN in fact be passed through in Xen, I tried the official NVidia binary driver, but it failed with error message "The NVIDIA GPU at PCI:0:5:0 is not supported by the 367.27 NVIDIA driver".
I think that's the proprietary driver refusing to work when it detects that it's running under hypervisor (the Code 43 issue in Windows). Since KVM has for a while supported hiding both the "KVMKVMKVMKVM" signature (with "-cpu kvm=off" flag) as well as the Viridian hypervisor signature ("-cpu hv_vendor_id="..." flag), and currently there's no such functionality in Xen, I patched it in quite similar way to what Alex Willimson did to KVM.
Attached is a patch for Xen 4.6.1 that spoofs Xen signature ("XenVMMXenVMM" to "ZenZenZenZen") and Viridian signature ("Microsoft Hv" to "Wetware Labs") when "spoof_xen=1" and "spoof_viridian=1" are added VM configuration file.
The signatures are currently hard-coded, and currently there's no way to modify them (beyond re-compiling Xen), since HVMLoader also uses a hard-coded string to detect Xen and there's no API (understandably) to change that signature in real-time.
WARNING! In case you try the patch, you MUST re-compile and install also core-libvirt (in addition to vmm-xen) packages. Otherwise starting all DomUs will fail! You have been warned :)
-----------------
With this patch, the *NVidia binary driver (version 367.27) works also on Arch Linux* :)
However this was not enough on Windows 7 and 8.1 VMs (driver version 368.39) still announce Error 43 :(
I would love if others could test this as well. Maybe the Windows driver uses some other functionality to check for hypervisor, or maybe it's not a spoofing issue at all.
More investigation coming in..
Forgot to add that if spoofing is turned on for an already-installed Windows VM, there was a BSOD during boot (Windows really doesn't like if hypervisor suddenly disappears..). Re-installing Windows (with spoofing on) fixes this (maybe fixing installation with rescue CD could work also, but I did not test that).
- Tried Core2Duo CPUID from KVM VM
- Ported NoSnoop patch from KVM
Sadly, neither of these would help with BSODs / Code 43 errors.
I posted the results (with patches and more detailed information) on Xen-devel (https://lists.xenproject.org/archives/html/xen-devel/2016-07/msg01713.html). I hope the experts there might have more suggestions.
I'm guessing that it has to do with the nvidia-specific quirks implemented where the PCI BAR's are used to access the PCI Config Space and other BAR's. See: https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c
I think before others worked around it somewhat by reserving the memory and having 1:1 mappings ( http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through ), but that isn't really a proper solution.
I'm bit confused with this
> Edit /etc/default/grub and add following options (change the pci address if needed)
Which version of Qubes is this? Aint 3.1 EFI-only?
And EFI version of kernel args are to be passed via /boot/efi/EFI/qubes (kernel=)?
regards,
Tom
Hi Tom,
I use 3.1 and 3.2 rc2. Actually I haven't thought about this before. It seems on my system the default state is 'BIOS compatibilty mode' even though it's a new motherboard which is running on UEFI firmware. As for the partition table type on my SDD, it has always been 'dos' type MBR and that was never converted to GPT by Qubes Installer.
I'm not familiar how to configure EFI type bootloader, but it seems editing /boot/efi/EFI/qubes/xen.cfg should work. There's lots of discussion about it here: https://github.com/QubesOS/qubes-issues/issues/794
regards,
tom