VTdHowTo - Xen Wiki

[Peter Teoh]

http://wiki.xensource.com/xenwiki/VTdHowTo

How to turn on VT-d in Xen

1 ) cd xen-unstable.hg
2 ) make install
3 ) make linux-2.6-xen-config CONFIGMODE=menuconfig
4 ) change XEN->"PCI-device backend driver" from "M" to "*".
5 ) make linux-2.6-xen-build
6 ) make linux-2.6-xen-install
7 ) depmod 2.6.18.8-xen
8 ) mkinitrd -v -f --with=ahci --with=aacraid --with=sd_mod --with=scsi_mod initrd-2.6.18-xen.img 2.6.18.8-xen
9 ) cp initrd-2.6.18-xen.img /boot
10) lspci - select the PCI BDF you want to assign to guest OS
11) "hide" pci device from dom0 as following sample grub entry:
title Xen-Fedora Core (2.6.18-xen)
        root (hd0,0)
        kernel /boot/xen.gz com1=115200,8n1 console=com1 iommu=1
        module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro xencons=ttyS console=tty0 console=ttyS0, pciback.hide=(01:00.0)(03:00.0)
        module /boot/initrd-2.6.18-xen.img
12) reboot system
13) add "pci" line in /etc/xen/hvm.conf for to assigned devices
        pci = [ '01:00.0', '03:00.0' ]
15) start hvm guest and use "lspci" to see the passthru device and
    "ifconfig" to see if IP address has been assigned to NIC devices.

VT-d Works on OS

1) Host OS: PAE, 64-bit
2) Guest OS: 32-bit, PAE, 64-bit

Combinations Tested

1) 64-bit host: 32/PAE/64 Linux/XP/Win2003/Vista guests
2) PAE host: 32/PAE Linux/XP/Win2003/Vista guests

Enabled Systems

1) For VT-d enabling work on Xen, we have been using development systems using following Intel motherboards:
   - DQ35MP
   - DQ35JO

        Note from Intel on VT-d compatbility:
        VT-d is enabled on the following chipsets:
                Intel Q35
                Intel Q45
        The following chipsets have VT-d capability, but OEMs may not have it enabled on boards based on these:
                Intel X38
                Intel X48
        For Intel Desktop Boards, these have VT-d support enabled:
                Intel DQ35JO
                Intel DQ35MP
                Intel DX38BT
                Intel DX48BT2
                Intel DQ45CB (BIOS 0061 required, previous versions are known to cause problems)
                Intel DQ45EK
                Intel DX58SO
        For ASUS Desktop Boards, these have VT-d support enabled:
                ASUS P5E-VM DO (Intel Q35 chipset)
        For ASUS Desktop Boards, these DO NOT have VT-d support enabled:
                ASUS P5E (Intel X38 chipset)

2) As far as we know, following OEM systems also has vt-d enabled. Feel free to add others as they become available.
   - Dell: Optiplex 755 http://www.dell.com/content/products/category.aspx/optix?c=us&cs=555&l=en&s=biz
   - HP Compaq: DC7800 http://h10010.www1.hp.com/wwpc/us/en/en/WF04a/12454-12454-64287-321860-3328898.html
   - Fujitsu-Siemens: Esprimo 5925 http://facts.fujitsu-siemens.com/index.cfm?fuseaction=dspcontent&lid=134&page=3

Caveat on Conventional PCI Device Passthrough

VT-d spec specifies that all conventional PCI devices behind a PCIe-to-PCI bridge have to be assigned to the same domain.

PCIe devices do not have this restriction

VTd device hotplug

2 virtual PCI slots (6~7) are reserved in HVM guest to support VTd hotplug. If you have more VTd devices, only 2 of them can support hotplug. Usage is simple:

1. List the VTd device by dom. You can see a VTd device 0:2:0.0 is inserted in the HVM domain's PCI slot 6. lspci inside the guest should see the same.

[root@vt-vtd ~]# xm pci-list HVMDomainVtd
VSlt domain   bus   slot   func
0x6    0x0  0x02   0x00    0x0

1. detach the device from the guest by the physical BDF. Then HVM guest will receive a virtual PCI hot removal event to detach the physical device

[root@vt-vtd ~]# xm pci-detach HVMDomainVtd 0:2:0.0

1. attach a PCI device to the guest by the physical BDF and desired virtual slot(optional). Following command would insert the physical device into guest's virtual slot 7

[root@vt-vtd ~]# xm pci-attach HVMDomainVtd 0:2:0.0 7

VTd hotplug usage model

• for live migration: As you know, VTd device would break the live migration as physical device can't be save/restored like virtual device. With hotplug, live migration is back again. Just hot remove all the VTd devices before live migration and hot add new VTd devices on target machine after live migration.
• VTd hotplug for device switch: VTd hotplug can be used to dynamically switch physical device between different HVM guest without shutdown.

How to enable MSI/MSI-X for assigned devices?

Before changeset 18127: 89d05940cc1c: Add an option "msi_irq_enable=1" in grub. It should appear like "kernel xen.gz msi_irq_enable=1 <your other options>".

From changeset 18127: 89d05940cc1c to changeset 18453: 7f1c71c6d4c8: (Changeset 18127 changed the parametar to "msi".) Add an option "msi=1" in grub. It should appear like "kernel xen.gz msi=1 <your other options>".

From changeset 18454: 65dc37be0443 on, we always have msi=1. We can't use a Xen grub parameter to set the value to 0.

How to list the currently assignable devices in a host?

From changeset: 18047:39c2cab9e765 on (Later some improvements were made, so we'd betetr use a changeset >=18149:13690b68fd46), there is a shell utility "xm pci-list-assignable-devices". The utility finds all the devices owned by pciback, checks if they have proper FLR method, checks if they have page-alinged MMIO BARs, and checks if the devices have been already assigned, finally it prints out the assignable devices. Note: the utility assumes the integrated devices (whose bus numbers are 0) have proper FLR method.