On Xen and KVM: PCI Virtualisation Terminology:

[Peter Teoh]

http://www.vergenet.net/~horms/pleb_blossom/archives/

Me on Xen and KVM

Computerworld are running an interview with me on some opinions about Xen and KVM. more...

Fri, 23 Jan 2009 16:42:39 +1100

---

PCI Virtualisation Terminology

My last post was titled "Building qemu-dm for HVM VT-d PCI Pass-Through", a piece about building a component of Xen. But what does all this terminology mean?

Firstly, a very brief overview of the Xen architecture. The three main components of a system running Xen are the hypervisor, privileged domains and unprivileged domains. The hypervisor is run when the system first boots and controls access to the hardware. Privileged domains essentially provide both drivers and a management interface to xen. Usually there is one privileged domain, called dom0. They hypervisor by itself is not a functional system and generally the hypervisor + dom0 is required to boot the system. Unprivileged domains are the guest OSes on the system, often referred to as domU. As the name suggests they generally have to access hardware via the hypervisor - which may delegate the task to a privileged domain.

Xen supports two methods of domain (guest) virtualisation. In the beginning there were paravirtualised (PV) domains. In paravirtualised domains portions of the guest OS that execute privileged instructions are replaced by hypercalls. The hypercall asks the hypervisor to perform the relevant action. At the time that Xen was originally developed this was the only type of virtualisation that was possible on x86 and ia64.

With the advent of Intel's VT-x for x86 and VT-i for ia64, and AMD's AMD-V it has became possible to run unmodified operating systems as guests. This is called full-virtualisation or hardare virtual machine (HVM). It has obvious advantages when the guest operating sytem can't be modified.

There are several techniques used to give unprivileged domains access to PCI devices. For PV domains the common approach is to split the driver into front-end and back-end portions. The front-end driver sits in the unprivileged domain and the back-end driver sits in a privileged domain. The front-end driver communicates with the back-end driver. And the back-end driver communicates with the hardware. The back-end driver may multiplex IO from multiple front-end drivers. And thus a PCI device may be shared between domains.

Another approach used with PV domains is PCI pass-through. This basically involves giving a PCI device exclusively to an unprivileged domain. The advantage of this is a potential performance increase, as I/O no longer has to go through a back-end driver sitting in a privileged domain. The disadvantages of this approach are:

1. The PCI device in question can no longer be shared between domains
2. Hotplug needs to be used to achieve live migration - not all devices are supported
3. As the owner of a PCI card can potentially DMA main memory, passing through a PCI device to an unprivileged domain effectively makes the domain privileged

Systems that have Intel VT-d have IOMMU capabilities. The IOMMU constrains the area to with a device passed-through to a guest may DMA. Which removes the security problem relating to passing-through PCI devices to unprivileged guests. Xen allows pass-through to paravirtualised to optionally use the IOMMU if present.

For fully virtualised domains devices are commonly emulated. This is done by running the device emulation code from qemu as qemu-dm in a privileged domain. In order to improve I/O performance in fully virtualised domains it is possible to use PV drivers. These essentially provide front-end drivers to fully virtualised domains. And with the advent of Intel's VT-d it is possible to pass-through PCI devices to fully virtualised domains.

Fri, 12 Dec 2008 17:35:11 +1100

---

Building qemu-dm for HVM VT-d PCI Pass-Through

If you are building the qemu-dm, device models for xen, from http://xenbits.xensource.com/git-http/qemu-xen-unstable.git or the tools/ioemu-dir/ directory that is created when building http://xenbits.xensource.com/xen-unstable.hg (due to the wonders of the xen build system, they are the same thing). And you are doing this on Debian. And you would like PCI pass-through to work for HVM domains. Then make sure you have the libpci-dev package installed before building. Otherwise qemu-dm will be built without pass-through support. If you install libpci-dev after building then run make clean before rebuilding, to ensure that all files are rebuilt correctly - they won't be otherwise.

If you do this then the instructions in the Xen VT-d How-To should work.

Otherwise you will end up the following nasty message, which just cost me more time than I care to admit, in /var/log/xen/xend.log

ERROR (XendDomainInfo:581) Device model didn't tell the vslots for PCI device

On a semi-related note, if you wish to use VT-d PCI pass-through in paravirtualised domains then pass iommu=pv as an argument to xen when booting. Otherwise non-VT-d pass-through will be used for paravirtualised domains. This is independent whether qemu-dm was compiled with libpci-dev present or not.

Fri, 12 Dec 2008 16:58:52 +1100