Performance penalty for Block Window in virtual machines

[Stefan Hajnoczi]

Hi,
Block window is not virtualization-friendly because direct access from the virtual machine to physical NVDIMM BW registers would allow the virtual machine to access any DPA on the physical NVDIMM.  Therefore, BW register accesses must be to trapped (vmexit) and this is costly.

I haven't thought through the extra steps required for interleaving, but I think the basic solution to this problem is:
1. Each BW register set on the NVDIMM can be programmed with a DPA Base Offset and Size Limit.  The hardware checks that command register DPA + LBA Size < Size Limit, and then calculates the physical DPA by adding the DPA Base Offset to the DPA.
2. Apertures and BW register sets are located in memory in a way that allows mapping into a virtual machine's memory space (e.g. each register set needs to be on a separate page so it can be directly accessed by a virtual machine without also giving access to other register sets).

With this in place, the VMM chooses which register sets and apertures to assign to a virtual machine.  Before running the virtual machine, it programs the DPA Base Offset and Size Limits for those register sets.  Now the virtual machine can directly access the registers without breaking isolation.  It can only access addresses in the range [DPA Base Offset, Size Limit) on the physical NVDIMM because the DPAs written to the registers will be checked and added to the offset and limit.

Thoughts?

Stefan

[Andy Rudoff]

Hi Stefan,

Block window is not virtualization-friendly because direct access from the virtual machine to physical NVDIMM BW registers would allow the virtual machine to access any DPA on the physical NVDIMM.  Therefore, BW register accesses must be to trapped (vmexit) and this is costly.

Agreed.  At least, I agree that allowing direct access to BWs when a guest doesn't own the entire physical DIMM won't work without vmexits (more below).

 

I haven't thought through the extra steps required for interleaving, but I think the basic solution to this problem is:
1. Each BW register set on the NVDIMM can be programmed with a DPA Base Offset and Size Limit.  The hardware checks that command register DPA + LBA Size < Size Limit, and then calculates the physical DPA by adding the DPA Base Offset to the DPA.

Interleaving does break this, even with your offset & size addition, because interleaving turns the BW into a discontiguous range of addresses with BWs from other NVDIMMs interleaved in.  But assuming interleaving is turned off for the moment, this still isn't quite right unless we decide to limit namespaces to always consist of a single contiguous range of DPAs.  Right now, the NVDIMM Namespace Spec allows block namespaces to consist of multiple regions that are concatenated.  This is to simplify the admin model when namespaces are deleted and added over time -- the labels handle a certain amount of fragmentation without requiring existing namespaces to be relocated.

 

2. Apertures and BW register sets are located in memory in a way that allows mapping into a virtual machine's memory space (e.g. each register set needs to be on a separate page so it can be directly accessed by a virtual machine without also giving access to other register sets).

Without interleaving, this would be easy.

 

With this in place, the VMM chooses which register sets and apertures to assign to a virtual machine.  Before running the virtual machine, it programs the DPA Base Offset and Size Limits for those register sets.  Now the virtual machine can directly access the registers without breaking isolation.  It can only access addresses in the range [DPA Base Offset, Size Limit) on the physical NVDIMM because the DPAs written to the registers will be checked and added to the offset and limit.

Thoughts?

The example BW interface is designed so that the VMM can export virtual BWs, allowing the guest VM to run the same driver as on bare metal.  These block windows would actually just be areas of RAM, requiring the guest to read/write from the RAM areas and the VMM to copy the data a second time using actual BWs.  While that is less optimal that the direct BW usage you were talking about, we defined the BWs in a way that requires only a single VM-exit per I/O.  But even then, I think most VMMs would just punt on using BWs in the guest and expose NVM for block usage the same way any other block storage is exposed to the guest.

I'm not trying to discourage you from coming up with a better BW example than ours, one that includes the range protection you mention.  I'm just saying to when we were thinking it through it got more and more complex to allow a guest to use BWs directly and the value didn't seem there when we looked at the big picture...

[Stefan Hajnoczi]

On Mon, May 4, 2015 at 10:40 PM, Andy Rudoff <an...@rudoff.com> wrote:
> But even then, I think most VMMs would just punt on using
> BWs in the guest and expose NVM for block usage the same way any other block
> storage is exposed to the guest.

Functionally that's fine: it solves the address space consumption and
error reporting problems just like BW does. I suspect the overhead of
block I/O is higher than the cost of the BW operation itself though,
so it makes NVDIMMs slow for VMs.

> I'm not trying to discourage you from coming up with a better BW example
> than ours, one that includes the range protection you mention. I'm just
> saying to when we were thinking it through it got more and more complex to
> allow a guest to use BWs directly and the value didn't seem there when we
> looked at the big picture...

Thanks for explaining the complications that I hadn't understood yet
with interleaving and non-contiguous namespaces.

It won't be possible to solve this in a simple fashion. The other
challenge is that hardware provides a finite number of BW register
sets/apertures, so there is a limit in the number of VMs that can do
pass-through. At some point the VMM needs to fall back to traditional
block storage emulation anyway.

Stefan