Inter-cell communication: Notes on the usability of PCI EA

[Jan Kiszka]

Hi all,

while trying to dig deeper into the topic of how our inter-cell
communication infrastructure, I started with something that looked like
a low-hanging fruit. As it turned out to not work that way, I would like
to record my findings and thoughts this way for future reference:

Our current inter-cell communication mechanism is built on top of an
ivshmem[1]-derived device. One particular difference to QEMU's ivshmem
version is that we do not expose the shared memory region that this
device provides via a PCI BAR. The reason is that Jailhouse does not
want to allow the guest to remap that region and rather expose it at a
fixed guest-physical address. PCI BARs support remapping thus can't be
used. Instead, we report address and size of the shared memory via
custom fields in the PCI config space of our ivshmem devices.

Recently, PCI gained an interesting extension that is called "Enhanced
Allocation" [2]. It allows (among other things) to specify
hardware-allocated unmoveable resources via a PCI capability, replacing
regular BARs. Linux understands this extension now, so it seemed only
logical to use it instead of our special config space registers.

However, there are some catches that prevent this elegant solution:

- Major blocker: Linux expect all EA resources of a device to have
parent resources that contain them. But the ivshmem device's parent
resources, those of the host bridge, do not contain the RAM we want
to expose. Therefore, Linux refuses to enable the device because no
one "claims" the EA resource.

- One could try to tweak the host bridge to be aware of those additional
memory region. But there are no proper interface in Linux to support
such a scenario, and we would end up in generating fragile hacks.

- Another conceptual problem with EA vs. our ivshmem devices: devices
that use EA must always be present, never be removed or hot-added.
But that is exactly what we do with the ivshmem devices when Jailhouse
is enabled or disabled: the root cell will see them appear and
disappear again. So, used EA here is actually against the
specification, thus bears the risk of future breakages as Linux is
reworking its PCI handling for whatever reasons.

Long story short: PCI's EA would have been nice for describing the
unmoveable shared memory regions of our ivshmem (PCI) devices, but it is
simply not made for this special use case. We will have to stick with a
custom information channel to report these resources.

Jan

[1] Offical ivshmem spec:
http://git.qemu.org/?p=qemu.git;a=blob;f=docs/specs/ivshmem_device_spec.txt
[2]
https://pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf