Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
jasperk...@gmail.com
I'm trying to set up a system with a bare-metal guest/inmate, with shared memory between Linux root cell and inmate, and an (IPI) doorbell interrupt from inmate to Linux.
My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified to only have only one IVSHMEM device instead of two.
Many things are working; gic-demo demo application runs correctly and from Linux I can access the shared memory via an uio device. However I'm stuck at generating the doorbell interrupt from inmate to Linux.
# lsuio
uio0: name=uio_ivshmem, version=0.0.1, events=0
map[0]: addr=0x00000000FC100000, size=4096
map[1]: addr=0x0000000800400000, size=1048576
#cat /proc/interrupts
...
229: 0 0 0 GICv2 136 Edge uio_ivshmem
...
I modified the gic-demo application slightly to generate a doorbell in the timer interrupt handler method. I attempt the generate the doorbell by writing the doorbell register in the ivshmem register region (map0). (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt)
unsigned long* reg = (unsigned long*)0xFC100000;
*(reg + 3) = 1; // doorbell at offset 12byte
On Linux side the doorbell is seen (events=1) only as soon as the inmate is destroyed, see log below. For the inmate, the gic-demo timer interrupt is only handled once, likely because there is another interrupt pending.
# ./jailhouse cell start inmate0
Started cell "inmate0"
# lsuio
uio0: name=uio_ivshmem, version=0.0.1, events=0
map[0]: addr=0x00000000FC100000, size=4096
map[1]: addr=0x0000000800400000, size=1048576
# ./jailhouse cell shutdown inmate0
Cell "inmate0" can be loaded
# lsuio
uio0: name=uio_ivshmem, version=0.0.1, events=0
map[0]: addr=0x00000000FC100000, size=4096
map[1]: addr=0x0000000800400000, size=1048576
# ./jailhouse cell destroy inmate0
Closing cell "inmate0"
Page pool usage after cell destruction: mem 43/996, remap 69/131072
[ 222.899818] ivshmem_handler irq: 229
[ 222.920548] Detected VIPT I-cache on CPU3
[ 222.920599] CPU3: Booted secondary processor [410fd034]
[ 222.934074] Destroyed Jailhouse cell "inmate0"
# lsuio
uio0: name=uio_ivshmem, version=0.0.1, events=1
map[0]: addr=0x00000000FC100000, size=4096
map[1]: addr=0x0000000800400000, size=1048576
Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info *dev_info)", which seems to be called only when the cpu is returned to Linux.
It seems the doorbell is generated for the core I perform the register write on. How do I generate doorbells for other cores? Am I using a correct method to generate doorbells in the first place? Are there other options to do so?
Kind regards, Jasper
jasperk...@gmail.com
I want to add that since my original post I have changed flags for the IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION.
/* IVSHMEM register region for 00:00.0 */ {
.phys_start = 0xfc100000,
.virt_start = 0xfc100000,
.size = 4096,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_COMM_REGION,
},
The result is that the inmate no longer hangs when attempting to access the register space (I discovered that even reading IVPosition resulted in hang).
However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in line below:
mmio_write32(reg + (i*8) + 3, 1);
jasperk...@gmail.com
Some additional comments..
Fixed my mistake in inmate0.c: "unsigned int" instead of "unsigned long" to access correct offsets via pointer aritmetics.. sorry.
But that still does not give working doorbells.
Following: https://github.com/siemens/jailhouse/blob/master/Documentation/inter-cell-communication.txt I also attempt to mmio_write the LSTATE register but no success.
Jan Kiszka
>
> /* IVSHMEM register region for 00:00.0 */ {
> .phys_start = 0xfc100000,
> .virt_start = 0xfc100000,
> .size = 4096,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_COMM_REGION,
> },
>
> The result is that the inmate no longer hangs when attempting to access the register space (I discovered that even reading IVPosition resulted in hang).
>
> However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in line below:
>
> mmio_write32(reg + (i*8) + 3, 1);
>
Please check first if you get ivshmem-net communication between root and
non-root cell using the original setup. That is known to work. Then
stepwise modify settings and test setups to your needs.
The (a bit unfortunate) alternative is instrumenting the hypervisor /wrt
ivshmem interactions to find out what happens from its POV. The code
isn't very complex.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
jasperk...@gmail.com
Hi Jan,
I managed to generate doorbells succesfully and they are picked up by the root cell.
I still use:
unsigned int* reg = (unsigned int*)0xfc100000; //address of /dev/uio0 map[0] in root cell
mmio_write32(reg + 3, 1);
However, I completely removed the memory region from the inmate cell config and instead added code below to the end of ivshmem_init (hypervisor/ivshmem.c)
if (cell->config->id > 0) { // inmate only
mmio_region_register(device->cell, 0xfc100000, IVSHMEM_BAR0_SIZE, ivshmem_register_mmio, ive);
}
This traps the mmio_write above, eventually generating the interrupt via ivshmem_register_mmio -> ... -> irqchip_set_pending. So it seems functional except for that the region is not registered. device->bar[0] and device->bar[4] are not properly initialized.
Am I doing something wrong which prevents the automatic registration? Current solution works but it's obviously not ideal...
Jan Kiszka
apparently works around a configuration issue in your guest configs or
in the inmate (did you properly map the MMIO region that the ivshmem
device exposes INSIDE the guest?).
That is the reason I asked you to first set up the inter-cell
communication between two Linux guests via the unmodified configs from
Jailhouse. Try to dig your way from there - and do not change the
hypervisor... ;)