questions related to IVSHMEM/uio_ivshmem

[Constantin Petra]

Hi,

I have a question related to this above driver implementation, in an Ultrascale+ environment.

This was taken from https://github.com/henning-schild/ivshmem-guest-code/blob/master/kernel_module/uio/uio_ivshmem.c, in an attempt to set up IVSHMEM connection (not ivshmem-net) with a different inmate OS (not Linux).

here, in ivshmem_pci_probe():
info->mem[1].addr = pci_resource_start(dev, 2);

the returned "addr" is zero, causing the probe to return -ENODEV when ivshmem_pci_probe() is called by the OS.

It's hard for me to follow the logic without a through pci driver knownledge, do you have any idea what could cause this or would you mind sharing some tips for debugging this?

Thank you,
Constantin

[Henning Schild]

Am Fri, 1 Sep 2017 05:36:31 -0700
schrieb Constantin Petra <constant...@gmail.com>:

> Hi,
>
> I have a question related to this above driver implementation, in an
> Ultrascale+ environment.
>
> This was taken from
> https://github.com/henning-schild/ivshmem-guest-code/blob/master/kernel_module/uio/uio_ivshmem.c,
> in an attempt to set up IVSHMEM connection (not ivshmem-net) with a
> different inmate OS (not Linux).
>
> here, in ivshmem_pci_probe():
> info->mem[1].addr = pci_resource_start(dev, 2);

You will have to check out the "jailhouse" branch from the above github
repository. In jailhouse the location and size of the shared memory are
not in the BAR but somewhere in custom config space.

The next thing you will run into might be problems with legacy
interrupts. A patch for that is in "jailhouse-next" but i never tested
it myself or received any feedback on it. If you can confirm that it
works i will merge it into the jailhouse-branch.
@Jonas, can you confirm that the patch in jailhouse-next made the
interrupts work, or did you have to change ivshmem-guest-code further?

Henning

[Constantin Petra]

Hi Henning,

Ok, I missed that. The "jailhouse" branch works, at least I see the expected messages:

[   46.200693] uio_ivshmem 0001:00:03.0: using jailhouse mode

[   46.206260] uio_ivshmem 0001:00:03.0: regular IRQs

and 

/dev/uio0 & /dev/uio1 up

Related to the patch on jailhouse-next, is that the commit 87fbf1f / May 17th

"?

Best Regards,

Constantin

[Henning Schild]

Am Fri, 1 Sep 2017 16:40:09 +0300
schrieb Constantin Petra <constant...@gmail.com>:

> Hi Henning,
>
> Ok, I missed that. The "jailhouse" branch works, at least I see the
> expected messages:
>
> [ 46.200693] uio_ivshmem 0001:00:03.0: using jailhouse mode
> [ 46.206260] uio_ivshmem 0001:00:03.0: regular IRQs
> and
> /dev/uio0 & /dev/uio1 up
>
> Related to the patch on jailhouse-next, is that the commit 87fbf1f /
> May 17th
> "?

Yes. It probably works and is required. If you can confirm that some
day, i will include it into "jailhouse".

Henning

[Constantin Petra]

Hi,

I can now confirm that the patch 87fbf1f works and interrupts are received correctly, /dev/uiox is accessible and works as expected.
Without it, I got a:
"FATAL: forbidden access (exception class 0x24)" but I don't remember if this was triggered by Jailhouse or Linux.

Best Regards,
Constantin

[Henning Schild]

Am Tue, 5 Sep 2017 00:41:52 -0700
schrieb Constantin Petra <constant...@gmail.com>:

Thanks, applied to "jailhouse" and removed "jailhouse-next".

Henning

> Best Regards,
> Constantin
>

[Luca Cuomo]

> > Hi,
> >
> > I can now confirm that the patch 87fbf1f works and interrupts are
> > received correctly, /dev/uiox is accessible and works as expected.
> > Without it, I got a: "FATAL: forbidden access (exception class 0x24)"
> > but I don't remember if this was triggered by Jailhouse or Linux.
>
> Thanks, applied to "jailhouse" and removed "jailhouse-next".
>
> Henning
>
> > Best Regards,
> > Constantin
> >

Hi all,

i've got a strange situation. I'm using the latest jailhouse branch on x86 with a linux root cell and a Bare Metal cell connected via 2 pci device (bdf e.00 and f.00).

Two Ivshmem device are correctly mapped to /dev/uio0 and /dev/uio1.
But if on /dev/uio0 i can correctly map registers ( offset 0, size 0x1000) and shmem (offset 0x1000, size 0x1000), for /dev/uio1 the first mapping fails with the ENODEV errno set (shmem mapping is done correctly). Parameters of mapping are the same for uio0 and uio1.

Here i list a dmesg snippet of the root cell kernel:

[ 302.092280] pci 0000:00:0e.0: [1af4:1110] type 00 class 0xff0000
[ 302.092622] pci 0000:00:0e.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit]
[ 302.092776] hpet_rtc_timer_reinit: 39 callbacks suppressed
[ 302.092777] hpet1: lost 62 rtc interrupts
[ 302.093025] pci 0000:00:0e.0: reg 0x20: [mem 0x00000000-0x0000001f 64bit]
[ 302.093634] pci 0000:00:0e.0: BAR 0: assigned [mem 0xc0000000-0xc00000ff 64bit]
[ 302.093765] pci 0000:00:0e.0: BAR 4: assigned [mem 0xc0000100-0xc000011f 64bit]
[ 302.093982] virtio-pci 0000:00:0e.0: enabling device (0000 -> 0002)
[ 302.094169] uio_ivshmem 0000:00:0e.0: using jailhouse mode
[ 302.094625] uio_ivshmem 0000:00:0e.0: MSI-X enabled
[ 302.094918] pci 0000:00:0f.0: [1af4:1110] type 00 class 0xff0000
[ 302.095259] pci 0000:00:0f.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit]
[ 302.095725] pci 0000:00:0f.0: reg 0x20: [mem 0x00000000-0x0000001f 64bit]
[ 302.096468] pci 0000:00:0f.0: BAR 0: assigned [mem 0xc0000200-0xc00002ff 64bit]
[ 302.096667] pci 0000:00:0f.0: BAR 4: assigned [mem 0xc0000120-0xc000013f 64bit]
[ 302.096962] virtio-pci 0000:00:0f.0: enabling device (0000 -> 0002)
[ 302.097139] uio_ivshmem 0000:00:0f.0: using jailhouse mode
[ 302.097568] uio_ivshmem 0000:00:0f.0: MSI-X enabled

What i'm doing wrong? I do exactly the same things on /dev/uio0 and /dev/uio1

Thanks,

--Luca

[Henning Schild]

Am Mon, 18 Dec 2017 05:58:11 -0800
schrieb Luca Cuomo <l.c...@evidence.eu.com>:

I do not remember whether i tested the uio-driver with multiple
devices, i can not rule out a hidden bug in there. The output looks
pretty much the same for both instances and does not seem too useful.

What would be interesting is which call fails and where it fails in the
kernel? Could you "unbind" the two devices from the driver and "bind"
them the other way around, does that make the "second" work and the
"first" fail?
What test-code are you using, if anyone wanted to reproduce the setup
for further assistance.

Henning

[Luca Cuomo]

Even the /sys/class/uio/uiox ... directories seem to be ok.

> What would be interesting is which call fails and where it fails in the
> kernel? Could you "unbind" the two devices from the driver and "bind"
> them the other way around, does that make the "second" work and the
> "first" fail?

This could be a good idea but how do you suggest to unbind them?

> What test-code are you using, if anyone wanted to reproduce the setup
> for further assistance.

The same result using both the uio_send test which is in the repo and a self made test which basically does:

int firstFd = open(/dev/uio0, O_RDWR) //ok
mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, firstFd, 0) //regs ok
mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, firstFd, 4096) //shmem ok

int secondFd = open(/dev/uio1, O_RDWR) //ok
mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, secondFd, 0) //regs FAIL
mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, secondFd, 4096) //shmem ok

The way i can overcome the problem for the second device is reading the address from /sys/class/uio/uio1/maps/map0/addr and then mmapping from /dev/mem with the proper offset (/sys/class/uio/uio1/maps/map0/offset)

[Henning Schild]

Am Mon, 18 Dec 2017 07:41:34 -0800

# echo 0000:00:0f.0 > /sys/bus/pci/drivers/uio_ivshmem/unbind
# echo 0000:00:0e.0 > /sys/bus/pci/drivers/uio_ivshmem/unbind

# echo 0000:00:0f.0 > /sys/bus/pci/drivers/uio_ivshmem/bind

Now 0f should be your uio0.

> > What test-code are you using, if anyone wanted to reproduce the
> > setup for further assistance.
>
> The same result using both the uio_send test which is in the repo and
> a self made test which basically does:
>
> int firstFd = open(/dev/uio0, O_RDWR) //ok
> mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, firstFd,
> 0) //regs ok mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED,
> firstFd, 4096) //shmem ok
>
> int secondFd = open(/dev/uio1, O_RDWR) //ok
> mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, secondFd,
> 0) //regs FAIL mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED,
> secondFd, 4096) //shmem ok

Ok, let us see where this is coming from with ftrace.

Could you try something like this, best on an otherwise idle system:

trace-cmd record -p function -l '*mmap*' yourtest
trace-cmd report

Maybe check out the documentation of ftrace to get closer to the root
cause of the error. The mmap itself is not even implemented in the
driver, i am guessing a parameter like size or location might be wrong.
But if you can mmap it from /dev/mem i might also be an alignment issue.
We basically have to trace that mmap-call to the condition in the
kernel returning the error.

> The way i can overcome the problem for the second device is reading
> the address from /sys/class/uio/uio1/maps/map0/addr and then mmapping
> from /dev/mem with the proper offset
> (/sys/class/uio/uio1/maps/map0/offset)

Well from a security point of view, i would not suggest that. Rather
have your uios in a group via udev ...
For now a good workaround but i would like to understand and fix it, if
it is something in the uio-driver.

Henning

[Luca Cuomo]

I've tried and now the problem in on /dev/uio0. So the problem in on the same device (the one that previously was /dev/uio1.

>
> > > What test-code are you using, if anyone wanted to reproduce the
> > > setup for further assistance.
> >
> > The same result using both the uio_send test which is in the repo and
> > a self made test which basically does:
> >
> > int firstFd = open(/dev/uio0, O_RDWR) //ok
> > mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, firstFd,
> > 0) //regs ok mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED,
> > firstFd, 4096) //shmem ok
> >
> > int secondFd = open(/dev/uio1, O_RDWR) //ok
> > mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, secondFd,
> > 0) //regs FAIL mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED,
> > secondFd, 4096) //shmem ok
>
> Ok, let us see where this is coming from with ftrace.
>
> Could you try something like this, best on an otherwise idle system:
>
> trace-cmd record -p function -l '*mmap*' yourtest
> trace-cmd report
>
> Maybe check out the documentation of ftrace to get closer to the root
> cause of the error. The mmap itself is not even implemented in the
> driver, i am guessing a parameter like size or location might be wrong.
> But if you can mmap it from /dev/mem i might also be an alignment issue.
> We basically have to trace that mmap-call to the condition in the
> kernel returning the error.

This will take me some time, so when i'll setup the environment i will post the results.

>
> > The way i can overcome the problem for the second device is reading
> > the address from /sys/class/uio/uio1/maps/map0/addr and then mmapping
> > from /dev/mem with the proper offset
> > (/sys/class/uio/uio1/maps/map0/offset)
>
> Well from a security point of view, i would not suggest that. Rather
> have your uios in a group via udev ...
> For now a good workaround but i would like to understand and fix it, if
> it is something in the uio-driver.

I will try this on arm too.

[Henning Schild]

Am Wed, 20 Dec 2017 23:42:26 -0800

Ok with that i think the driver is not to blame.

> >
> > > > What test-code are you using, if anyone wanted to reproduce the
> > > > setup for further assistance.
> > >
> > > The same result using both the uio_send test which is in the repo
> > > and a self made test which basically does:
> > >
> > > int firstFd = open(/dev/uio0, O_RDWR) //ok
> > > mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, firstFd,
> > > 0) //regs ok mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED,
> > > firstFd, 4096) //shmem ok
> > >
> > > int secondFd = open(/dev/uio1, O_RDWR) //ok
> > > mmap( NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, secondFd,
> > > 0) //regs FAIL mmap( NULL, 4096, PROT_READ | PROT_WRITE,
> > > MAP_SHARED, secondFd, 4096) //shmem ok

Aha, the register ressource is only 256 bytes in size, maybe that is
causing the mapping to fail. No clue why it would work for uio0 but
that could be a problem.

You should be able to discover those values in userspace, the driver
fills them into a struct. Maybe in sysfs or with an ioctl on the device.

Henning

[Henning Schild]

Am Thu, 21 Dec 2017 15:10:09 +0100
schrieb "[ext] Henning Schild" <henning...@siemens.com>:

Did not try but i guess you can find the values here, instead of
hardcoding them:
/sys/class/uio/uio0/maps/

Henning