Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

Hello,

I am trying to run the Jailhouse gic-demo on the Xilinx ZCU102 dev board. However, when I start the demo, I got an unhandled trap. Here are my steps:

1) modprobe jailhouse

syslog output:

Dec 11 18:39:54 linaro-gnome kernel: [ 25.666781] jailhouse: loading out-of-tree module taints kernel.

2) jailhouse enable jailhouse/configs/zynqmp-zcu102.cell

Initializing Jailhouse hypervisor v0.7 (157-g2149299-dirty) on CPU 1
Code location: 0x0000ffffc0200060
Page pool usage after early setup: mem 33/979, remap 64/131072
Initializing processors:
CPU 1... OK
CPU 0... OK
CPU 2... OK
CPU 3... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 42/979, remap 69/131072
Activating hypervisor
[ 54.655832] OF: ERROR: Bad of_node_put() on /amba/xilinx_drm
[ 54.662352] xilinx-drm-dp fd4a0000.dp: failed to get phy lane
[ 54.668376] ahci-ceva fd0c0000.ahci: couldn't get PHY in node ahci: -517

3) jailhouse cell create jailhouse/configs/zynqmp-zcu102-gic-demo.cell
Created cell "gic-demo"
Page pool usage after cell creation: mem 54/979, remap 69/131072

Syslog output:

Dec 11 18:46:55 linaro-gnome kernel: [ 446.433244] CPU3: shutdown
Dec 11 18:46:55 linaro-gnome kernel: [ 446.433251] psci: CPU3 killed.
Dec 11 18:46:55 linaro-gnome kernel: [ 446.473758] Created Jailhouse cell "gic-demo

4) jailhouse cell load gic-demo jailhouse/inmates/demos/arm64/gic-demo.bin

Output: Cell "gic-demo" can be loaded

5) jailhouse cell start gic-demo

The output below is repeated forever:

Parking CPU 3 (Cell: "gic-demo")
Unhandled data write at 0x90003fe0(1)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
pc: 0000000000001214 lr: 0000000000000000 spsr: 00000005 EL1
sp: 0000000090004000 esr: 24 1 0000045
x0: 0000000000300000 x1: 0000000000000000 x2: 0000000000000000
x3: 0000000000000000 x4: 0000000000000000 x5: 0000000000000000
x6: 0000000000000000 x7: 0000000000000000 x8: 0000000000000000
x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000000
x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000
x18: 0000000000000000 x19: 0000000000000000 x20: 0000000000000000
x21: 0000000000000000 x22: 0000000000000000 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000 x26: 0000000000000000
x27: 0000000000000000 x28: 0000000000000000 x29: 0000000000000000

It seems the generated interrupt is not being correctly handled. The question is why? Any ideas?

The kernel I am using is Linux linaro-gnome 4.9.0 and Jailhouse v0.7.

Best
Giovani

[Giovani Gracioli]

Quick correction: the current the Linux kernel is the Xilinx one, 4.9.0.

[Jan Kiszka]

Start with mainline + the Jailhouse queue before trying downstream. If
that works, you can start playing with vendor kernels (if there is a
real need for that).

Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[Giovani Gracioli]

Thanks Jan.

Recompiled with kernel 4.15.4 (the last stable from mainline) and the error disappeared.

However, I am not able to see the gic-demo output. I am connected in the terminal /dev/ttyUSB0 (ttyPS0). There are four ttyUSB[0-3] mounted.

Where should gic-demo be printing to?

Giovani

[Jan Kiszka]

On the one that is behind the address 0xff010000 - IIRC, the second UART
line.

[Giovani Gracioli]

After I rebooted, the error is back.

[Jan Kiszka]

On 2017-12-12 19:18, Giovani Gracioli wrote:
> After I rebooted, the error is back.
>

Then check what is behind 0x90003fe0 (/proc/iomem) and possibly add some
permission to the root cell config. It might be a device we didn't use,
thus didn't cover in the upstream config.

Jan

[Giovani Gracioli]

0x90003fe0 is not listed (mapped?).

$ cat /proc/iomem

00000000-7fffffff : System RAM
00080000-00d7ffff : Kernel code
00e80000-00ffcfff : Kernel data
fc000000-fc0fffff : PCI ECAM
fc100000-fc103fff : //vpci@0
fc100000-fc1000ff : 0000:00:00.0
fc100100-fc1001ff : 0000:00:01.0
fd500000-fd500fff : /amba/dma@fd500000
fd510000-fd510fff : /amba/dma@fd510000
fd520000-fd520fff : /amba/dma@fd520000
fd530000-fd530fff : /amba/dma@fd530000
fd540000-fd540fff : /amba/dma@fd540000
fd550000-fd550fff : /amba/dma@fd550000
fd560000-fd560fff : /amba/dma@fd560000
fd570000-fd570fff : /amba/dma@fd570000
fd6e9000-fd6edfff : /amba/cci@fd6e0000/pmu@9000
ff000000-ff000fff : xuartps
ff010000-ff010fff : xuartps
ff020000-ff020fff : /amba/i2c@ff020000
ff030000-ff030fff : /amba/i2c@ff030000
ff070000-ff070fff : /amba/can@ff070000
ff0a0000-ff0a0fff : /amba/gpio@ff0a0000
ff0e0000-ff0e0fff : /amba/ethernet@ff0e0000
ff0f0000-ff0f0fff : /amba/spi@ff0f0000
ff170000-ff170fff : /amba/sdhci@ff170000
ffa60000-ffa600ff : /amba/rtc@ffa60000
800000000-8003fffff : Jailhouse hypervisor
840000000-877ffffff : System RAM

[Jan Kiszka]

On 2017-12-12 21:04, Giovani Gracioli wrote:
> 0x90003fe0 is not listed (mapped?).
>

OK, I should have read the initial email more carefully: This is an
access of the non-root cell. I'll comment on that mail, something is
broken with the inmate demo...

Jan

[Jan Kiszka]

If we are looking at the same code and disassembly (I just rebuild mine
from next), this address is the first line in inmate_main and stores
something on the stack.

> sp: 0000000090004000 esr: 24 1 0000045

But the stack pointer is totally off. It should be 0x4000, see the code,
and it should be loaded from a constant variable (stack_top) by
__reset_entry that is located at 0xf90 in my code and correctly contains
0x4000 in my binary.

Please check your compilation output (objdump -d), maybe also try a
different toolchain if that is broken.

Here are the relevant parts as they should look like:

0000000000000000 <__reset_entry>:
0: 58007c40 ldr x0, f88 <vectors+0x788>
4: d518c000 msr vbar_el1, x0
8: 58007c40 ldr x0, f90 <vectors+0x790>
c: 9100001f mov sp, x0
10: d2a00600 mov x0, #0x300000
14: d5181040 msr cpacr_el1, x0
18: d51b423f msr daif, xzr
1c: d5033fdf isb
20: 1400047d b 1214 <inmate_main>
[...]
f80: 14000000 .word 0x14000000
f84: 00000000 .word 0x00000000
f88: 00000800 .word 0x00000800
f8c: 00000000 .word 0x00000000
f90: 00004000 .word 0x00004000
f94: 00000000 .word 0x00000000
[...]
0000000000001214 <inmate_main>:
1214: a9be53f3 stp x19, x20, [sp,#-32]!

Jan

> x0: 0000000000300000 x1: 0000000000000000 x2: 0000000000000000
> x3: 0000000000000000 x4: 0000000000000000 x5: 0000000000000000
> x6: 0000000000000000 x7: 0000000000000000 x8: 0000000000000000
> x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
> x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000000
> x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000
> x18: 0000000000000000 x19: 0000000000000000 x20: 0000000000000000
> x21: 0000000000000000 x22: 0000000000000000 x23: 0000000000000000
> x24: 0000000000000000 x25: 0000000000000000 x26: 0000000000000000
> x27: 0000000000000000 x28: 0000000000000000 x29: 0000000000000000
>
> It seems the generated interrupt is not being correctly handled. The question is why? Any ideas?
>
> The kernel I am using is Linux linaro-gnome 4.9.0 and Jailhouse v0.7.
>
> Best
> Giovani
>

[Giovani Gracioli]

The __reset_entry is in the wrong memory address:

0000000090000000 <__reset_entry>:
90000000: 58007c40 ldr x0, 90000f88 <vectors+0x788>
90000004: d518c000 msr vbar_el1, x0
90000008: 58007c40 ldr x0, 90000f90 <vectors+0x790>
9000000c: 9100001f mov sp, x0
90000010: d2a00600 mov x0, #0x300000 // #3145728
90000014: d5181040 msr cpacr_el1, x0
90000018: d51b423f msr daif, xzr
9000001c: d5033fdf isb
90000020: 14000470 b 900011e0 <inmate_main>
....
90000f88: 90000800 .word 0x90000800
90000f8c: 00000000 .word 0x00000000
90000f90: 90004000 .word 0x90004000
90000f94: 00000000 .word 0x00000000
....

00000000900011e0 <inmate_main>:
900011e0: a9be53f3 stp x19, x20, [sp,#-32]!
900011e4: b0000000 adrp x0, 90002000 <gic_v2_enable>
900011e8: 91038800 add x0, x0, #0xe2
900011ec: 90000013 adrp x19, 90001000 <cmdline>
900011f0: f9000bfe str x30, [sp,#16]
900011f4: 940000c0 bl 900014f4 <printk>
900011f8: 90000000 adrp x0, 90001000 <cmdline>
900011fc: 9104d000 add x0, x0, #0x134
90001200: 9400002a bl 900012a8 <gic_setup>

What is your toolchain version?

Mine is:

aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Linaro GCC Snapshot 6.2-2016.11) 6.2.1 20161114
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[Giovani Gracioli]

Found the error. I removed this line from the config.h:

#define CONFIG_INMATE_BASE 0x90000000

The entry address is correct now:

0000000000000000 <__reset_entry>:
0: 58007c40 ldr x0, f88 <vectors+0x788>
4: d518c000 msr vbar_el1, x0
8: 58007c40 ldr x0, f90 <vectors+0x790>
c: 9100001f mov sp, x0

10: d2a00600 mov x0, #0x300000 // #3145728

14: d5181040 msr cpacr_el1, x0
18: d51b423f msr daif, xzr
1c: d5033fdf isb
20: 1400047d b 1214 <inmate_main>

Thanks!

[Jan Kiszka]

On 2017-12-13 16:35, Giovani Gracioli wrote:
> Found the error. I removed this line from the config.h:
>
> #define CONFIG_INMATE_BASE 0x90000000
>

Yeah, we may need a clarifying comment in
Documentation/hypervisor-configuration.md that this line is an example
for moving the base and then implies loading the inmate at this address
as well.

Jan