Jailhouse on Ultrascale+ (ZCU102)

[constant...@gmail.com]

Hi,

Is it possible to run Jailhouse in QEMU for ZCU102 (Using Petalinux 2016.4)?
There is apparently support for configuration and applications, but QEMU just hangs when calling "jailhouse enable jailhouse/configs/zynqmp-zcu102.cell"
In "zynqmp-zcu102.c" it is mentioned:
"Reservation via device tree: 0x800000000..0x83fffffff"
How to reserve this more specifically?
I have tried to reserve the RAM area above 1GB for Jailhouse, but there is no difference in behavior.

Thank you,
Constantin

[Jan Kiszka]

We didn't enable downstream so far (I prefer upstream as far as
possible). See [1] for more details.

Jan

[1]
https://www.mail-archive.com/jailho...@googlegroups.com/msg02239.html

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[Constantin Petra]

Thanks Jan,

I'll check it out!

Constantin

On Tue, Apr 11, 2017 at 4:39 PM, Jan Kiszka <jan.k...@siemens.com> wrote:

On 2017-04-11 15:19, constant...@gmail.com wrote:
> Hi,
>
> Is it possible to run Jailhouse in QEMU for ZCU102 (Using Petalinux 2016.4)?
> There is apparently support for configuration and applications, but QEMU just hangs when calling "jailhouse enable jailhouse/configs/zynqmp-zcu102.cell"
> In "zynqmp-zcu102.c" it is mentioned:
> "Reservation via device tree: 0x800000000..0x83fffffff"
> How to reserve this more specifically?
> I have tried to reserve the RAM area above 1GB for Jailhouse, but there is no difference in behavior.

We didn't enable downstream so far (I prefer upstream as far as
possible). See [1] for more details.

Jan

[1]

https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg02239.html

[constant...@gmail.com]

On Tuesday, April 11, 2017 at 6:35:55 PM UTC+3, Constantin Petra wrote:
> Thanks Jan,
>
>
> I'll check it out!
>
>
> Constantin
>
>
> On Tue, Apr 11, 2017 at 4:39 PM, Jan Kiszka <jan.k...@siemens.com> wrote:
> On 2017-04-11 15:19, constant...@gmail.com wrote:
>
> > Hi,
>
> >
>
> > Is it possible to run Jailhouse in QEMU for ZCU102 (Using Petalinux 2016.4)?
>
> > There is apparently support for configuration and applications, but QEMU just hangs when calling "jailhouse enable jailhouse/configs/zynqmp-zcu102.cell"
>
> > In "zynqmp-zcu102.c" it is mentioned:
>
> > "Reservation via device tree: 0x800000000..0x83fffffff"
>
> > How to reserve this more specifically?
>
> > I have tried to reserve the RAM area above 1GB for Jailhouse, but there is no difference in behavior.
>
>
>
> We didn't enable downstream so far (I prefer upstream as far as
>
> possible). See [1] for more details.
>
>
>
> Jan
>
>
>
> [1]
>

> https://www.mail-archive.com/jailho...@googlegroups.com/msg02239.html

>
>
>
> --
>
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
>
> Corporate Competence Center Embedded Linux

Hi again,

I have used the proposed linux kernel and also changed the system.dts and config. The system goes on a bit further; Any idea why the log shows the "FATAL: instruction abort at 0x87fffeee0"? Is this some configuration issue? QEMU support maybe?
Sorry to bother you with this, but the petalinux system is also new to me (so Im not sure every change was correctly taken into account) and there are too many variables around to be able to spot the issue.

Thanks a lot,
Constantin

[Jan Kiszka]

Something is still broken with the zynqmp-zcu102.cell. I suspect that
this address is right at the instruction in jailhouse.ko where the
hypervisor returns to after enabling: the module should be loaded at
ffffff8000a00000, see /proc/modules (may change on next run), and
objdump -d jailhouse.ko should report that enter_hypervisor is around ee0.

No idea, though, what is inconsistent, but the root cell lacks basic
access to its RAM.

QEMU or KVM must not be the loop. They have nothing to do with running
Jailhouse.

Jan

[constant...@gmail.com]

Indeed, I have now reverted the cell config to the original value and that issue went away; However there's another one, somewhere in the internals of jailhouse_pci_virtual_root_devices_add(), no less daunting to me. Any other ideas?

Best Regards,
Constantin

[Jan Kiszka]

On 2017-04-12 11:28, constant...@gmail.com wrote:
> Indeed, I have now reverted the cell config to the original value and that issue went away; However there's another one, somewhere in the internals of jailhouse_pci_virtual_root_devices_add(), no less daunting to me. Any other ideas?

That looks... interesting. Maybe a regression with 4.11 - I haven't
tested my branch on arm64 since 4.10, only on x86 so far.

You could try to rebase the patches on top of a 4.10 release (should be
straightforward, just the first patch becomes obsolete) and retry.

I'll cross-check on one of our targets ASAP.

[Constantin Petra]

Hello,

I have tried a kernel 4.10 from torvalds/linux branch with applied patches, however something is not in place. When running petalinux-build or petalinux-config, I get:

 make: Entering directory '/home/osboxes/work/peta/xilinx-zcu102-zu9-es2-rev1.0-2016.4/components/linux-kernel/components/linux-kernel/xlnx-kiszka-4.10/linux'

| make: *** No rule to make target 'oldnoconfig'.  Stop

I have tried also a kernel 4.9 from the linux-xlnx branch however the eth drivers don't start correctly so I'm blocked a bit.

Do you have any snapshot I can use, or some hints?

Thanks,

Constantin

[Jan Kiszka]

On 2017-04-14 09:50, Constantin Petra wrote:
> Hello,
>
> I have tried a kernel 4.10 from torvalds/linux branch with applied
> patches, however something is not in place. When running petalinux-build
> or petalinux-config, I get:
>
> make: Entering directory
> '/home/osboxes/work/peta/xilinx-zcu102-zu9-es2-rev1.0-2016.4/components/linux-kernel/components/linux-kernel/xlnx-kiszka-4.10/linux'
> | make: *** No rule to make target 'oldnoconfig'. Stop
>
> I have tried also a kernel 4.9 from the linux-xlnx branch however the
> eth drivers don't start correctly so I'm blocked a bit.
> Do you have any snapshot I can use, or some hints?

I just reproduced the issue with 4.11 on a HiKey board. Will let you
know once I have a solution.

Jan

[Jan Kiszka]

Seems like the fix is trivial:

diff --git a/driver/pci.c b/driver/pci.c
index e3c281e6..9750ac44 100644
--- a/driver/pci.c
+++ b/driver/pci.c
@@ -336,8 +336,6 @@ static bool create_vpci_of_overlay(struct
jailhouse_system *config)
if (overlay_id < 0)
goto out;

- of_node_put(overlay);
-
success = true;

out:

Jan

[constant...@gmail.com]

Hi,

I have made the changes to pci.c, but QEMU seems to freeze after enabling Jailhouse after the "opening" message is displayed (I have previously started a SSH connection which also freezes - to make sure the Serial port is not somehow hijacked by hypervisor as described in the documentaiton). I cannot start GDB using petalinux tools due to a Python 2.6 dependency (the system has Python 2.7) so I don't really know what happens. Any ideas? (Attached also a log file with the boot messages; tried relying on configured dtb and here also tried memmap for the reserved memory, with the same result).

root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/zynqmp-zcu102.cell

Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on CPU 0
Code location: 0x0000ffffc0200040
Page pool usage after early setup: mem 33/996, remap 64/131072
Initializing processors:
CPU 0... OK
CPU 1... OK
CPU 2... OK
CPU 3... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 44/996, remap 69/131072
Activating hypervisor
[ 71.547900] OF: PCI: host bridge //vpci@0 ranges:
[ 71.549101] OF: PCI: MEM 0xfc100000..0xfc103fff -> 0xfc100000
[ 71.552493] pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0fffff] for [bus 00]
[ 71.555644] pci-host-generic fc000000.vpci: PCI host bridge to bus 0001:00
[ 71.556337] pci_bus 0001:00: root bus resource [bus 00]
[ 71.556698] pci_bus 0001:00: root bus resource [mem 0xfc100000-0xfc103fff]
[ 71.569008] pci 0001:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 64bit]
[ 71.570284] pci 0001:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 64bit]
[ 71.576654] The Jailhouse is opening.
root@plnx_aarch64:~#

Best Regards,
Constantin

[Jan Kiszka]

On 2017-04-18 10:22, constant...@gmail.com wrote:
> Hi,
>
> I have made the changes to pci.c, but QEMU seems to freeze after enabling Jailhouse after the "opening" message is displayed (I have previously started a SSH connection which also freezes - to make sure the Serial port is not somehow hijacked by hypervisor as described in the documentaiton). I cannot start GDB using petalinux tools due to a Python 2.6 dependency (the system has Python 2.7) so I don't really know what happens. Any ideas? (Attached also a log file with the boot messages; tried relying on configured dtb and here also tried memmap for the reserved memory, with the same result).
>

Oh, now I get it: You are running the whole stack emulated in QEMU, not
on real hardware, right? Maybe there is a subtle difference in the QEMU
system configuration compared to the board, or Jailhouse is stressing
some aspect of virtualization that the QEMU model does not yet handle
correctly or at all.

Adding Edgar, maybe he can comment on the state of hypervisor execution
in the Xilinx QEMU version. He will probably need some details about the
setup (versions, configuration etc.).

Jan

[Constantin Petra]

Hi,

Yes, that's correct, I'm trying to run the whole stack in QEMU, not hardware.

Best Regards,

Constantin

[Constantin Petra]

Hello,

I have tried the same binaries on a ZCU102 board and it seems to work. Cannot be sure though, the board is remote and I don't have access to the secondary UART as I understand it would be required for the uart-demo.

So from some reason, under QEMU something is not in place, but what can that be? It would be convenient from our perspective, if it would work...

root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell list

ID      Name                    State           Assigned CPUs           Failed CPUs

0       ZynqMP-ZCU102           running         0-2

1       gic-demo                running         3

Best Regards,

Constantin

[Jan Kiszka]

On 2017-04-19 05:48, Constantin Petra wrote:
> Hello,
>
> I have tried the same binaries on a ZCU102 board and it seems to work.
> Cannot be sure though, the board is remote and I don't have access to
> the secondary UART as I understand it would be required for the uart-demo.

You can always reconfigure this to the primary one. We share it in other
setups as well.

> So from some reason, under QEMU something is not in place, but what can
> that be? It would be convenient from our perspective, if it would work...

I fully agree, and I'm waiting for official EL2 support in QEMU in order
to introduce a virtual target for ARM[64], just like we have on x86 already.

I suppose you are using Xilinx' version of QEMU, not upstream, correct?
I didn't track recently what the state of upstream is. Just checking the
list again, seeing some interesting patches from early this year...
Peter, can you comment on what is still missing? I'm reading that you
successfully booted KVM inside QEMU. Seems like we are pretty close now
and it could make sense to debug the remaining issues.

Jan

>
> root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell list
> ID Name State Assigned CPUs
> Failed CPUs
> 0 ZynqMP-ZCU102 running 0-2
> 1 gic-demo running 3
>
>
> Best Regards,
> Constantin
>
> On Tue, Apr 18, 2017 at 1:32 PM, Constantin Petra
> <constant...@gmail.com <mailto:constant...@gmail.com>> wrote:
>
> Hi,
>
> Yes, that's correct, I'm trying to run the whole stack in QEMU, not
> hardware.
>
> Best Regards,
> Constantin
>
> On Tue, Apr 18, 2017 at 12:52 PM, Jan Kiszka <jan.k...@web.de
> <mailto:jan.k...@web.de>> wrote:
>
> On 2017-04-18 10:22, constant...@gmail.com

[Constantin Petra]

Yes, I am using the QEMU version included by Xilinx in PetaLinux 2016.4.

[Peter Maydell]

On 19 April 2017 at 05:37, Jan Kiszka <jan.k...@web.de> wrote:
> On 2017-04-19 05:48, Constantin Petra wrote:
>> So from some reason, under QEMU something is not in place, but what can
>> that be? It would be convenient from our perspective, if it would work...
>
> I fully agree, and I'm waiting for official EL2 support in QEMU in order
> to introduce a virtual target for ARM[64], just like we have on x86 already.

This has been in upstream QEMU for a little while now,
and will be in the upcoming 2.9 release. (Applies only for
AArch64 hosts using GICv3, needs some board support changes
which have currently only been done for 'virt'.)

thanks
-- PMM

[Jan Kiszka]

Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
GICv3 silicon in our hands that we could test against (do you know some
recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
broken by now (due to lacking tests), plus it is not yet enabled for our
ARM64 port. But I'm a bit reluctant to enable this only over QEMU...

Jan

[Edgar E. Iglesias]

Hi,

Xilinx QEMU has support for the virtualization extensions in the GICv2 but
we're seeing issues with running hypervisors in multi-core setups.
It seems like guest migration between cores is causing trouble but we
haven't pinpointed the exact cause.

I'd be interested in debugging the problem seen with Jailhouse.
Can the images and instructions on howto run be shared with me?

Thanks,
Edgar

[Constantin Petra]

Hi,

Sure, I have added the archive containing the image files and jailhouse build here (the archive contains the /tftpboot folder directly)

https://drive.google.com/file/d/0B2bhbcx8eg-caTBNdnZ4cWhJcWc/view?usp=sharing

I use the following sequence to start qemu / kernel / jailhouse, provided the /tftpboot folder has the structure in the previously archive and, of course, there is tftp access to the host)

petalinux-boot --qemu --uboot --qemu-args "-redir tcp:10022:10.0.2.15:22"

In QEMU:

setenv bootargs 'console=ttyPS0,115200'; tftpboot 0x10000000  image.ub; bootm

tftp -g 10.0.2.2 jailhouse.tar.gz -r jailhouse.tar.gz

tar zvxf jailhouse.tar.gz

mv tftpboot/jailhouse jailhouse

mkdir -p  /lib/firmware

cp ./jailhouse/lib/firmware/jailhouse.bin /lib/firmware

insmod ./jailhouse/lib/modules/4.11.0-rc4/extra/driver/jailhouse.ko

sleep 2

jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/zynqmp-zcu102.cell

Thank you,

Constantin

[Constantin Petra]

OK,

Changed configs for inmate to UART0, and the messages appeared on the console (on the ZCU102 board, not QEMU), so it is indeed functional.

root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell start ZynqMP-linux-demo

Started cell "ZynqMP-linux-demo"

Hello 1 from cell!

Hello 2 from cell!

Hello 3 from cell!

---

etc

Edgar, are the images I have sent of any help?

Let me know if you need any other information about the setup.

Best Regards,

Constantin

P.S. Sent the above previously by chance only to Edgar, reposted again for the larger audience :)

[Edgar E. Iglesias]

On Fri, Apr 21, 2017 at 10:04:56AM +0300, Constantin Petra wrote:
> OK,
>
> Changed configs for inmate to UART0, and the messages appeared on the
> console (on the ZCU102 board, not QEMU), so it is indeed functional.
> root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell start
> ZynqMP-linux-demo
> Started cell "ZynqMP-linux-demo"
> Hello 1 from cell!
> Hello 2 from cell!
> Hello 3 from cell!
> ---
> etc
>
> Edgar, are the images I have sent of any help?

Thanks Constantin, yes, I can reproduce a lockup.
This is what I see:

root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/zynqmp-zcu102.cell

Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on CPU 2

Code location: 0x0000ffffc0200040
Page pool usage after early setup: mem 33/996, remap 64/131072
Initializing processors:

CPU 2... OK
CPU 3... OK
CPU 0... OK
CPU 1... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 44/996, remap 69/131072
Activating hypervisor

[ 73.743265] OF: PCI: host bridge //vpci@0 ranges:
[ 73.744161] OF: PCI: MEM 0xfc100000..0xfc103fff -> 0xfc100000
[ 73.747033] pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0fffff] for [bus 00]
[ 73.748659] pci-host-generic fc000000.vpci: PCI host bridge to bus 0001:00
[ 73.748972] pci_bus 0001:00: root bus resource [bus 00]
[ 73.749644] pci_bus 0001:00: root bus resource [mem 0xfc100000-0xfc103fff]
[ 73.758116] pci 0001:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 64bit]
[ 73.758928] pci 0001:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 64bit]
[ 73.761465] The Jailhouse is opening.
root@plnx_aarch64:~# [ 74.710264] ivshmem-net 0001:00:00.0: enabling device (0000 -> 0002)
[ 74.713514] ivshmem-net: probe of 0001:00:00.0 failed with error -16
[ 74.714971] ivshmem-net 0001:00:01.0: enabling device (0000 -> 0002)
[ 74.715904] ivshmem-net: probe of 0001:00:01.0 failed with error -16


After that, it all deadlocks.
I'll see what I can find.

Cheers,
Edgar

[Constantin Petra]

Hi,

Any findings on this case? Is there any chance PetaLinux 2017.1 addressed this issue?

Thank you,
Constantin