Jailhouse Images Issues

[michael....@gmail.com]

Hello,

I'm struggling to get a Jailhouse demo up and running in QEMU using jailhouse-images.

In jailhouse-images/, I was able to build the QEMU images via the `build-images.sh` script on an Ubuntu 17 machine, and I was able to start my custom-built QEMU 2.11 by setting the path of the QEMU_PATH environmental variable to point to qemu/x86_64-softmmu.

I then ran:

start-qemu.sh x86

The QEMU VM starts up as expected, I log in as root, and then I run these commands:

jailhouse hardware check /etc/jailhouse/qemu-x86.cell
jailhouse enable /etc/jailhuose/qemu-x86.cell
jailhouse cell create /etc/jailhouse/apic-demo.cell
jailhouse cell load apic-demo /usr/libexec/jailhouse/demos/apic-demo.bin
jailhouse cell start apic-demo

The hardware check passes. However, I get the following output and error on my Ubuntu:

Initializing Jailhouse hypervisor v0.8 (0-g07341fa3-dirty) on CPU 0
Code location: 0xfffffffff0000060
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
Initializing processors:
CPU 0... (APIC ID 0) OK
CPU 2... (APIC ID 2) OK
CPU 3... (APIC ID 3) OK
CPU 1... (APIC ID 1) OK
DMAR unit @0xfed90000/0x1000
Reserving 24 interrupt(s) for device ff00 at index 0
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 0010 at index 24
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00d8 at index 29
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00fa at index 30
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00ff at index 31
Adding virtual PCI device 00:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 254/1480, remap 65607/131072
Activating hypervisor
Created cell "apic-demo"
Page pool usage after cell creation: mem 270/1480, remap 65607/131072
Cell "apic-demo" can be loaded
Started cell "apic-demo"
CPU 3 received SIPI, vector 100
FATAL: Unhandled VM-Exit, reason 2
qualification 0
vectoring info: 0 interrupt info: 0
RIP: 0x000000000000119f RSP: 0x0000000000008fd8 FLAGS: 10006
RAX: 0x3b9aca0000000000 RBX: 0x0000000000000280 RCX: 0x0000000000000000
RDX: 0x0000000000000000 RSI: 0x000000000000504b RDI: 0x0000000000000100
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080010031 CR3: 0x0000000000002000 CR4: 0x0000000000002020
EFER: 0x0000000000000500
Parking CPU 3 (Cell: "apic-demo")

Consulting this (https://groups.google.com/forum/#!topic/jailhouse-dev/8IO2Ot4iXlw), it appears that memory is not being mapped properly in the inmate. However, I'm not really sure how to even debug this or where to look for error logs. Am I doing something wrong? How can I debug this?

Thanks,
Michael Hinton

[Jan Kiszka]

This is a divide-by-zero crash in the demo, more precisely in
tsc_init(). The TSC frequency is supposed to be provided by the root
Linux (in the VM, not the host Linux). For some reason, 0 seems to be
passed here (works for me, though).

Could you check in the guest Linux if the clocksources is tsc?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

If not, please provide dmesg from inside the guest.

We probably need some detection in the inmate library and graceful error
handling...

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[michael....@gmail.com]

Is there a reason why my original response was deleted? I'm confused.

[Jan Kiszka]

On 2018-05-11 16:52, michael....@gmail.com wrote:
>> Could you check in the guest Linux if the clocksources is tsc?
>> cat /sys/devices/system/clocksource/clocksource0/current_clocksource
>

> It says hpet. avaliable_clocksource shows hpet and acpi_pm.

So, tsc is missing. Some disturbances on the host may have caused the
guest to give up on the tsc. Full dmesg should tell us.

>
>> If not, please provide dmesg from inside the guest.
>

> I've attached a screenshot of the dmesg below.

Text files, please.

Jan

PS: Your reply got stuck in Google's spam filter. Don't ask me why they
rank mails for their own users as spam...

[michael....@gmail.com]

Thanks for your response. Sorry it took so long to respond.

I've attached the dmesg output of the guest. Of note, it says `tsc: Fast TSC calibration failed. Unable to calibrate against PIT. No reference (HPET/PMTIMER) available. HPET/PMTIMER calibration failed. Marking TSC unstable due to could not calculate TSC khz`.

Also, the guest takes forever to start. This might be related to the failures with the timer, like you mentioned.

[Jan Kiszka]

Very strange. I cannot reproduce the issue here, even when running on
battery (slowed down) and when giving whole QEMU only a single physical
CPU. The early TSC calibration then fails, but HPET-based calibration
will always succeed later on, and the TSC is fine.

What will likely work for you is removing "-kvmclock" from the CPU flags
in start-qemu.sh. However, this will cause an access violation detected
by Jailhouse when you restart a CPU in the root cell (e.g. after cell
destruction). Jailhouse does not pass the KVM paravirtual MSRs through
and will rather complain about those unknown accesses.

Another alternative: Extract the TSC calibration value from the host
boot, e.g., and put it in configs/x86/qemu-x86.c,
.platform_info.x86.tsc_khz.

[michael....@gmail.com]

One thing I should have mentioned earlier is that this attempt was done on a Digital Ocean droplet running Ubuntu. Since Digital Ocean provides droplets via vm trickery, it makes sense that it doesn't play well with Jailhouse.

I have since abandoned trying to get this working, and instead am running jailhouse using jailhouse-images on a physical machine running Ubuntu 18.04, which is working for me (see https://groups.google.com/forum/#!topic/jailhouse-dev/R4e8vjFnhJg).

Consider this closed, unless someone else has insights on how to get Jailhouse running in a droplet.

Thanks,
-Michael