QEMU VM hang after runnig jailhouse enable configs/qemu-vm.cell

[vinee...@gmail.com]

Hi
started the qemu vm with following command line. qemu-version: 2.9

#qemu-system-x86_64 -machine q35,kernel_irqchip=split -m 4G -enable-kvm -smp 4 -device intel-iommu,intremap=on,x-buggy-eim=on -cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx,+x2apic -drive file=ubuntu.img,format=raw,id=disk,if=none -device ide-hd,drive=disk -serial mon:stdio -serial vc -netdev user,id=net -device e1000e,addr=2.0,netdev=net -device intel-hda,addr=1b.0 -device hda-duplex -fsdev local,path=/home/vkk,security_model=passthrough,id=vfs -device virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs

procedure:

Qemu guest OS: ubuntu 16.04, kernel version: 4.8. Host os: ubuntu 16.04, kernelversion: 4.8

mounted the host file system for accessing the jailhouse on guest os, then ran the command: make firmware_install
after that, inserted the jailhouse.ko driver

then tried to run the vm cell
./tools/jailhouse enable configs/qemu-vm.cell

it hung the vm and shows following logs in host commandline.
Please help me to fix this

Initializing Jailhouse hypervisor v0.7 on CPU 2
Code location: 0xfffffffff0000050
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
Initializing processors:
CPU 2... (APIC ID 2) OK
CPU 0... (APIC ID 0) OK
CPU 1... (APIC ID 1) OK
CPU 3... (APIC ID 3) OK
DMAR unit @0xfed90000/0x1000
Reserving 24 interrupt(s) for device ff00 at index 0
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 0010 at index 24
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00d8 at index 29
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00fa at index 30
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00ff at index 31
Adding virtual PCI device 00:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 254/1480, remap 65607/131072
ActivFATAL: unable to get MMIO instruction
FATAL: Invalid MMIO/RAM write, addr: 0x0000000176027ff8 size: 0
RIP: 0xffffffffc039bc77 RSP: 0xffffa06fbfd83f48 FLAGS: 10002
RAX: 0x0000000000000000 RBX: 0xffffa06fbfd9d720 RCX: 0x0000000000000000
RDX: 0x0000000000000300 RSI: 0x0000000000000000 RDI: 0xfffffffff0045060
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x0000000176027000 CR4: 0x00000000000026e0
EFER: 0x0000000000000d01
Parking CPU 3 (Cell: "QEMU-VM")
VT-d fault event reported by IOMMU 0:
Source Identifier (bus:dev.func): 00:02.0
Fault Reason: 0x6 Fault Info: 1778c4 Type 1

regards
Vineesh

[Henning Schild]

Am Mon, 19 Jun 2017 23:05:27 -0700
schrieb <vinee...@gmail.com>:

That is the ethernet card accessing RAM for DMA. You started your qemu
with -m 4G instead of -m 1G and the additional 3G are not mentioned in
the default config. You could add the additional memory to the config
or go with 1G.

Henning

>
> regards
> Vineesh
>
>
>

[Vineesh Kumar]

thanks for the reply Henning.

Tried with -m 1G. it showed the following serial logs and Guest OS was rebooted. is this expected? I dont see any cell listed after running jailhouse cell list. Do i need to do any other changes?

qemu-system-x86_64 -machine q35,kernel_irqchip=split -m 1G -enable-kvm -smp 4 -device intel-iommu,intremap=on,x-buggy-eim=on -cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx,+x2apic -drive file=ubuntu.img,format=raw,id=disk,if=none -device ide-hd,drive=disk -serial mon:stdio -serial vc -netdev user,id=net -device e1000e,addr=2.0,netdev=net -device intel-hda,addr=1b.0 -device hda-duplex -fsdev local,path=/home/vkk,security_model=passthrough,id=vfs -device virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs
audio: Could not init `oss' audio driver

Initializing Jailhouse hypervisor v0.7 on CPU 3

Code location: 0xfffffffff0000050
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
Initializing processors:

CPU 3... (APIC ID 3) OK

CPU 0... (APIC ID 0) OK
CPU 2... (APIC ID 2) OK

rgds
vineesh

[Henning Schild]

Am Tue, 20 Jun 2017 01:40:52 -0700
schrieb Vineesh Kumar <vinee...@gmail.com>:

After "jailhouse enable" you should see one cell in the list, otherwise
something did not work as expected.

> qemu-system-x86_64 -machine q35,kernel_irqchip=split -m 1G
> -enable-kvm -smp 4 -device intel-iommu,intremap=on,x-buggy-eim=on
> -cpu
> kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx,+x2apic
> -drive file=ubuntu.img,format=raw,id=disk,if=none -device
> ide-hd,drive=disk -serial mon:stdio -serial vc -netdev user,id=net
> -device e1000e,addr=2.0,netdev=net -device intel-hda,addr=1b.0
> -device hda-duplex -fsdev
> local,path=/home/vkk,security_model=passthrough,id=vfs -device
> virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs audio: Could not
> init `oss' audio driver

Please make sure that is exactly the command-line from the readme.

> Initializing Jailhouse hypervisor v0.7 on CPU 3
> Code location: 0xfffffffff0000050
> Using x2APIC
> Page pool usage after early setup: mem 44/1480, remap 64/131072
> Initializing processors:
> CPU 3... (APIC ID 3) OK
> CPU 0... (APIC ID 0) OK
> CPU 2... (APIC ID 2) OK

If you saw that on the terminal where you typed "jailhouse enable"
something went wrong. A similar output to the first time is expected,
but this time without the VT-d fault crashing the root-cell.
You could enable CONFIG_TRACE_ERROR to see where things go wrong. (see
Documentation/hypervisor-configuration.md)

Henning

> rgds
> vineesh

[Vineesh Kumar]

Put additional parameters are for mounting host file system.Otherwise, it is the same commandline.

-fsdev local,path=/home/vkk,security_model=passthrough,id=vfs -device virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs

Sometimes, It just hang without any log print.

CONFIG_TRACE_ERROR is enabled by default in ci/jailhouse-config-x86.h file. but there is no debug print in the console.
I dont see any config.h file as mentioned in documentation.

[Henning Schild]

Am Tue, 20 Jun 2017 21:22:27 -0700

You will have to create that file yourself, if it does not exist you
are building the default config. I am not sure how things would go
wrong if nested kvm is not enabled, please make sure
host:/sys/module/kvm_intel/parameters/nested says Y. Another thing to
check is whether you have the memmap kernel parameter for the guest.
If you forgot one of the two you might not need the error tracing.

regards,
Henning

[Vineesh Kumar]

nested was enabled by default in /etc/modprobe.d/qemu-system-x86.conf.
And also verified now on the path /sys/module/kvm_intel/parameters/nested.
memmap parameter was also given as mentioned in jailhouse doc.
tried with all these combinations:
memmap=66M\\\$0x3b000000
or
memmap=0x4200000\$0x3b000000

regards
Vineesh

[Vineesh Kumar]

after enabling the debug. I am able to see the print "activating Hypervisor". but crashed after that. Anyone faced this issue?

#qemu-system-x86_64 -machine q35,kernel_irqchip=split -m 1G -enable-kvm -smp 4 -device intel-iommu,intremap=on,x-buggy-eim=on -cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx,+x2apic -drive file=ubuntu.img,format=raw,id=disk,if=none -device ide-hd,drive=disk -serial mon:stdio -serial vc -netdev user,id=net -device e1000e,addr=2.0,netdev=net -device intel-hda,addr=1b.0 -device hda-duplex -fsdev local,path=/home/vkk,security_model=passthrough,id=vfs -device virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs

audio: Could not init `oss' audio driver

Initializing Jailhouse hypervisor v0.7 on CPU 3

Code location: 0xfffffffff0000050
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
Initializing processors:
CPU 3... (APIC ID 3) OK

CPU 2... (APIC ID 2) OK
CPU 0... (APIC ID 0) OK
CPU 1... (APIC ID 1) OK

DMAR unit @0xfed90000/0x1000
Reserving 24 interrupt(s) for device ff00 at index 0
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 0010 at index 24
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00d8 at index 29
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00fa at index 30
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00ff at index 31
Adding virtual PCI device 00:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 254/1480, remap 65607/131072

Activating hypervisor
FATAL: Invalid MMIO/RAM write, addr: 0x000000003b544d40 size: 4
RIP: 0xffffffffbe5d307e RSP: 0xffff95564002fc68 FLAGS: 10202
RAX: 0xffff95567531e648 RBX: 0xffff95567531e640 RCX: 0x0000000000000010
RDX: 0xffff95567b544d40 RSI: 0xffff95567b9fb740 RDI: 0xffff95567531e640

CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1

CR0: 0x0000000080050033 CR3: 0x0000000000054000 CR4: 0x00000000000026e0
EFER: 0x0000000000000d01
Parking CPU 1 (Cell: "QEMU-VM")
Ignoring NMI IPI

rgds
vineesh

[Henning Schild]

Am Wed, 21 Jun 2017 21:42:22 -0700

That is Linux trying to access hypervisor memory. You should
double-check the memmap reservation. /proc/cmdline should show it and
you should see the reservation in /proc/iomem as well.
Things to check: the escaping of the dollar sign in grub2, running
grub-mkconfig to actually apply the changes in /etc/default/grub to the
real grub config file.

Henning

> rgds
> vineesh

[Vineesh Kumar]

Thanks

regards
Vineesh