Jailhouse under qemu on a Xeon leads to an unresponsive shell

[randy....@intel.com]

Host = Dual Socket Ivy Bridge Xeon(though I see the same behavior on others)

Using an Arch Linux image I created for jailhouse, I boot into qemu and follow the instructions for starting jailhouse. ie.:

insmod /root/jailhouse/driver/jailhouse.ko
/root/jailhouse/tools/jailhouse enable /root/jailhouse/configs/qemu-vm.cell

On a corei7, this works as expected and drops me back to the shell where I can then start another inmate such as the apci demo.

However, when using Xeon, the shell becomes unresponsive.

According to gdb attached to qemu, the four cores are either in native_safe_halt() or native_apic_msr_eoi_write(). Now of course, I'm only looking at snapshots in time right now, and haven't done full instruction dumps.

However, before I did that I wanted to see if this is expected behavior, or if I potentially need to do something else to get this to work on Xeon.

The interest comes from wanting to potentially do some light smoke tests without needing physical hardware outside of a build machine.

[Jan Kiszka]

Did you use all -cpu QEMU switches mentioned in the readme and only
varied the host CPU?

It might be a nested KVM issue. What is your host kernel version? Can
you retry with the latest one? Taking ftraces on the host, focusing on
KVM events can then be a next step. In a first step, a comparison of the
KVM activities between working and non-working would be good. Further
analysis may eventually require to get some KVM folks involved (I'm no
longer up-to-date with nested virt of KVM on Intel).

Jan
--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[zedia...@gmail.com]

On Tuesday, August 1, 2017 at 11:50:51 PM UTC-7, J. Kiszka wrote:
> On 2017-08-01 20:20, randy....@intel.com wrote:
> > Host = Dual Socket Ivy Bridge Xeon(though I see the same behavior on others)
> >
> > Using an Arch Linux image I created for jailhouse, I boot into qemu and follow the instructions for starting jailhouse. ie.:
> >
> > insmod /root/jailhouse/driver/jailhouse.ko
> > /root/jailhouse/tools/jailhouse enable /root/jailhouse/configs/qemu-vm.cell
> >
> > On a corei7, this works as expected and drops me back to the shell where I can then start another inmate such as the apci demo.
> >
> > However, when using Xeon, the shell becomes unresponsive.
> >
> > According to gdb attached to qemu, the four cores are either in native_safe_halt() or native_apic_msr_eoi_write(). Now of course, I'm only looking at snapshots in time right now, and haven't done full instruction dumps.
> >
> > However, before I did that I wanted to see if this is expected behavior, or if I potentially need to do something else to get this to work on Xeon.
> >
> > The interest comes from wanting to potentially do some light smoke tests without needing physical hardware outside of a build machine.
> >
>
> Did you use all -cpu QEMU switches mentioned in the readme and only
> varied the host CPU?
>

Yes, I believe so. For sanity's sake here is the command:

qemu-system-x86_64 -machine q35,kernel_irqchip=split -m 1G -enable-kvm \
-smp 4 -device intel-iommu,intremap=on,x-buggy-eim=on \
-cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx \
-drive file=archlinux-jailhouse.img,format=raw,id=disk,if=none \
-device ide-hd,drive=disk -serial mon:stdio -serial vc \
-netdev user,id=net -device e1000e,addr=2.0,netdev=net \
-device intel-hda,addr=1b.0 -device hda-duplex -s

> It might be a nested KVM issue. What is your host kernel version? Can
> you retry with the latest one?

I was using 4.11 on the host.(4.11 in the guest as well) I just tried with 4.12 on the host and see the same behavior.

> Taking ftraces on the host, focusing on
> KVM events can then be a next step. In a first step, a comparison of the
> KVM activities between working and non-working would be good. Further
> analysis may eventually require to get some KVM folks involved (I'm no
> longer up-to-date with nested virt of KVM on Intel).
>

Ok, thanks. I'll try some tracing and see if I can see some obvious differences between the working and non-working case.