Jailhouse installation fine but kernel traps occuring

[swara...@intel.com]

Hi,

It's been quite some time I have been working with Jan to setup Jailhouse on my board as it is an Apollolake board which is quite rare such as adding MMIO UART support, using 32 bit registers for UART output, extending IOAPIC to 120 pins, modified the root config file etc.

The thing now is that from the hypervisor output it seems like Jailhouse is now being setup fine (hypervisor_output.txt), no hangs , no crashes however I cannot interact with Linux anymore, i.e. I am getting traps(in hypervisor_output.txt). the `cd` command works fine but others such as `ls` or 'top' are not working hence I cannot create my non root cell.

I am currently using the next branch and already reverted commit 6ebd111c86 which can allow me to run jailhouse without VT-d (entries missing in my ACPI table, BIOS team looking into it) however till then, it's a bit hard to find out why I am getting traps, may be I am missing a step such as a parameter in my grub file or a particular command during starting jailhouse or something wrong in config file? (jailhouse output seems ok though..).

P.S I have attached my grub file, hypervisor output and root cell config file, Any suggestions would be helpful.

Regards,
Swaraj Dube.

[Jan Kiszka]

> root@intel-corei7-64:~/jailhouse# ls
> [ 4614.157016] traps: ls[1812] trap invalid opcode ip:37a0620998 sp:7ffff99e81d8 error:0 in libc-2.22.so[37a0600000+19a000]

Please disassemble what's in your libc-2.22.so at 0x20998 (objdump -d
/lib64/libc-2.22.so).

Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[Dube, Swaraj]

File too big to send , anyway no need to send even a zip file. After searching '20998' (of course this is the invalid opcode ip for all the commands), this is the only thing I am seeing:

37a0620998: 0f 01 d0 xgetbv

Reading value from extended control register (XCR) it seems, will try to see further into this.

Regards,
Swaraj Dube.

--
You received this message because you are subscribed to the Google Groups "Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan Kiszka]

On 2016-08-23 23:13, Dube, Swaraj wrote:
> File too big to send , anyway no need to send even a zip file. After searching '20998' (of course this is the invalid opcode ip for all the commands), this is the only thing I am seeing:
>
> 37a0620998: 0f 01 d0 xgetbv
>
> Reading value from extended control register (XCR) it seems, will try to see further into this.

OK, this means something prevents the enabling of CR4.OSXSAVE for the
guest. Used to work, strange...

Could you catch the output with this instrumentation:

diff --git a/hypervisor/arch/x86/vmx.c b/hypervisor/arch/x86/vmx.c
index e77c97a..197db7f 100644
--- a/hypervisor/arch/x86/vmx.c
+++ b/hypervisor/arch/x86/vmx.c
@@ -424,6 +424,7 @@ static bool vmx_set_guest_cr(unsigned int cr_idx, unsigned long val)
{
bool ok = true;

+if (cr_idx) printk("CR4: val %x, maybe1 %x, required1 %x, host %x\n", val, cr_maybe1[cr_idx], cr_required1[cr_idx], read_cr4());
if (cr_idx)
val |= X86_CR4_VMXE; /* keeps the hypervisor visible */


Thanks,

[Dube, Swaraj]

I see , this is the changed output now (CPU Initialization) after adding the print messages:

CPU 2... (APIC ID 4) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
OK
CPU 3... (APIC ID 6) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
OK
CPU 0... (APIC ID 0) CR4: val 3006f0, maybe1 3527ff, required1 2000, host 42020
OK
CPU 1... (APIC ID 2) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
OK

Regards,
Swaraj Dube.

-----Original Message-----
From: Jan Kiszka [mailto:jan.k...@siemens.com]
Sent: Wednesday, August 24, 2016 12:41 PM
To: Dube, Swaraj <swara...@intel.com>; Jailhouse <jailho...@googlegroups.com>
Cc: Hii, Roland King Guan <roland.kin...@intel.com>
Subject: Re: Jailhouse installation fine but kernel traps occuring

On 2016-08-23 23:13, Dube, Swaraj wrote:
> File too big to send , anyway no need to send even a zip file. After searching '20998' (of course this is the invalid opcode ip for all the commands), this is the only thing I am seeing:
>
> 37a0620998: 0f 01 d0 xgetbv
>
> Reading value from extended control register (XCR) it seems, will try to see further into this.

OK, this means something prevents the enabling of CR4.OSXSAVE for the guest. Used to work, strange...

Could you catch the output with this instrumentation:

diff --git a/hypervisor/arch/x86/vmx.c b/hypervisor/arch/x86/vmx.c index e77c97a..197db7f 100644
--- a/hypervisor/arch/x86/vmx.c
+++ b/hypervisor/arch/x86/vmx.c
@@ -424,6 +424,7 @@ static bool vmx_set_guest_cr(unsigned int cr_idx, unsigned long val) {
bool ok = true;

+if (cr_idx) printk("CR4: val %x, maybe1 %x, required1 %x, host %x\n",

+val, cr_maybe1[cr_idx], cr_required1[cr_idx], read_cr4());

[Dube, Swaraj]

Apologies for the format, see if this one looks fine.

[Jan Kiszka]

On 2016-08-24 01:12, Dube, Swaraj wrote:
> I see , this is the changed output now (CPU Initialization) after adding the print messages:
>
> CPU 2... (APIC ID 4) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
> OK
> CPU 3... (APIC ID 6) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
> OK
> CPU 0... (APIC ID 0) CR4: val 3006f0, maybe1 3527ff, required1 2000, host 42020
> OK
> CPU 1... (APIC ID 2) CR4: val 3006e0, maybe1 3527ff, required1 2000, host 42020
> OK

Weird: In order to allow userspace to use xgetbv, bit 18 has to be
enabled in CR4. On the host side (Jailhouse), this is the case (last
value) because Jailhouse detects the availability of OSXSAVE and turns
this on for itself. But the value read in arch_cpu_init() while
initializing Jailhouse apparently does not have it set.

Please try this instrumentation / workaround combo, also sending the
output:

diff --git a/driver/main.c b/driver/main.c
index 746de07..85014e3 100644
--- a/driver/main.c
+++ b/driver/main.c
@@ -145,6 +145,8 @@ static void enter_hypervisor(void *info)
int (*entry)(unsigned int);
int err;

+printk("Linux CR4: %lx %lx\n", this_cpu_read(cpu_tlbstate.cr4), native_read_cr4());
+native_write_cr4(this_cpu_read(cpu_tlbstate.cr4));
entry = header->entry + (unsigned long) hypervisor_mem;

if (cpu < header->max_cpus)

[Jan Kiszka]

We have a bug in Jailhouse, reproducible by forcing Linux to not use
xsave (noxsave kernel param) while the hardware supports this. Wrapping
my head around it...

[Jan Kiszka]

Please try the latest next branch, it should fix the issue.

Thanks,

[swara...@intel.com]

Yes, the bug is gone now the OSXSAVE bit in CR4 is enabled, Jailhouse is running fine.