[PATCH 0/5] Linux 4.0 adaptions, Jetson TK1 board support

[Jan Kiszka]

This adjusts some details in the driver and in VMX code for running
Jailhouse with latest Linux 4.0-rc1.

Moreover, it comes with support for a new ARM board: The NVIDIA Jetson
TK1. Same level of support as for the Banana Pi so far, but we should be
able to move beyond that (SMMU...). Note that the TK1 currently requires
some not yet upstream (and likely also finished) patches. Latest version
is https://github.com/siemens/u-boot/tree/jetson-tk1-v4.

Jan

Jan Kiszka (5):
TODO: Add items about MMIO dispatching enhancements
driver: Account for removal of cpumask_scnprintf
x86: vmx: Hide VMXE from guest-visible CR4
arm: Add support for Jetson TK1 board
arm: Power off non-root CPUs before shutting down the hypervisor

TODO.md | 10 ++
configs/jetson-tk1-demo.c | 54 +++++++++
configs/jetson-tk1.c | 128 +++++++++++++++++++++
driver.c | 10 ++
hypervisor/arch/arm/Makefile | 2 +
hypervisor/arch/arm/control.c | 15 ++-
hypervisor/arch/arm/include/asm/platform.h | 17 +++
hypervisor/arch/arm/include/asm/uart-tegra.h | 53 +++++++++
hypervisor/arch/arm/smp-tegra124.c | 26 +++++
hypervisor/arch/arm/uart-tegra.c | 25 ++++
hypervisor/arch/x86/vmx.c | 3 -
inmates/lib/arm/Makefile.lib | 2 +
.../lib/arm/include/mach-tegra124/mach/gic_v2.h | 14 +++
inmates/lib/arm/include/mach-tegra124/mach/timer.h | 14 +++
inmates/lib/arm/include/mach-tegra124/mach/uart.h | 13 +++
inmates/lib/arm/uart-tegra.c | 24 ++++
16 files changed, 405 insertions(+), 5 deletions(-)
create mode 100644 configs/jetson-tk1-demo.c
create mode 100644 configs/jetson-tk1.c
create mode 100644 hypervisor/arch/arm/include/asm/uart-tegra.h
create mode 100644 hypervisor/arch/arm/smp-tegra124.c
create mode 100644 hypervisor/arch/arm/uart-tegra.c
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/gic_v2.h
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/timer.h
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/uart.h
create mode 100644 inmates/lib/arm/uart-tegra.c

--
2.1.4

[Jan Kiszka]

4.0 provides cpumask formatting via the %*pb format specifier.
cpumask_scnprintf was removed. So we need to implement both variants
for staying compatible with older kernels.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
driver.c | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/driver.c b/driver.c
index 5fb9a7f..bc6353d 100644
--- a/driver.c
+++ b/driver.c
@@ -300,8 +300,13 @@ static ssize_t cpus_assigned_show(struct kobject *kobj,
struct cell *cell = container_of(kobj, struct cell, kobj);
int written;

+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,0,0)
+ written = scnprintf(buf, PAGE_SIZE, "%*pb\n",
+ cpumask_pr_args(&cell->cpus_assigned));
+#else
written = cpumask_scnprintf(buf, PAGE_SIZE, &cell->cpus_assigned);
written += scnprintf(buf + written, PAGE_SIZE - written, "\n");
+#endif
return written;
}

@@ -322,8 +327,13 @@ static ssize_t cpus_failed_show(struct kobject *kobj,
JAILHOUSE_CPU_FAILED)
cpu_set(cpu, *cpus_failed);

+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,0,0)
+ written = scnprintf(buf, PAGE_SIZE, "%*pb\n",
+ cpumask_pr_args(cpus_failed));
+#else
written = cpumask_scnprintf(buf, PAGE_SIZE, cpus_failed);
written += scnprintf(buf + written, PAGE_SIZE - written, "\n");
+#endif

free_cpumask_var(cpus_failed);

--
2.1.4

[Jan Kiszka]

Linux gained CR4 shadowing in 4.0: it now keeps a software copy of the
mm's CR4 state and only write updates on changes. But this also means it
will miss changes to this register when enabling or disabling Jailhouse.
That will cause sporadic access violations or false reporting of
virtualization-busy CPUs on Jailhouse enable.

We could fix up CR4 in the driver path, introducing more arch-dependent
code there. Rather go for the simpler solution and drop our visibility.
This comes at the price that trying to start KVM while Jailhouse is in
control will now raise an access violation, no longer a more graceful
-EBUSY of KVM.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

hypervisor/arch/x86/vmx.c | 3 ---
1 file changed, 3 deletions(-)

diff --git a/hypervisor/arch/x86/vmx.c b/hypervisor/arch/x86/vmx.c
index 77eaa0f..8bab76d 100644
--- a/hypervisor/arch/x86/vmx.c
+++ b/hypervisor/arch/x86/vmx.c
@@ -359,9 +359,6 @@ static bool vmx_set_guest_cr(int cr, unsigned long val)
fixed1 &= ~(X86_CR0_NW | X86_CR0_CD);
required1 &= ~(X86_CR0_PE | X86_CR0_PG);
required1 |= X86_CR0_ET;
- } else {
- /* keeps the hypervisor visible */
- val |= X86_CR4_VMXE;
}
ok &= vmcs_write64(cr ? GUEST_CR4 : GUEST_CR0,
(val & fixed1) | required1);
--
2.1.4

[Jan Kiszka]

The Jetson TK1 board uses the NVIDIA K1 processor, namely the Tegra124.
The CPU has 4 Cortex-A15 cores and a low-power companion core which we
do not support, though. This SoC apparently contains all virtualization
features we need, specifically full SMMU coverage of I/O devices.

For now, board support is similar to the Banana Pi: We replicate UART
and SMP/PSCI features and add some configs (only one for both demos as
only one UART is accessible).

Open issues, besides adding SMMU support, are managing the access to the
so-called legacy Interrupt controller of the K1 (an additional stage
between devices and GIC) and sub-page access control to various devices
(same issue as with the Allwinner A20). Also, it becomes clearer than
ever that we urgently need to refactor the UART layer as well as
SMP/PSCI support to reduce duplications.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

configs/jetson-tk1-demo.c | 54 +++++++++
configs/jetson-tk1.c | 128 +++++++++++++++++++++

hypervisor/arch/arm/Makefile | 2 +

hypervisor/arch/arm/include/asm/platform.h | 17 +++
hypervisor/arch/arm/include/asm/uart-tegra.h | 53 +++++++++
hypervisor/arch/arm/smp-tegra124.c | 26 +++++
hypervisor/arch/arm/uart-tegra.c | 25 ++++

inmates/lib/arm/Makefile.lib | 2 +
.../lib/arm/include/mach-tegra124/mach/gic_v2.h | 14 +++
inmates/lib/arm/include/mach-tegra124/mach/timer.h | 14 +++
inmates/lib/arm/include/mach-tegra124/mach/uart.h | 13 +++
inmates/lib/arm/uart-tegra.c | 24 ++++

12 files changed, 372 insertions(+)

create mode 100644 configs/jetson-tk1-demo.c
create mode 100644 configs/jetson-tk1.c
create mode 100644 hypervisor/arch/arm/include/asm/uart-tegra.h
create mode 100644 hypervisor/arch/arm/smp-tegra124.c
create mode 100644 hypervisor/arch/arm/uart-tegra.c
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/gic_v2.h
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/timer.h
create mode 100644 inmates/lib/arm/include/mach-tegra124/mach/uart.h
create mode 100644 inmates/lib/arm/uart-tegra.c

diff --git a/configs/jetson-tk1-demo.c b/configs/jetson-tk1-demo.c
new file mode 100644
index 0000000..02c924d
--- /dev/null
+++ b/configs/jetson-tk1-demo.c
@@ -0,0 +1,54 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Configuration for gic-demo or uart-demo inmate on Jetson TK1:
+ * 1 CPU, 64K RAM, serial port 0
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/types.h>
+#include <jailhouse/cell-config.h>
+
+#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])
+
+struct {
+ struct jailhouse_cell_desc cell;
+ __u64 cpus[1];
+ struct jailhouse_memory mem_regions[2];
+} __attribute__((packed)) config = {
+ .cell = {
+ .name = "jetson-tk1-demo",
+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG,
+
+ .cpu_set_size = sizeof(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ },
+
+ .cpus = {
+ 0x8,
+ },
+
+ .mem_regions = {
+ /* UART */ {
+ .phys_start = 0x70006000,
+ .virt_start = 0x70006000,
+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* RAM */ {
+ .phys_start = 0xfbfe0000,
+ .virt_start = 0,
+ .size = 0x00010000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },
+ },
+};
diff --git a/configs/jetson-tk1.c b/configs/jetson-tk1.c
new file mode 100644
index 0000000..e230590
--- /dev/null
+++ b/configs/jetson-tk1.c
@@ -0,0 +1,128 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Test configuration for Jetson TK1 board
+ * (NVIDIA Tegra K1 quad-core Cortex-A15, 2G RAM)
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/types.h>
+#include <jailhouse/cell-config.h>
+
+#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])
+
+struct {
+ struct jailhouse_system header;
+ __u64 cpus[1];
+ struct jailhouse_memory mem_regions[10];
+ struct jailhouse_irqchip irqchips[1];
+} __attribute__((packed)) config = {
+ .header = {
+ .hypervisor_memory = {
+ .phys_start = 0xfc000000,
+ .size = 0x4000000 - 0x100000, /* -1MB (PSCI) */
+ },
+ .debug_uart = {
+ .phys_start = 0x70006000,
+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_IO,
+ },
+ .root_cell = {
+ .name = "Jetson-TK1",
+
+ .cpu_set_size = sizeof(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = 1,
+ },
+ },
+
+ .cpus = {
+ 0xf,
+ },
+
+ .mem_regions = {
+ /* PCIe */ {
+ .phys_start = 0x01000000,
+ .virt_start = 0x01000000,
+ .size = 0x3f000000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* HACK: Legacy Interrupt Controller */ {
+ .phys_start = 0x60004000,
+ .virt_start = 0x60004000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* HACK: Clock and Reset Controller */ {
+ .phys_start = 0x60006000,
+ .virt_start = 0x60006000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* I2C5/6, SPI */ {
+ .phys_start = 0x7000d000,
+ .virt_start = 0x7000d000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* HACK: Memory Controller */ {
+ .phys_start = 0x70019000,
+ .virt_start = 0x70019000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* MMC0/1 */ {
+ .phys_start = 0x700b0000,
+ .virt_start = 0x700b0000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* RTC + PMC */ {
+ .phys_start = 0x7000e000,
+ .virt_start = 0x7000e000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* USB */ {
+ .phys_start = 0x7d004000,
+ .virt_start = 0x7d004000,
+ .size = 0x00008000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* UART */ {
+ .phys_start = 0x70006000,
+ .virt_start = 0x70006000,
+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_IO,
+ },
+ /* RAM */ {
+ .phys_start = 0x80000000,
+ .virt_start = 0x80000000,
+ .size = 0x7c000000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE,
+ },
+ },
+ .irqchips = {
+ /* GIC */ {
+ .address = 0x50041000,
+ .pin_bitmap = 0xffffffffffffffff,
+ },
+ },
+};
diff --git a/hypervisor/arch/arm/Makefile b/hypervisor/arch/arm/Makefile
index 51e46af..efa56c8 100644
--- a/hypervisor/arch/arm/Makefile
+++ b/hypervisor/arch/arm/Makefile
@@ -25,5 +25,7 @@ obj-$(CONFIG_ARM_GIC_V3) += gic-v3.o
obj-$(CONFIG_ARM_GIC) += gic-v2.o
obj-$(CONFIG_SERIAL_AMBA_PL011) += dbg-write-pl011.o
obj-$(CONFIG_SERIAL_8250_DW) += uart-8250-dw.o
+obj-$(CONFIG_SERIAL_TEGRA) += uart-tegra.o
obj-$(CONFIG_MACH_VEXPRESS) += smp-vexpress.o
obj-$(CONFIG_MACH_SUN7I) += smp-sun7i.o
+obj-$(CONFIG_MACH_TEGRA124) += smp-tegra124.o
diff --git a/hypervisor/arch/arm/include/asm/platform.h b/hypervisor/arch/arm/include/asm/platform.h
index b0d51ee..fa7bfdc 100644
--- a/hypervisor/arch/arm/include/asm/platform.h
+++ b/hypervisor/arch/arm/include/asm/platform.h
@@ -69,6 +69,23 @@

#endif /* CONFIG_MACH_SUN7I */

+#ifdef CONFIG_MACH_TEGRA124
+
+# define GICD_BASE ((void *)0x50041000)
+# define GICD_SIZE 0x1000
+# define GICC_BASE ((void *)0x50042000)
+# define GICC_SIZE 0x2000
+# define GICH_BASE ((void *)0x50044000)
+# define GICH_SIZE 0x2000
+# define GICV_BASE ((void *)0x50046000)
+# define GICV_SIZE 0x2000
+
+# include <asm/gic_v2.h>
+
+# define MAINTENANCE_IRQ 25
+
+#endif /* CONFIG_MACH_TEGRA124 */
+
#define HOTPLUG_SPIN 1
/*
#define HOTPLUG_PSCI 1
diff --git a/hypervisor/arch/arm/include/asm/uart-tegra.h b/hypervisor/arch/arm/include/asm/uart-tegra.h
new file mode 100644
index 0000000..5bed630
--- /dev/null
+++ b/hypervisor/arch/arm/include/asm/uart-tegra.h
@@ -0,0 +1,53 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <jailhouse/mmio.h>
+#include <jailhouse/processor.h>
+#include <asm/debug.h>
+
+#define UART_TX 0x0
+#define UART_DLL 0x0
+#define UART_DLM 0x4
+#define UART_LCR 0xc
+#define UART_LCR_8N1 0x03
+#define UART_LCR_DLAB 0x80
+#define UART_LSR 0x14
+#define UART_LSR_THRE 0x20
+
+static void uart_init(struct uart_chip *chip)
+{
+ chip->virt_base += 0x300;
+/* if (chip->clock_reg)
+ mmio_write32(chip->clock_reg,
+ mmio_read32(chip->clock_reg) |
+ (1 << chip->gate_nr));
+
+ mmio_write32(chip->virt_base + UART_LCR, UART_LCR_DLAB);
+ mmio_write32(chip->virt_base + UART_DLL, 0x0d);
+ mmio_write32(chip->virt_base + UART_DLM, 0);
+ mmio_write32(chip->virt_base + UART_LCR, UART_LCR_8N1);*/
+}
+
+static void uart_wait(struct uart_chip *chip)
+{
+ while (!(mmio_read32(chip->virt_base + UART_LSR) & UART_LSR_THRE))
+ cpu_relax();
+}
+
+static void uart_busy(struct uart_chip *chip)
+{
+}
+
+static void uart_write(struct uart_chip *chip, char c)
+{
+ mmio_write32(chip->virt_base + UART_TX, c);
+}
diff --git a/hypervisor/arch/arm/smp-tegra124.c b/hypervisor/arch/arm/smp-tegra124.c
new file mode 100644
index 0000000..81f9649
--- /dev/null
+++ b/hypervisor/arch/arm/smp-tegra124.c
@@ -0,0 +1,26 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Siemens AG, 2014, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <asm/cell.h>
+#include <asm/psci.h>
+#include <asm/smp.h>
+
+static struct smp_ops tegra124_smp_ops = {
+ .type = SMP_SPIN,
+ .init = psci_cell_init,
+ .cpu_spin = psci_emulate_spin,
+};
+
+void register_smp_ops(struct cell *cell)
+{
+ cell->arch.smp = &tegra124_smp_ops;
+}
diff --git a/hypervisor/arch/arm/uart-tegra.c b/hypervisor/arch/arm/uart-tegra.c
new file mode 100644
index 0000000..634523e
--- /dev/null
+++ b/hypervisor/arch/arm/uart-tegra.c
@@ -0,0 +1,25 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) ARM Limited, 2014
+ * Copyright (c) Siemens AG, 2014, 2015
+ *
+ * Authors:
+ * Jean-Philippe Brucker <jean-phili...@arm.com>
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <asm/uart-tegra.h>
+
+/* All the helpers are in the header, to make them re-usable by the inmates */
+void uart_chip_init(struct uart_chip *chip)
+{
+ chip->wait = uart_wait;
+ chip->busy = uart_busy;
+ chip->write = uart_write;
+
+ uart_init(chip);
+}
diff --git a/inmates/lib/arm/Makefile.lib b/inmates/lib/arm/Makefile.lib
index 2746eda..0cb6a19 100644
--- a/inmates/lib/arm/Makefile.lib
+++ b/inmates/lib/arm/Makefile.lib
@@ -29,11 +29,13 @@ endef

mach-$(CONFIG_MACH_VEXPRESS) := vexpress
mach-$(CONFIG_MACH_SUN7I) := sun7i
+mach-$(CONFIG_MACH_TEGRA124) := tegra124

gic-$(CONFIG_ARM_GIC) := gic-v2.o
gic-$(CONFIG_ARM_GIC_V3) := gic-v3.o
uart-$(CONFIG_SERIAL_AMBA_PL011) := uart-pl011.o
uart-$(CONFIG_SERIAL_8250_DW) := uart-8250-dw.o
+uart-$(CONFIG_SERIAL_TEGRA) := uart-tegra.o

MACHINE := mach-$(mach-y)
DRIVERS := $(gic-y) $(uart-y)
diff --git a/inmates/lib/arm/include/mach-tegra124/mach/gic_v2.h b/inmates/lib/arm/include/mach-tegra124/mach/gic_v2.h
new file mode 100644
index 0000000..e513194
--- /dev/null
+++ b/inmates/lib/arm/include/mach-tegra124/mach/gic_v2.h
@@ -0,0 +1,14 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#define GICD_BASE ((void *)0x50041000)
+#define GICC_BASE ((void *)0x50042000)
diff --git a/inmates/lib/arm/include/mach-tegra124/mach/timer.h b/inmates/lib/arm/include/mach-tegra124/mach/timer.h
new file mode 100644
index 0000000..478f6cb
--- /dev/null
+++ b/inmates/lib/arm/include/mach-tegra124/mach/timer.h
@@ -0,0 +1,14 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#define TIMER_IRQ 27
+#define TIMER_FREQ 12000000
diff --git a/inmates/lib/arm/include/mach-tegra124/mach/uart.h b/inmates/lib/arm/include/mach-tegra124/mach/uart.h
new file mode 100644
index 0000000..7b97dc7
--- /dev/null
+++ b/inmates/lib/arm/include/mach-tegra124/mach/uart.h
@@ -0,0 +1,13 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) Siemens AG, 2015
+ *
+ * Authors:
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#define UART_BASE ((void *)0x70006000)
diff --git a/inmates/lib/arm/uart-tegra.c b/inmates/lib/arm/uart-tegra.c
new file mode 100644
index 0000000..6506195
--- /dev/null
+++ b/inmates/lib/arm/uart-tegra.c
@@ -0,0 +1,24 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) ARM Limited, 2014
+ * Copyright (c) Siemens AG, 2014, 2015
+ *
+ * Authors:
+ * Jean-Philippe Brucker <jean-phili...@arm.com>
+ * Jan Kiszka <jan.k...@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+#include <asm/uart-tegra.h>
+#include <mach/uart.h>
+
+void uart_chip_init(struct uart_chip *chip)
+{
+ chip->virt_base = UART_BASE;
+ chip->wait = uart_wait;
+ chip->write = uart_write;
+ chip->busy = uart_busy;
+ uart_init(chip);
+}
--
2.1.4

[Jan Kiszka]

When handing back a CPU previously used by a non-root cell directly to
Linux during hypervisor shutdown, we have to power it off when Linux is
using PSCI. Otherwise, the CPU_ON command issued by Linux later on to
gain control over the CPU again may not work as expected.

All our supported boards except for the Versatile Express come with PSCI
support. Try both v0.2 and the v0.1 encoding of U-Boot for CPU_OFF. At
least one of them must work.

The GICC is not reset by the power-down, thus we have to reset it shut
it down explicitly.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

hypervisor/arch/arm/control.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/hypervisor/arch/arm/control.c b/hypervisor/arch/arm/control.c
index 0f37064..6b502a5 100644
--- a/hypervisor/arch/arm/control.c
+++ b/hypervisor/arch/arm/control.c
@@ -115,8 +115,19 @@ void arch_reset_self(struct per_cpu *cpu_data)
*/
irqchip_eoi_irq(SGI_CPU_OFF, true);

- /* irqchip_cpu_shutdown already resets the GIC on all CPUs. */
- if (!is_shutdown) {
+ if (is_shutdown) {
+#ifndef CONFIG_MACH_VEXPRESS
+ if (cell != &root_cell) {
+ irqchip_cpu_shutdown(cpu_data);
+
+ smc(PSCI_CPU_OFF, 0, 0, 0);
+ smc(PSCI_CPU_OFF_V0_1_UBOOT, 0, 0, 0);
+ printk("FATAL: PSCI_CPU_OFF failed\n");
+ panic_stop();
+ }
+#endif
+ /* arch_shutdown_self resets the GIC on all remaining CPUs. */
+ } else {
err = irqchip_cpu_reset(cpu_data);
if (err)
printk("IRQ setup failed\n");
--
2.1.4

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

TODO.md | 10 ++++++++++

1 file changed, 10 insertions(+)

diff --git a/TODO.md b/TODO.md
index acf00be..9c8754d 100644
--- a/TODO.md
+++ b/TODO.md
@@ -72,3 +72,13 @@ Monitoring
- cell software watchdog via comm region messages
-> time out pending comm region messages and kill failing cells
(includes timeouts of unanswered shutdown requests)
+
+Misc
+ - generic sub-page access filtering
+ - use bitmap, likely with byte granularity, to filter access on specific
+ registers in a MMIO page
+ - generic and faster MMIO dispatching
+ - use binary search on an per-cell array of (start, size, handler, opaque)
+ entries
+ - should be able to deal with both existing devices as well as sub-page
+ filtering
--
2.1.4

[Valentine Sinitsyn]

Hi Jan,

On 23.02.2015 12:41, Jan Kiszka wrote:
> We could fix up CR4 in the driver path, introducing more arch-dependent
> code there. Rather go for the simpler solution and drop our visibility.

Why not fixing it in the driver, as you propose? This doesn't look
utterly hard at the first glance, but resolves access violation issue. I
haven't checked this myself, but I feel KVM should do something similar.

Moreover, Jailhouse is visible by design on AMD systems, and it doesn't
feel particularly neat to have different behaviors on AMD and Intel here.

Regards,
Valentine

[Jan Kiszka]

I guess I have ARM in mind here where both Jailhouse and KVM crash
utterly when they collide in whichever order.

It's likely not hard, but it adds, as I said, more #ifdef ARCH code to
the driver. And #if KERNEL_VERSION. Well, let me try, then we know how
it looks like and if it works (it should, KVM indeed faces the same issue).

Jan

--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

[Valentine Sinitsyn]

On 23.02.2015 12:55, Jan Kiszka wrote:
> I guess I have ARM in mind here where both Jailhouse and KVM crash
> utterly when they collide in whichever order.

Haven't thought about ARM myself. Do you happen to know how KVM protects
against running under (say) Xen there BTW?

Valentine

[Jan Kiszka]

Under Xen, the guests are not started in HYP mode but in SVC (OS) mode.
This will prevent KVM from initializing. But as Jailhouse loads after
the system booted, this does not help. We will once need an explicit
test to make this smoother.

BTW, I spread wrong information: there is no "load KVM after Jailhouse"
on ARM. KVM can only be built into the kernel, thus it will always load
first.

[Valentine Sinitsyn]

On 23.02.2015 13:49, Jan Kiszka wrote:
> Under Xen, the guests are not started in HYP mode but in SVC (OS) mode.
> This will prevent KVM from initializing. But as Jailhouse loads after
> the system booted, this does not help. We will once need an explicit
> test to make this smoother.

Okay, thanks for explaining.

> BTW, I spread wrong information: there is no "load KVM after Jailhouse"
> on ARM. KVM can only be built into the kernel, thus it will always load
> first.

Sounds like another reason to try squeezing a few extra #ifdefs into
driver code :) I mean, all of this will probably look hairy if we
account for all possible scenarios anyway.

Valentine

[Jan Kiszka]

Linux gained CR4 shadowing in 4.0: it now keeps a software copy of the
mm's CR4 state and only write updates on changes. But this also means it
will miss changes to this register when enabling or disabling Jailhouse.
That will cause sporadic access violations or false reporting of
virtualization-busy CPUs on Jailhouse enable.

Fix it by re-initializing the shadow on x86 after returning from the
hypervisor entry or the disable command. We are still protected from
interrupts at that point, so nothing can leak to other tasks.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

This would replace the VMXE-hiding patch. It seems to work, but nested
VMX is broken again and doesn't allow me to start qemu/kvm even without
any Jailhouse presence reliably right now. Need to recheck on real hw.

driver.c | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/driver.c b/driver.c
index bc6353d..86299d5 100644
--- a/driver.c
+++ b/driver.c
@@ -29,6 +29,7 @@
#include <linux/pci.h>
#include <asm/smp.h>
#include <asm/cacheflush.h>
+#include <asm/tlbflush.h>

#include "jailhouse.h"
#include <jailhouse/cell-config.h>
@@ -515,6 +516,11 @@ static void enter_hypervisor(void *info)
if (err)
error_code = err;

+#if defined(CONFIG_X86) && LINUX_VERSION_CODE >= KERNEL_VERSION(4,0,0)
+ /* on Intel, VMXE is now on - update the shadow */
+ cr4_init_shadow();
+#endif
+
atomic_inc(&call_done);
}

@@ -703,6 +709,11 @@ static void leave_hypervisor(void *info)
if (err)
error_code = err;

+#if defined(CONFIG_X86) && LINUX_VERSION_CODE >= KERNEL_VERSION(4,0,0)
+ /* on Intel, VMXE is now off - update the shadow */
+ cr4_init_shadow();
+#endif
+
atomic_inc(&call_done);
}

--
2.1.4

[Valentine Sinitsyn]

On 23.02.2015 15:28, Jan Kiszka wrote:
> This would replace the VMXE-hiding patch. It seems to work, but nested

Many thanks for trying out my proposal.

> VMX is broken again and doesn't allow me to start qemu/kvm even without
> any Jailhouse presence reliably right now. Need to recheck on real hw.

Yes, this sounds like a KVM issue to me.

Valentine

[Jan Kiszka]

Oops, dead code. Will remove before merge.