[RFC v1 01/42] driver: RISC-V: check for availability of h-extensions

[Ralf Ramsauer]

Like on all other architectures, check for the availability of
h-extensions in the driver to early prevent damage, if h-extensions are
not available.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---
driver/main.c | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/driver/main.c b/driver/main.c
index d13afeb6..ef1c2a90 100644
--- a/driver/main.c
+++ b/driver/main.c
@@ -444,6 +444,13 @@ static int jailhouse_cmd_enable(struct jailhouse_system __user *arg)
}
}
#endif
+#ifdef CONFIG_RISCV
+ if (!riscv_isa_extension_available(NULL, h)) {
+ pr_err("jailhouse: hypervisor extension not available\n");
+ err = -ENODEV;
+ goto error_put_module;
+ }
+#endif

/* Load hypervisor image */
err = request_firmware(&hypervisor, fw_name, jailhouse_dev);
--
2.36.1

[Ralf Ramsauer]

tl;dr: this is a first approach towards RISC-V support in Jailhouse. It works,
but there are still some open architectural discussions that have to be
addressed.

Hi,

this is the first RFC series for RISC-V support in Jailhouse. The series was
initially developed by Konrad Schwarz at SIEMENS in context of the SELENE
project [0], and later extended by Stefan an me (see Contributors below) at OTH
Regensburg.

The series is not yet ready to be integrated, but I think it's a good starting
point to have first discussions about the architectural aspects and directions
of the RISC-V port.

The series adds support for:
- "wip/new-cpu-config" (by Jan, rebased and ported by Ralf)
- RISC-V h-extensions (RV64 only)
- MMU support: SV39, SV48, SV57
- PLIC interrupt controller
- Multi-Core non-root cells
- Basic demo inmates
- Non-Root Linux support
- IVSHMEM (IRQs are implemented by virtualising the PLIC, sigh…)

Target platforms:
- Qemu virtual targets
- NOEL-V [1]

Architectural limitations:
- IRQs arrive in S-Mode, and must be reinjected to VS-Mode. This applies for
IPIs, Timers, and physical IRQs at the PLIC.
- The PLIC needs full virtualisation, as its design is uncapable for
virtualisation. Cf.: [2].
- IPIs are sent via SBI firmware. This requires double trapping (Jailhouse +
SBI), and manual reinjection on receiver site.
- IPIs can not be parameterised: there's only one single IPI. We have to
differentiate between management and guest IPIs in software. Sigh…


What is missing and should be discussed:
- APLIC support (and MSI support)
The PLIC does not support to be virtualised at all, and we should
probably add support for the APLIC soon. The question is, if Jailhouse
should still offer support for the PLIC: At the moment, only Qemu
supports the APLIC, and we don't have a real hardware target.

- Minor: missing support for jailhouse-config-check, support for
jailhouse-linux

- Prioritisation of vIRQs: they have lowest prio.
vIRQ support is only suppported for testing / demonstration at the
moment. Full virtualisation of the PLIC comes at a high cost, as
priorities of the vIRQs need to be respectes as well. As this level of
virtualisation is contrary to the concepts of Jailhouse, I deliberately
did not implement prioritisation of vIRQs.

- Stable CPU enumeration:
On Qemu, for example, the mapping from logical CPU ID to physical HART ID
differs each boot. Hence, Jailhouse's static enumeration may drift. This
is only a problem for jailhouse cell stats, as erroneous CPUs will be
addressed. We need to stabilise CPU enumeration. The driver will drop a
warning if CPU mapping differs.

- Some spots need better locking and testing. E.g., IPI handling needs better
locking.

- There are some overlaps with arm-common/arm64; see irq handling, bitops.
Should we factor those bits out?

Linux support:
- We need this single downstream patch [3]. Patches can be found on the branch
ralf/riscv-jailhouse/V10.

Contributors:
- Konrad Schwarz, Siemens (base support, first draft)
- Ralf Ramsauer, OTH (exception handling, control flow, reworked PLIC support)
- Stefan Huber, OTH (inmate support)
- Jan Kiszka, Siemes (wip/new-cpu-config)

Besides this patch series, there's a testbed for jailhouse on RISC-V that eases
debugging and development [4].

The series can also be found here [5].

Ralf

[0] https://www.selene-project.eu/
[1] https://www.gaisler.com/index.php/products/processors/noel-v
[2] https://github.com/riscv/riscv-plic-spec/blob/master/riscv-plic.adoc
[3] https://github.com/lfd/linux/commit/e4c207ca00395c0fec578ae9b79b6f6a615fddeb
[4] https://github.com/lfd/jailv-testbed
[5] https://github.com/lfd/jailhouse/tree/ralf/riscv/V8

Konrad Schwarz (1):
core: riscv: implement paging structures

Ralf Ramsauer (40):
driver: RISC-V: check for availability of h-extensions
driver: sysfs: prepare for exit accounting on RISC-V
core: riscv: define MMIO accessors
core: riscv: Add CSR definitions
core: riscv: introduce asm/bitops.h
core: riscv: add processor.h
core: riscv: Define percpu fields
core: riscv: introduce global sbi_ecall.h
core: riscv: introduce asm/sbi.h
core: riscv: add platform information
core: riscv: implement the hypercall interface
core: riscv: introduce cell-specific fields
core: riscv: add dbg-write helper
core: riscv: implement library routines
core: riscv: implement PCI support
core: riscv: paging: add headers
core: riscv: add asm-defines.c
core: riscv: introduce traps.c
core: riscv: Hypervisor entry code
core: riscv: implement arch_send_event
core: riscv: Add spinlock primitives
Documentation: Add some documentation
core: riscv: implement control.c
core: forward original HV memory location
core: riscv: implement setup.c
arm-common: export for_each_irqchip-macro
riscv: Add plic implementation
core: riscv: implement trap handlers
inmates: riscv: add timer demo
riscv: inmates: propagate hartid
riscv: inmates: implement Linux Loader
configs: riscv: qemu: add system configs
configs: riscv: qemu: add tiny demo config
configs: riscv: qemu: Add linux-demo
configs: riscv: noelv: Add system configs
configs: riscv: noelv: add tiny demo
configs: riscv: noelv: Add linux inmate config
core: riscv: plic: introduce vIRQ support
configs: riscv: add ivshmem-net devices
driver: riscv: Add vPCI support

Stefan Huber (1):
riscv: inmates: implement tiny-demo RISC-V

.gitignore | 1 +
Documentation/hypervisor-interfaces.txt | 16 +-
Documentation/memory-layout.txt | 20 +-
configs/riscv/dts/noelv-linux-inmate.dts | 147 ++++
configs/riscv/dts/qemu-linux-inmate.dts | 135 +++
configs/riscv/noelv-linux-demo.c | 111 +++
configs/riscv/noelv-tiny-demo.c | 63 ++
configs/riscv/noelv.c | 153 ++++
configs/riscv/qemu-linux-demo.c | 111 +++
configs/riscv/qemu-riscv64-tiny-demo.c | 63 ++
configs/riscv/qemu-riscv64.c | 228 ++++-
driver/main.c | 8 +
driver/pci.c | 33 +-
driver/sysfs.c | 15 +
hypervisor/arch/arm-common/irqchip.c | 5 -
hypervisor/arch/riscv/Kbuild | 3 +-
hypervisor/arch/riscv/asm-defines.c | 27 +
hypervisor/arch/riscv/control.c | 228 ++++-
hypervisor/arch/riscv/dbg-write.c | 14 +-
hypervisor/arch/riscv/entry.S | 349 +++++++-
hypervisor/arch/riscv/exception.S | 91 ++
hypervisor/arch/riscv/include/asm/bitops.h | 54 +-
hypervisor/arch/riscv/include/asm/cell.h | 13 +-
hypervisor/arch/riscv/include/asm/control.h | 13 +
hypervisor/arch/riscv/include/asm/csr64.h | 196 +++++
hypervisor/arch/riscv/include/asm/ivshmem.h | 1 +
.../arch/riscv/include/asm/jailhouse_header.h | 24 +-
hypervisor/arch/riscv/include/asm/mmio.h | 67 ++
hypervisor/arch/riscv/include/asm/paging.h | 119 ++-
.../arch/riscv/include/asm/paging_modes.h | 18 +
hypervisor/arch/riscv/include/asm/percpu.h | 39 +-
hypervisor/arch/riscv/include/asm/plic.h | 28 +
hypervisor/arch/riscv/include/asm/processor.h | 47 +-
hypervisor/arch/riscv/include/asm/sbi.h | 84 ++
hypervisor/arch/riscv/include/asm/setup.h | 14 +
hypervisor/arch/riscv/include/asm/spinlock.h | 52 +-
hypervisor/arch/riscv/ivshmem.c | 36 +-
hypervisor/arch/riscv/lib.c | 25 +-
hypervisor/arch/riscv/paging.c | 262 +++++-
hypervisor/arch/riscv/pci.c | 25 +-
hypervisor/arch/riscv/plic.c | 706 +++++++++++++++
hypervisor/arch/riscv/setup.c | 200 ++++-
hypervisor/arch/riscv/traps.c | 803 ++++++++++++++++++
hypervisor/include/jailhouse/control.h | 5 +
hypervisor/include/jailhouse/gen-defines.h | 8 +-
hypervisor/include/jailhouse/header.h | 7 +
hypervisor/include/jailhouse/mmio.h | 4 +
hypervisor/include/jailhouse/paging.h | 2 +-
include/arch/riscv/asm/jailhouse_hypercall.h | 68 +-
include/arch/riscv/asm/sbi_ecall.h | 96 +++
include/jailhouse/cell-config.h | 11 +-
include/jailhouse/console.h | 1 +
inmates/demos/riscv/Makefile | 20 +
inmates/demos/riscv/timer-demo.c | 57 ++
inmates/demos/riscv/tiny-demo.c | 19 +
inmates/lib/include/inmate_common.h | 1 +
inmates/lib/riscv/Makefile | 46 +
inmates/lib/riscv/Makefile.lib | 64 ++
inmates/lib/riscv/header.S | 76 ++
inmates/lib/riscv/include/inmate.h | 184 ++++
inmates/lib/riscv/inmate.lds.S | 77 ++
inmates/lib/riscv/irq.c | 107 +++
inmates/lib/riscv/printk.c | 61 ++
inmates/lib/riscv/setup.c | 45 +
inmates/lib/riscv/uart.c | 47 +
inmates/tools/riscv/Makefile | 19 +
inmates/tools/riscv/linux-loader.c | 25 +
pyjailhouse/config_parser.py | 17 +-
tools/jailhouse-config-check | 9 +-
69 files changed, 5584 insertions(+), 139 deletions(-)
create mode 100644 configs/riscv/dts/noelv-linux-inmate.dts
create mode 100644 configs/riscv/dts/qemu-linux-inmate.dts
create mode 100644 configs/riscv/noelv-linux-demo.c
create mode 100644 configs/riscv/noelv-tiny-demo.c
create mode 100644 configs/riscv/noelv.c
create mode 100644 configs/riscv/qemu-linux-demo.c
create mode 100644 configs/riscv/qemu-riscv64-tiny-demo.c
create mode 100644 hypervisor/arch/riscv/exception.S
create mode 100644 hypervisor/arch/riscv/include/asm/csr64.h
create mode 100644 hypervisor/arch/riscv/include/asm/plic.h
create mode 100644 hypervisor/arch/riscv/include/asm/sbi.h
create mode 100644 hypervisor/arch/riscv/include/asm/setup.h
create mode 100644 hypervisor/arch/riscv/plic.c
create mode 100644 hypervisor/arch/riscv/traps.c
create mode 100644 include/arch/riscv/asm/sbi_ecall.h
create mode 100644 inmates/demos/riscv/timer-demo.c
create mode 100644 inmates/demos/riscv/tiny-demo.c
create mode 100644 inmates/lib/riscv/Makefile.lib
create mode 100644 inmates/lib/riscv/header.S
create mode 100644 inmates/lib/riscv/include/inmate.h
create mode 100644 inmates/lib/riscv/inmate.lds.S
create mode 100644 inmates/lib/riscv/irq.c
create mode 100644 inmates/lib/riscv/printk.c
create mode 100644 inmates/lib/riscv/setup.c
create mode 100644 inmates/lib/riscv/uart.c
create mode 100644 inmates/tools/riscv/linux-loader.c

--
2.36.1

[Ralf Ramsauer]

Besides generic traps, we will have a bunch of RISC-V specific traps:

Platform handling:
- VMEXITS_SBI: Calls from the guest (VS-Mode) to the SBI firmware

IRQ handling:
- VMEXITS_TIMER: Timers will arrive in S-Mode
- VMEXITS_IPI: IPIs arrive in S-Mode
- VMEXITS_VIRQ: external IRQs arrive in S-Mode

Generally, using a simple PLIC, all IRQs will arrive in S-Mode.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

driver/sysfs.c | 15 +++++++++++++++
include/arch/riscv/asm/jailhouse_hypercall.h | 7 ++++++-
2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/driver/sysfs.c b/driver/sysfs.c
index 61b851cc..5ea1a8b0 100644
--- a/driver/sysfs.c
+++ b/driver/sysfs.c
@@ -163,6 +163,11 @@ JAILHOUSE_CPU_STATS_ATTR(vmexits_smccc, JAILHOUSE_CPU_STAT_VMEXITS_SMCCC);
#ifdef CONFIG_ARM
JAILHOUSE_CPU_STATS_ATTR(vmexits_cp15, JAILHOUSE_CPU_STAT_VMEXITS_CP15);
#endif
+#elif CONFIG_RISCV
+JAILHOUSE_CPU_STATS_ATTR(vmexits_sbi, JAILHOUSE_CPU_STAT_VMEXITS_SBI);
+JAILHOUSE_CPU_STATS_ATTR(vmexits_timer, JAILHOUSE_CPU_STAT_VMEXITS_TIMER);
+JAILHOUSE_CPU_STATS_ATTR(vmexits_ipi, JAILHOUSE_CPU_STAT_VMEXITS_IPI);
+JAILHOUSE_CPU_STATS_ATTR(vmexits_virq, JAILHOUSE_CPU_STAT_VMEXITS_VIRQ);
#endif

static struct attribute *cell_stats_attrs[] = {
@@ -188,6 +193,11 @@ static struct attribute *cell_stats_attrs[] = {
#ifdef CONFIG_ARM
&vmexits_cp15_cell_attr.kattr.attr,
#endif
+#elif CONFIG_RISCV
+ &vmexits_sbi_cell_attr.kattr.attr,
+ &vmexits_timer_cell_attr.kattr.attr,
+ &vmexits_ipi_cell_attr.kattr.attr,
+ &vmexits_virq_cell_attr.kattr.attr,
#endif
NULL
};
@@ -221,6 +231,11 @@ static struct attribute *cpu_stats_attrs[] = {
#ifdef CONFIG_ARM
&vmexits_cp15_cpu_attr.kattr.attr,
#endif
+#elif CONFIG_RISCV
+ &vmexits_sbi_cpu_attr.kattr.attr,
+ &vmexits_timer_cpu_attr.kattr.attr,
+ &vmexits_ipi_cpu_attr.kattr.attr,
+ &vmexits_virq_cpu_attr.kattr.attr,
#endif
NULL
};
diff --git a/include/arch/riscv/asm/jailhouse_hypercall.h b/include/arch/riscv/asm/jailhouse_hypercall.h
index 7d54047c..1a76ba2b 100644
--- a/include/arch/riscv/asm/jailhouse_hypercall.h
+++ b/include/arch/riscv/asm/jailhouse_hypercall.h
@@ -36,7 +36,12 @@
* THE POSSIBILITY OF SUCH DAMAGE.
*/

-#define JAILHOUSE_NUM_CPU_STATS JAILHOUSE_GENERIC_CPU_STATS
+/* CPU statistics, RISC-V-specific part */
+#define JAILHOUSE_CPU_STAT_VMEXITS_SBI JAILHOUSE_GENERIC_CPU_STATS
+#define JAILHOUSE_CPU_STAT_VMEXITS_TIMER JAILHOUSE_GENERIC_CPU_STATS + 1
+#define JAILHOUSE_CPU_STAT_VMEXITS_IPI JAILHOUSE_GENERIC_CPU_STATS + 2
+#define JAILHOUSE_CPU_STAT_VMEXITS_VIRQ JAILHOUSE_GENERIC_CPU_STATS + 3
+#define JAILHOUSE_NUM_CPU_STATS JAILHOUSE_GENERIC_CPU_STATS + 4

#ifndef __ASSEMBLY__

--
2.36.1

[Ralf Ramsauer]

GCC may emit 4x 1 Byte reads in case of our regular mmio_read32
accessor, when used in combination with -Os. Yes, I've seen it.

Define safe handlers to overcome this issue.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/mmio.h | 67 ++++++++++++++++++++++++
hypervisor/include/jailhouse/mmio.h | 4 ++
2 files changed, 71 insertions(+)

diff --git a/hypervisor/arch/riscv/include/asm/mmio.h b/hypervisor/arch/riscv/include/asm/mmio.h
index e69de29b..e19d0cec 100644
--- a/hypervisor/arch/riscv/include/asm/mmio.h
+++ b/hypervisor/arch/riscv/include/asm/mmio.h
@@ -0,0 +1,67 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+/*
+ * We need those definitions, as I saw the compiler emitting 4x 1B loads in
+ * case of mmio_read32.
+ */
+
+#define DEFINE_MMIO_READ
+#define DEFINE_MMIO_WRITE
+
+static inline void mmio_write8(volatile void *addr, u8 val)
+{
+ asm volatile("sb %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+static inline void mmio_write16(volatile void *addr, u16 val)
+{
+ asm volatile("sh %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+static inline void mmio_write32(volatile void *addr, u32 val)
+{
+ asm volatile("sw %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+static inline void mmio_write64(volatile void *addr, u64 val)
+{
+ asm volatile("sd %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+static inline u8 mmio_read8(const volatile void *addr)
+{
+ u8 val;
+ asm volatile("lb %0, 0(%1)" : "=r" (val) : "r" (addr));
+ return val;
+}
+
+static inline u16 mmio_read16(const volatile void *addr)
+{
+ u16 val;
+ asm volatile("lh %0, 0(%1)" : "=r" (val) : "r" (addr));
+ return val;
+}
+
+static inline u32 mmio_read32(const volatile void *addr)
+{
+ u32 val;
+ asm volatile("lw %0, 0(%1)" : "=r" (val) : "r" (addr));
+ return val;
+}
+
+static inline u64 mmio_read64(const volatile void *addr)
+{
+ u64 val;
+ asm volatile("ld %0, 0(%1)" : "=r" (val) : "r" (addr));
+ return val;
+}
diff --git a/hypervisor/include/jailhouse/mmio.h b/hypervisor/include/jailhouse/mmio.h
index aa85a089..f5ace9e9 100644
--- a/hypervisor/include/jailhouse/mmio.h
+++ b/hypervisor/include/jailhouse/mmio.h
@@ -32,6 +32,7 @@ struct cell;
* Define MMIO read accessor.
* @param size Access size.
*/
+#ifndef DEFINE_MMIO_READ
#define DEFINE_MMIO_READ(size) \
static inline u##size mmio_read##size(void *address) \
{ \
@@ -49,12 +50,14 @@ DEFINE_MMIO_READ(8)
DEFINE_MMIO_READ(16)
DEFINE_MMIO_READ(32)
DEFINE_MMIO_READ(64)
+#endif
/** @} */

/**
* Define MMIO write accessor.
* @param size Access size.
*/
+#ifndef DEFINE_MMIO_WRITE
#define DEFINE_MMIO_WRITE(size) \
static inline void mmio_write##size(void *address, u##size value) \
{ \
@@ -71,6 +74,7 @@ DEFINE_MMIO_WRITE(8)
DEFINE_MMIO_WRITE(16)
DEFINE_MMIO_WRITE(32)
DEFINE_MMIO_WRITE(64)
+#endif
/** @} */

/**
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/bitops.h | 54 ++++++++++++++++++----
1 file changed, 46 insertions(+), 8 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/bitops.h b/hypervisor/arch/riscv/include/asm/bitops.h
index 4b7e31b1..f62f382a 100644
--- a/hypervisor/arch/riscv/include/asm/bitops.h
+++ b/hypervisor/arch/riscv/include/asm/bitops.h
@@ -2,9 +2,11 @@

* Jailhouse, a Linux-based partitioning hypervisor

*
* Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

@@ -13,25 +15,61 @@
#ifndef _JAILHOUSE_ASM_BITOPS_H
#define _JAILHOUSE_ASM_BITOPS_H

+/*
+ * TBD: Introduce generic arch-independent bitops? This routine is shared with
+ * arm-common
+ */
static inline __attribute__((always_inline)) int
test_bit(unsigned int nr, const volatile unsigned long *addr)
{
- return 0;
+ return ((1UL << (nr % BITS_PER_LONG)) &
+ (addr[nr / BITS_PER_LONG])) != 0;
}

-static inline int atomic_test_and_set_bit(int nr, volatile unsigned long *addr)
+static inline long unsigned atomic_test_and_set_bit(int nr, volatile unsigned long *addr)
{
- return 0;
+ unsigned long res, mask;
+
+ mask = 1UL << (nr % BITS_PER_LONG);
+ asm volatile("amoor.w.aqrl %0, %2, %1"
+ : "=r"(res), "+A"(addr[nr / BITS_PER_LONG])
+ : "r" (mask)
+ : "memory");
+
+ return (res & mask) != 0;
}

-static inline unsigned long ffzl(unsigned long word)
+/*
+ * note: the RISC-V Bit Manipulation standard extension will undoubtedly
+ * have more performant implementations of these routines
+ * Returns the position of the least significant 1, MSB=31, LSB=0
+ * CAUTION: in contrast to POSIX ffs(), LSB = 0
+ */
+static inline unsigned long ffsl(unsigned long word)
{
- return 0;
+ register int result;
+
+ if (!word)
+ return BITS_PER_LONG;
+ result = BITS_PER_LONG - 1;
+
+ word = (word - 1) ^ word;
+ /* word now contains 0s followed by n 1s, if
+ n - 1 was the least significant set bit */
+
+ while (0 <= (long) word) {
+ word <<= 1;
+ --result;
+ }
+ return result;
}

-static inline unsigned long ffsl(unsigned long word)
+/* Returns the position of the least significant 0, MSB=31, LSB=0
+ * CAUTION: in contrast to the POSIX ffs(), LSB = 0
+ */
+static inline unsigned long ffzl (register unsigned long word)
{
- return 0;
+ return ffsl(~word);
}

#endif /* !_JAILHOUSE_ASM_BITOPS_H */
--
2.36.1

[Ralf Ramsauer]

RISC-V has a bunch of Control and Status Registers (CSRs). Add
definitions and accessors for those registers. We will need them later.

These definitions are copied over from the Linux kernel.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/csr64.h | 196 ++++++++++++++++++++++
1 file changed, 196 insertions(+)
create mode 100644 hypervisor/arch/riscv/include/asm/csr64.h

diff --git a/hypervisor/arch/riscv/include/asm/csr64.h b/hypervisor/arch/riscv/include/asm/csr64.h
new file mode 100644
index 00000000..15f67213
--- /dev/null
+++ b/hypervisor/arch/riscv/include/asm/csr64.h
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+/*
+ * Copyright (C) 2015 Regents of the University of California
+ */
+
+/* This file primarily bases upon definitions from the Linux Kernel */
+
+#ifdef __ASSEMBLY__
+#define _AC(X,Y) X
+#define _AT(T,X) X
+#else
+#define __AC(X,Y) (X##Y)
+#define _AC(X,Y) __AC(X,Y)
+#define _AT(T,X) ((T)(X))
+#endif
+
+/* Definition for CSR register numbers, that are not available for RISC-V
+ * binutils < 2.38
+ */
+#define CSR_SATP 0x180
+#define CSR_VSSTATUS 0x200
+#define CSR_VSIE 0x204
+#define CSR_VSTVEC 0x205
+#define CSR_VSSCRATCH 0x240
+#define CSR_VSEPC 0x241
+#define CSR_VSCAUSE 0x242
+#define CSR_VSTVAL 0x243
+#define CSR_VSATP 0x280
+#define CSR_HSTATUS 0x600
+#define CSR_HEDELEG 0x602
+#define CSR_HIDELEG 0x603
+#define CSR_HIE 0x604
+#define CSR_HTIMEDELTA 0x605
+#define CSR_HCOUNTEREN 0x606
+#define CSR_HGEIE 0x607
+#define CSR_HTVAL 0x643
+#define CSR_HIP 0x644
+#define CSR_HVIP 0x645
+#define CSR_HTINST 0x64a
+#define CSR_HGATP 0x680
+
+/* Status register flags */
+#define SR_SIE _AC(0x00000002, UL) /* Supervisor Interrupt Enable */
+#define SR_MIE _AC(0x00000008, UL) /* Machine Interrupt Enable */
+#define SR_SPIE _AC(0x00000020, UL) /* Previous Supervisor IE */
+#define SR_MPIE _AC(0x00000080, UL) /* Previous Machine IE */
+#define SR_SPP _AC(0x00000100, UL) /* Previously Supervisor */
+#define SR_MPP _AC(0x00001800, UL) /* Previously Machine */
+#define SR_SUM _AC(0x00040000, UL) /* Supervisor User Memory Access */
+
+#define SR_FS _AC(0x00006000, UL) /* Floating-point Status */
+#define SR_FS_OFF _AC(0x00000000, UL)
+#define SR_FS_INITIAL _AC(0x00002000, UL)
+#define SR_FS_CLEAN _AC(0x00004000, UL)
+#define SR_FS_DIRTY _AC(0x00006000, UL)
+
+#define SR_XS _AC(0x00018000, UL) /* Extension Status */
+#define SR_XS_OFF _AC(0x00000000, UL)
+#define SR_XS_INITIAL _AC(0x00008000, UL)
+#define SR_XS_CLEAN _AC(0x00010000, UL)
+#define SR_XS_DIRTY _AC(0x00018000, UL)
+
+#define SR_VS_DIRTY _AC(0x00000600, UL)
+
+#define SR_UXL_64 _AC(0x200000000, UL)
+
+#define SR_SD _AC(0x8000000000000000, UL) /* FS/XS dirty */
+
+/* Exception causes */
+#define EXC_INST_MISALIGNED 0
+#define EXC_INST_ACCESS 1
+#define EXC_INST_ILLEGAL 2
+#define EXC_BREAKPOINT 3
+#define EXC_LOAD_ACCESS_MISALIGNED 4
+#define EXC_LOAD_ACCESS 5
+#define EXC_AMO_ADDRESS_MISALIGNED 6
+#define EXC_STORE_ACCESS 7
+#define EXC_SYSCALL 8
+#define EXC_HYPERVISOR_SYSCALL 9
+#define EXC_SUPERVISOR_SYSCALL 10
+#define EXC_INST_PAGE_FAULT 12
+#define EXC_LOAD_PAGE_FAULT 13
+#define EXC_STORE_PAGE_FAULT 15
+#define EXC_INST_GUEST_PAGE_FAULT 20
+#define EXC_LOAD_GUEST_PAGE_FAULT 21
+#define EXC_VIRTUAL_INST_FAULT 22
+#define EXC_STORE_GUEST_PAGE_FAULT 23
+
+/* Exception cause high bit - is an interrupt if set */
+#define CAUSE_IRQ_FLAG (_AC(1, UL) << (__riscv_xlen - 1))
+
+/* Interrupt causes (minus the high bit) */
+#define IRQ_S_SOFT 1
+#define IRQ_VS_SOFT 2
+#define IRQ_M_SOFT 3
+#define IRQ_S_TIMER 5
+#define IRQ_VS_TIMER 6
+#define IRQ_M_TIMER 7
+#define IRQ_S_EXT 9
+#define IRQ_VS_EXT 10
+#define IRQ_M_EXT 11
+
+/* VSIP & HVIP relation */
+#define VSIP_TO_HVIP_SHIFT (IRQ_VS_SOFT - IRQ_S_SOFT)
+
+/* IE/IP (Supervisor/Machine Interrupt Enable/Pending) flags */
+#define IE_SIE (_AC(0x1, UL) << IRQ_S_SOFT)
+#define IE_TIE (_AC(0x1, UL) << IRQ_S_TIMER)
+#define IE_EIE (_AC(0x1, UL) << IRQ_S_EXT)
+
+/* SATP flags */
+#define SATP_PPN _AC(0x00000FFFFFFFFFFF, UL)
+#define SATP_MODE_39 _AC(0x8000000000000000, UL)
+#define SATP_MODE_48 _AC(0x9000000000000000, UL)
+#define SATP_ASID_BITS 16
+#define SATP_ASID_SHIFT 44
+#define SATP_ASID_MASK _AC(0xFFFF, UL)
+
+/* HSTATUS flags */
+#define HSTATUS_VSXL _AC(0x300000000, UL)
+#define HSTATUS_VSXL_SHIFT 32
+
+#define HSTATUS_VTSR _AC(0x00400000, UL)
+#define HSTATUS_VTW _AC(0x00200000, UL)
+#define HSTATUS_VTVM _AC(0x00100000, UL)
+#define HSTATUS_VGEIN _AC(0x0003f000, UL)
+#define HSTATUS_VGEIN_SHIFT 12
+#define HSTATUS_HU _AC(0x00000200, UL)
+#define HSTATUS_SPVP _AC(0x00000100, UL)
+#define HSTATUS_SPV _AC(0x00000080, UL)
+#define HSTATUS_GVA _AC(0x00000040, UL)
+#define HSTATUS_VSBE _AC(0x00000020, UL)
+
+#define HGATP_VMID_SHIFT 22
+#define HGATP_VMID_WIDTH 7
+
+#define SCOUNTEREN_TM 0x00000002
+
+#ifndef __ASSEMBLY__
+
+#include <jailhouse/string.h>
+
+#define csr_from_csr(dst, src) \
+({ \
+ register unsigned long __tmp; \
+ __asm__ __volatile__("csrr %0, " __stringify(src) "\n" \
+ "csrw " __stringify(dst) ", %0\n" \
+ : "=r"(__tmp) : \
+ : "memory" \
+ ); \
+})
+
+#define csr_read(csr) \
+({ \
+ register unsigned long __v; \
+ __asm__ __volatile__ ("csrr %0, " __stringify(csr) \
+ : "=r" (__v) : \
+ : "memory"); \
+ __v; \
+})
+
+#define csr_write(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrw " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+#define csr_clear(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrc " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+#define csr_set(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrs " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+#define csr_swap(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrrw %0, " __stringify(csr) ", %1" \
+ : "=r" (__v) : "rK" (__v) \
+ : "memory"); \
+ __v; \
+})
+
+#endif /* __ASSEMBLY__ */
--
2.36.1

[Ralf Ramsauer]

Add processor.h. We have 32x 64-Bit registers, x0-x32. x0 is the zero
register. In our case, let's place the program counter at its location.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/processor.h | 47 ++++++++++++++++++-
1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/processor.h b/hypervisor/arch/riscv/include/asm/processor.h
index 8d4b1c60..f839d1f4 100644
--- a/hypervisor/arch/riscv/include/asm/processor.h
+++ b/hypervisor/arch/riscv/include/asm/processor.h

@@ -2,9 +2,11 @@
* Jailhouse, a Linux-based partitioning hypervisor
*
* Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

@@ -16,18 +18,59 @@
#include <jailhouse/types.h>

union registers {
-};
+ struct {
+ unsigned long pc; /* x0 is zero register, in our case it's PC */
+ unsigned long ra;
+ unsigned long sp;
+ unsigned long gp;
+ unsigned long tp;
+ unsigned long t0;
+ unsigned long t1;
+ unsigned long t2;
+ unsigned long s0;
+ unsigned long s1;
+ unsigned long a0;
+ unsigned long a1;
+ unsigned long a2;
+ unsigned long a3;
+ unsigned long a4;
+ unsigned long a5;
+ unsigned long a6;
+ unsigned long a7;
+ unsigned long s2;
+ unsigned long s3;
+ unsigned long s4;
+ unsigned long s5;
+ unsigned long s6;
+ unsigned long s7;
+ unsigned long s8;
+ unsigned long s9;
+ unsigned long s10;
+ unsigned long s11;
+ unsigned long t3;
+ unsigned long t4;
+ unsigned long t5;
+ unsigned long t6;
+ };
+ struct riscv_raw_registers {
+ unsigned long x[32];
+ } raw;
+} __attribute__ ((__aligned__ (1 << (7 - 3) /* bits/byte */)));
+/* RISC-V ABI requires 128-bit stack alignment */

static inline void cpu_relax(void)
{
+ asm volatile("" : : : "memory");
}

static inline void memory_barrier(void)
{
+ asm volatile("fence rw, rw" : : : "memory");
}

static inline void memory_load_barrier(void)
{
+ asm volatile("fence r, rw" : : : "memory");
}

#endif /* !_JAILHOUSE_ASM_PROCESSOR_H */
--
2.36.1

[Ralf Ramsauer]

This routines will later also be shared with inmates, so implement it
globally.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

include/arch/riscv/asm/sbi_ecall.h | 96 ++++++++++++++++++++++++++++++
1 file changed, 96 insertions(+)
create mode 100644 include/arch/riscv/asm/sbi_ecall.h

diff --git a/include/arch/riscv/asm/sbi_ecall.h b/include/arch/riscv/asm/sbi_ecall.h
new file mode 100644
index 00000000..cc0ce2dc
--- /dev/null
+++ b/include/arch/riscv/asm/sbi_ecall.h
@@ -0,0 +1,96 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ *
+ * Alternatively, you can use or redistribute this file under the following
+ * BSD license:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* see https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc */
+
+#ifndef _SBI_ECALL_H
+#define _SBI_ECALL_H
+
+#define SBI_SUCCESS 0
+#define SBI_ERR_FAILED -1
+#define SBI_ERR_NOT_SUPPORTED -2
+#define SBI_ERR_INVALID_PARAM -3
+#define SBI_ERR_DENIED -4
+#define SBI_ERR_INVALID_ADDRESS -5
+#define SBI_ERR_ALREADY_AVAILABLE -6
+
+#define SBI_EXT_TIME 0x54494D45
+#define SBI_EXT_TIME_SET_TIMER 0x0
+
+#ifndef __ASSEMBLY__
+
+struct sbiret {
+ long error;
+ long value;
+};
+
+static inline struct sbiret sbi_ecall(int ext, int fid, unsigned long arg0,
+ unsigned long arg1, unsigned long arg2,
+ unsigned long arg3, unsigned long arg4,
+ unsigned long arg5)
+{
+ struct sbiret ret;
+
+ register unsigned long a0 asm ("a0") = arg0;
+ register unsigned long a1 asm ("a1") = arg1;
+ register unsigned long a2 asm ("a2") = arg2;
+ register unsigned long a3 asm ("a3") = arg3;
+ register unsigned long a4 asm ("a4") = arg4;
+ register unsigned long a5 asm ("a5") = arg5;
+ register unsigned long a6 asm ("a6") = fid;
+ register unsigned long a7 asm ("a7") = ext;
+ asm volatile ("ecall"
+ : "+r" (a0), "+r" (a1)
+ : "r" (a2), "r" (a3), "r" (a4),
+ "r" (a5), "r" (a6), "r" (a7)
+ : "memory");
+ ret.error = a0;
+ ret.value = a1;
+
+ return ret;
+}
+
+static inline struct sbiret sbi_set_timer(unsigned long stime_value)
+{
+ return sbi_ecall(SBI_EXT_TIME, SBI_EXT_TIME_SET_TIMER, stime_value,
+ 0, 0, 0, 0, 0);
+}
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* _SBI_ECALL_H */
--
2.36.1

[Ralf Ramsauer]

It's pretty easy on RISC-V: All platform specific confi information that
we need is the PLIC.

Base/size fields are obvious, max_irq defines the highest possible IRQ
on the controller. max_priority the highest possible priority of an IRQ.

hart_to_context is a map that maps physical HART IDs to PLIC Contexts.

On regular RISC-V platforms, M and S-Mode of a HART are assigned to one
specific PLIC Context. For example:

PLIC Ctx 0 -> Hart 0, M-Mode
PLIC Ctx 1 -> Hart 0, S-Mode
PLIC Ctx 2 -> Hart 1, M-Mode
PLIC Ctx 3 -> Hart 1, S-Mode
PLIC Ctx 4 -> Hart 2, M-Mode
...

This is also how QEMU implements the PLIC. However, there are other
implementations out there, like the NOEL-V.

There, we have:
PLIC Ctx 0 -> Hart 0, M-Mode
PLIC Ctx 1 -> Hart 0, S-Mode
PLIC Ctx 2 -> ?
PLIC Ctx 3 -> ?
PLIC Ctx 4 -> Hart 1, M-Mode
PLIC Ctx 5 -> Hart 1, S-Mode
PLIC Ctx 6 -> ?
...

So we use the hart_to_context map to assign a hart to a Context ID.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

include/jailhouse/cell-config.h | 11 ++++++++++-
pyjailhouse/config_parser.py | 17 ++++++++++++++---
tools/jailhouse-config-check | 9 ++++++++-
3 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/include/jailhouse/cell-config.h b/include/jailhouse/cell-config.h
index 124373d5..c90a649d 100644
--- a/include/jailhouse/cell-config.h
+++ b/include/jailhouse/cell-config.h
@@ -50,7 +50,7 @@
* Incremented on any layout or semantic change of system or cell config.
* Also update formats and HEADER_REVISION in pyjailhouse/config_parser.py.
*/
-#define JAILHOUSE_CONFIG_REVISION 14
+#define JAILHOUSE_CONFIG_REVISION 15

#define JAILHOUSE_CELL_NAME_MAXLEN 31

@@ -357,6 +357,15 @@ struct jailhouse_system {
u64 gicv_base;
u64 gicr_base;
} __attribute__((packed)) arm;
+ struct {
+ struct {
+ u64 base_address;
+ u32 size;
+ u16 max_irq;
+ u32 max_priority;
+ s16 hart_to_context[32];
+ } __attribute__((packed)) plic;
+ } __attribute__((packed)) riscv;
} __attribute__((packed));
} __attribute__((packed)) platform_info;
struct jailhouse_cell_desc root_cell;
diff --git a/pyjailhouse/config_parser.py b/pyjailhouse/config_parser.py
index fc2158e1..37d8e039 100644
--- a/pyjailhouse/config_parser.py
+++ b/pyjailhouse/config_parser.py
@@ -19,7 +19,7 @@ import struct
from .extendedenum import ExtendedEnum

# Keep the whole file in sync with include/jailhouse/cell-config.h.
-_CONFIG_REVISION = 14
+_CONFIG_REVISION = 15


def flag_str(enum_class, value, separator=' | '):
@@ -245,6 +245,8 @@ class SystemConfig:
_NUM_IOMMUS = 8
_ARCH_ARM_FORMAT = '=BB2xQQQQQ'
_ARCH_X86_FORMAT = '=HBxIII28x'
+ _ARCH_RISCV_FORMAT = '=QIHI'
+ _ARCH_RISCV_FORMAT_HTC = '=32H'

def __init__(self, data, arch):
self.data = data
@@ -294,8 +296,17 @@ class SystemConfig:
self.x86_tsc_khz,
self.x86_apic_khz) = \
struct.unpack_from(self._ARCH_X86_FORMAT, self.data[offs:])
-
- offs += struct.calcsize(self._ARCH_ARM_FORMAT)
+ elif arch == 'riscv64':
+ (self.riscv_plic_base_address,
+ self.riscv_plic_size,
+ self.riscv_plic_max_irq,
+ self.riscv_plic_max_priority) = \
+ self.riscv_plic_hart_to_context = \
+ struct.unpack_from(self._ARCH_RISCV_FORMAT, self.data[offs:])
+ self.riscv_plic_hart_to_context = \
+ struct.unpack_from(self._ARCH_RISCV_FORMAT_HTC, self.data[offs:])
+ offs += struct.calcsize(self._ARCH_RISCV_FORMAT)
+ offs += struct.calcsize(self._ARCH_RISCV_FORMAT_HTC)
except struct.error:
raise RuntimeError('Not a root cell configuration')

diff --git a/tools/jailhouse-config-check b/tools/jailhouse-config-check
index d6ea7079..13fd6ff7 100755
--- a/tools/jailhouse-config-check
+++ b/tools/jailhouse-config-check
@@ -61,9 +61,11 @@ if not arch:
arch = 'arm'
elif arch_str == 'aarch64':
arch = 'arm64'
+ elif arch_str == 'riscv64':
+ arch = 'riscv64'
else:
arch = None
-if not arch in ('x86', 'arm', 'arm64'):
+if not arch in ('x86', 'arm', 'arm64', "riscv64"):
print('Unsupported architecture', file=sys.stderr)
exit(1)

@@ -188,6 +190,11 @@ elif arch == 'x86':
for irqchip in root_cell.irqchips:
arch_resources.append(ResourceRegion(irqchip.address, 0x1000,
"IOAPIC"))
+elif arch in ('riscv64'):
+ arch_resources = []
+ arch_resources.append(ResourceRegion(sysconfig.riscv_plic_base_address,
+ sysconfig.riscv_plic_size. "PLIC"))
+
for cell in cells:
for mem in cell.memory_regions:
idx = cell.memory_regions.index(mem)
--
2.36.1

[Ralf Ramsauer]

Add helpers for straight-forward human-readable SBI calls.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/sbi.h | 84 +++++++++++++++++++++++++
1 file changed, 84 insertions(+)
create mode 100644 hypervisor/arch/riscv/include/asm/sbi.h

diff --git a/hypervisor/arch/riscv/include/asm/sbi.h b/hypervisor/arch/riscv/include/asm/sbi.h
new file mode 100644
index 00000000..ee63bbf2
--- /dev/null
+++ b/hypervisor/arch/riscv/include/asm/sbi.h
@@ -0,0 +1,84 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:

+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+/* see https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc */
+

+#ifndef _SBI_H
+#define _SBI_H
+
+#include <asm/sbi_ecall.h>
+
+#define RISCV_SBI_HART_STATE_STARTED 0
+#define RISCV_SBI_HART_STATE_STOPPED 1
+#define RISCV_SBI_HART_START_REQUEST_PENDING 2
+#define RISCV_SBI_HART_STOP_REQUEST_PENDING 3
+
+#define SBI_EXT_BASE 0x10
+#define SBI_EXT_BASE_GET_SPEC_VERSION 0
+#define SBI_EXT_BASE_GET_IMP_ID 1
+#define SBI_EXT_BASE_GET_IMP_VERSION 2
+#define SBI_EXT_BASE_PROBE_EXT 3
+#define SBI_EXT_BASE_GET_MVENDORID 4
+#define SBI_EXT_BASE_GET_MARCHID 5
+#define SBI_EXT_BASE_GET_MIMPID 6
+
+#define SBI_EXT_0_1_SET_TIMER 0x0
+#define SBI_EXT_0_1_CONSOLE_PUTCHAR 0x1
+#define SBI_EXT_0_1_CONSOLE_GETCHAR 0x2
+
+#define SBI_EXT_SPI 0x735049
+#define SBI_EXT_IPI_SEND_IPI 0x0
+
+#define SBI_EXT_RFENCE 0x52464E43
+#define SBI_REMOTE_FENCE_I 0
+#define SBI_REMOTE_SFENCE_VMA 1
+#define SBI_REMOTE_SFENCE_VMA_ASID 2
+#define SBI_REMOTE_HFENCE_GVMA_VMID 3
+#define SBI_REMOTE_HFENCE_GVMA 4
+#define SBI_REMOTE_HFENCE_VVMA_ASID 5
+#define SBI_REMOTE_HFENCE_VVMA 6
+
+#define SBI_EXT_HSM 0x48534D
+#define SBI_EXT_HSM_HART_START 0
+#define SBI_EXT_HSM_HART_STOP 1
+#define SBI_EXT_HSM_HART_STATUS 2
+#define SBI_EXT_HSM_HART_SUSPEND 3
+
+#ifndef __ASSEMBLY__
+
+static inline struct sbiret sbi_send_ipi(unsigned long hart_mask,
+ unsigned long hart_mask_base)
+{
+ return sbi_ecall(SBI_EXT_SPI, SBI_EXT_IPI_SEND_IPI,
+ hart_mask, hart_mask_base, 0, 0, 0, 0);
+}
+
+static inline void sbi_console_putchar_legacy0_1(int ch)
+{
+ sbi_ecall(SBI_EXT_0_1_CONSOLE_PUTCHAR, 0, ch, 0, 0, 0, 0, 0);
+}
+
+static inline struct sbiret sbi_console_getchar_legacy_0_1(void)
+{
+ return sbi_ecall(SBI_EXT_0_1_CONSOLE_GETCHAR, 0, 0, 0, 0, 0, 0, 0);
+}
+
+static inline struct sbiret sbi_hart_stop(void)
+{
+ return sbi_ecall(SBI_EXT_HSM, SBI_EXT_HSM_HART_STOP, 0, 0, 0, 0, 0, 0);

+}
+
+#endif /* !__ASSEMBLY__ */
+

+#endif /* _SBI_H */
--
2.36.1

[Ralf Ramsauer]

On RISC-V, the logical CPU ID should be the same as Linux's ID. A
logical CPU ID is mapped to a HART, the physical ID (phys_id).

All communication with the SBI interface require specification of
physical HART IDs.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/control.c | 7 -------
hypervisor/arch/riscv/lib.c | 25 +++++++++++++++++++++----
2 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/hypervisor/arch/riscv/control.c b/hypervisor/arch/riscv/control.c
index ab0bb449..b848994b 100644
--- a/hypervisor/arch/riscv/control.c
+++ b/hypervisor/arch/riscv/control.c
@@ -69,10 +69,3 @@ void arch_park_cpu(unsigned int cpu_id)
void arch_send_event(struct public_per_cpu *target_data)
{
}
-
-// Likely misplaced
-#include <jailhouse/processor.h>
-unsigned int cpu_by_phys_processor_id(u64 phys_id)
-{
- return -1;
-}
diff --git a/hypervisor/arch/riscv/lib.c b/hypervisor/arch/riscv/lib.c
index 5fe45bc7..2ea1d5b6 100644
--- a/hypervisor/arch/riscv/lib.c
+++ b/hypervisor/arch/riscv/lib.c
@@ -1,19 +1,36 @@
/*

* Jailhouse, a Linux-based partitioning hypervisor

*
- * Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

*/

+#include <jailhouse/control.h>
+#include <jailhouse/printk.h>
#include <jailhouse/processor.h>
+#include <jailhouse/percpu.h>

-// Check if it be moved elsewhere
unsigned long phys_processor_id(void)
{
- return 0;
+ return this_cpu_public()->phys_id;
+}
+
+unsigned int cpu_by_phys_processor_id(u64 phys)
+{
+ const struct jailhouse_cpu *map = jailhouse_cell_cpus(root_cell.config);
+ unsigned int num_cpus = root_cell.config->num_cpus;
+ unsigned int cpu;
+
+ for (cpu = 0; cpu < num_cpus; cpu++, map++)
+ if (map->phys_id == phys)
+ return cpu;
+
+ /* this should never happen */
+ panic_printk("FATAL: unknown HART ID %llu\n", phys);
+ panic_stop();
}
--
2.36.1

[Ralf Ramsauer]

Nothing special here.

SBIs are parameterised bei an "Extension ID", which is typically 3-4 ASCII
characters, 'TIME', 'HSM', 'IPI', 'RFNC', …

Let's pray that no one else will implement 'JHOU' as official EID.

Then there are FIDs within the EID - Function IDs. For Jailhouse
Hypercalls, use FID = 'JHOU' and EID shall represent the hypercall
number.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

include/arch/riscv/asm/jailhouse_hypercall.h | 61 ++++++++++++++++----
1 file changed, 49 insertions(+), 12 deletions(-)

diff --git a/include/arch/riscv/asm/jailhouse_hypercall.h b/include/arch/riscv/asm/jailhouse_hypercall.h
index 1a76ba2b..1c0d80c6 100644
--- a/include/arch/riscv/asm/jailhouse_hypercall.h
+++ b/include/arch/riscv/asm/jailhouse_hypercall.h
@@ -1,10 +1,10 @@

/*
* Jailhouse, a Linux-based partitioning hypervisor
*
- * Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

@@ -36,6 +36,8 @@

* THE POSSIBILITY OF SUCH DAMAGE.

*/

+#include "sbi_ecall.h"
+

/* CPU statistics, RISC-V-specific part */

#define JAILHOUSE_CPU_STAT_VMEXITS_SBI JAILHOUSE_GENERIC_CPU_STATS
#define JAILHOUSE_CPU_STAT_VMEXITS_TIMER JAILHOUSE_GENERIC_CPU_STATS + 1
@@ -43,31 +45,66 @@
#define JAILHOUSE_CPU_STAT_VMEXITS_VIRQ JAILHOUSE_GENERIC_CPU_STATS + 3
#define JAILHOUSE_NUM_CPU_STATS JAILHOUSE_GENERIC_CPU_STATS + 4

+#define ASCII32(C, POS) (C << ((POS) * 8))
+#define JAILHOUSE_EID \
+ ASCII32('J', 3) | \
+ ASCII32('H', 2) | \
+ ASCII32('O', 1) | \
+ ASCII32('U', 0)
+
#ifndef __ASSEMBLY__

-static inline __u32 jailhouse_call(__u32 num)
+struct jailhouse_comm_region {
+ COMM_REGION_GENERIC_HEADER;
+} __attribute__((packed));
+
+static inline unsigned long jailhouse_call(unsigned long num)
{
- return -ENOSYS;

+ struct sbiret ret;
+

+ ret = sbi_ecall(JAILHOUSE_EID, num, 0, 0, 0, 0, 0, 0);
+
+ return ret.error;
}

-static inline __u32 jailhouse_call_arg1(__u32 num, __u32 arg1)
+static inline unsigned long
+jailhouse_call_arg1(unsigned long num, unsigned long arg1)
{
- return -ENOSYS;

+ struct sbiret ret;
+

+ ret = sbi_ecall(JAILHOUSE_EID, num, arg1, 0, 0, 0, 0, 0);
+
+ return ret.error;
}

-static inline __u32 jailhouse_call_arg2(__u32 num, __u32 arg1, __u32 arg2)
+static inline unsigned long
+jailhouse_call_arg2(unsigned long num, unsigned long arg1, unsigned long arg2)
{
- return -ENOSYS;
-}
+ struct sbiret ret;

-struct jailhouse_comm_region {
- COMM_REGION_GENERIC_HEADER;
-} __attribute__((packed));
+ ret = sbi_ecall(JAILHOUSE_EID, num, arg1, arg2, 0, 0, 0, 0);
+
+ return ret.error;
+}

static inline void
jailhouse_send_msg_to_cell(struct jailhouse_comm_region *comm_region,
__u32 msg)
{
+ comm_region->reply_from_cell = JAILHOUSE_MSG_NONE;
+ /* ensure reply was cleared before sending new message */

+ asm volatile("fence rw, rw" : : : "memory");

+ comm_region->msg_to_cell = msg;

+}
+
+static inline void

+jailhouse_send_reply_from_cell(struct jailhouse_comm_region *comm_region,
+ __u32 reply)
+{
+ comm_region->msg_to_cell = JAILHOUSE_MSG_NONE;
+ /* ensure message was cleared before sending reply */

+ asm volatile("fence rw, rw" : : : "memory");

+ comm_region->reply_from_cell = reply;
}

#endif /* !__ASSEMBLY__ */
--
2.36.1

[Ralf Ramsauer]

Add RISC-V cell-specific fields.

Unlike ARM, for example, the PLIC doesn't support hardware injection of
external IRQs. So we need to do it in software, sigh...

Prepare the fields that are needed for software injection. The
irq_bitmap, just like on ARM, denotes physical IRQs that the cell is
allowed to manage.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/cell.h | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/cell.h b/hypervisor/arch/riscv/include/asm/cell.h
index 63ba43ce..58412414 100644
--- a/hypervisor/arch/riscv/include/asm/cell.h
+++ b/hypervisor/arch/riscv/include/asm/cell.h
@@ -2,9 +2,11 @@

* Jailhouse, a Linux-based partitioning hypervisor
*

* Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>

+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

@@ -13,12 +15,14 @@
#ifndef _JAILHOUSE_ASM_CELL_H
#define _JAILHOUSE_ASM_CELL_H

+#include <jailhouse/paging.h>
#include <jailhouse/types.h>
-
-// this shouldn't be here
-#include <jailhouse/cell-config.h>
+#include <asm/plic.h>

struct arch_cell {
+ struct paging_structures mm;
+
+ u32 irq_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)];
};

#endif /* !_JAILHOUSE_ASM_CELL_H */
--
2.36.1

[Ralf Ramsauer]

Nothing special, but two important things: the ipi_cause and the hsm
field.

There's only one single IPI, and we can't differentiate between
different IPI causes. That makes things hard: if an IPI arrives at a
HART, we don't know if it is a management IPI or if the IPI is for our
guest. Hence, store the cause of the ipi for a CPU in the public per CPU
fields.

Simple protocol:
- The sender must wait until ipi_cause is IPI_CAUSE_NONE.
- The sender must set the cause before sending the IPI.
- The receiver must clear the cause before after finishing the IPI
handling

HSM stands for "HART State Management". Refer to
https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state-management-extension-eid-0x48534d-hsm

We will understand and support HSM for CPU On/Offlining.

For Onlining CPU, we need some data, which is stored in the hsm field.
Besides that, the state of the current hart is stored in hart_state.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/percpu.h | 37 ++++++++++++++++++----
hypervisor/arch/riscv/include/asm/plic.h | 18 +++++++++++
2 files changed, 49 insertions(+), 6 deletions(-)
create mode 100644 hypervisor/arch/riscv/include/asm/plic.h

diff --git a/hypervisor/arch/riscv/include/asm/percpu.h b/hypervisor/arch/riscv/include/asm/percpu.h
index fce6c052..4eda15b6 100644
--- a/hypervisor/arch/riscv/include/asm/percpu.h
+++ b/hypervisor/arch/riscv/include/asm/percpu.h
@@ -2,22 +2,47 @@

* Jailhouse, a Linux-based partitioning hypervisor
*
* Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

*/

-// this shouldn't be here

+#include <asm/plic.h>
#include <asm/processor.h>
+#include <asm/spinlock.h>

-#define STACK_SIZE PAGE_SIZE
+#define STACK_SIZE (PAGE_SIZE << 2)
+
+enum sbi_hart_state {
+ STARTED = 0,
+ STOPPED = 1,
+ START_PENDING = 2,
+ STOP_PENDING = 3,
+ SUSPENDED = 4,
+ SUSPEND_PENDING = 5,
+ RESUME_PENDING = 6,
+};

#define ARCH_PUBLIC_PERCPU_FIELDS \
+ unsigned long phys_id; \
+ enum { \
+ IPI_CAUSE_NONE, \
+ IPI_CAUSE_GUEST, \
+ IPI_CAUSE_MGMT, \
+ } ipi_cause; \
spinlock_t control_lock; \
- ;
+ struct { \
+ enum sbi_hart_state state; \
+ unsigned long start_addr; \
+ unsigned long opaque; \
+ } hsm; \
+ bool wait_for_power_on; \
+ bool reset; \
+ bool park;

-#define ARCH_PERCPU_FIELDS \
- ;
+#define ARCH_PERCPU_FIELDS
diff --git a/hypervisor/arch/riscv/include/asm/plic.h b/hypervisor/arch/riscv/include/asm/plic.h
new file mode 100644
index 00000000..04cdfa63
--- /dev/null
+++ b/hypervisor/arch/riscv/include/asm/plic.h
@@ -0,0 +1,18 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef _PLIC_H
+#define _PLIC_H
+
+#define PLIC_MAX_IRQS 1024
+
+#endif /* _PLIC_H */
--
2.36.1

[Ralf Ramsauer]

Basically the same like on arm systems.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/pci.c | 25 +++++++++++++++++++++----
1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/hypervisor/arch/riscv/pci.c b/hypervisor/arch/riscv/pci.c
index 067a7651..fc6ed5fa 100644
--- a/hypervisor/arch/riscv/pci.c
+++ b/hypervisor/arch/riscv/pci.c
@@ -10,7 +10,7 @@

* the COPYING file in the top-level directory.
*/

-#include <jailhouse/entry.h>
+#include <jailhouse/mmio.h>
#include <jailhouse/pci.h>

u32 arch_pci_read_config(u16 bdf, u16 address, unsigned int size)
@@ -24,7 +24,7 @@ void arch_pci_write_config(u16 bdf, u16 address, u32 value, unsigned int size)

int arch_pci_add_physical_device(struct cell *cell, struct pci_device *device)
{
- return -ENOSYS;
+ return 0;
}

void arch_pci_remove_physical_device(struct pci_device *device)
@@ -40,10 +40,27 @@ void arch_pci_set_suppress_msi(struct pci_device *device,
int arch_pci_update_msi(struct pci_device *device,
const struct jailhouse_pci_capability *cap)
{
- return -ENOSYS;
+ const struct jailhouse_pci_device *info = device->info;
+ unsigned int n;
+
+ /*
+ * NOTE: We don't have interrupt remapping yet. So we write the values
+ * the cell passed without modifications. Probably not safe on all
+ * platforms.
+ */
+ for (n = 1; n < (info->msi_64bits ? 4 : 3); n++)
+ pci_write_config(info->bdf, cap->start + n * 4,
+ device->msi_registers.raw[n], 4);
+
+ return 0;
}

int arch_pci_update_msix_vector(struct pci_device *device, unsigned int index)
{
- return -ENOSYS;
+ /* NOTE: See arch_pci_update_msi. */
+ mmio_write64_split(&device->msix_table[index].address,
+ device->msix_vectors[index].address);
+ mmio_write32(&device->msix_table[index].data,
+ device->msix_vectors[index].data);
+ return 0;
}
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/paging.h | 119 ++++++++++++++++--
.../arch/riscv/include/asm/paging_modes.h | 18 +++
hypervisor/include/jailhouse/header.h | 5 +
3 files changed, 130 insertions(+), 12 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/paging.h b/hypervisor/arch/riscv/include/asm/paging.h
index 866f9a3f..5044a5fb 100644
--- a/hypervisor/arch/riscv/include/asm/paging.h
+++ b/hypervisor/arch/riscv/include/asm/paging.h
@@ -4,7 +4,7 @@

* Copyright (c) Siemens AG, 2020

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

*
* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

@@ -14,41 +14,136 @@
#define _JAILHOUSE_ASM_PAGING_H

#include <jailhouse/types.h>
+#include <jailhouse/header.h>
+#include <asm/csr64.h>

#define PAGE_SHIFT 12

-#define MAX_PAGE_TABLE_LEVELS 3
+#define RISCV_PAGE_SV_DEPTH 39
+#define RISCV_HV_PAGE_SV_DEPTH 39

+/*
+ * SV32 requires 2, SV39 3, SV48 4. Once we support SV57, this should be 5.
+ */
+#define MAX_PAGE_TABLE_LEVELS 4
#define PAGE_FLAG_FRAMEBUFFER 0
+/* RISC-v does not insert caching/prefetching/... information into page tables.
+ * Instead, this is done via `Physical Memory Attributes'. However, the ISA
+ * does not define a concrete form of PMAs. E.g., they could be hardcoded by
+ * physical address in a SoC.
+ */
#define PAGE_FLAG_DEVICE 0

-#define PAGE_DEFAULT_FLAGS 0
-#define PAGE_READONLY_FLAGS 0
-#define PAGE_PRESENT_FLAGS 0
+#define RISCV_PAGE_WIDTH 12
+#define RISCV_PTE_SIZE 3
+#define RISCV_PTES_PER_NODE (RISCV_PAGE_WIDTH - RISCV_PTE_SIZE)
+
+#define RISCV_PTE_V 0
+#define RISCV_PTE_R 1
+#define RISCV_PTE_W 2
+#define RISCV_PTE_X 3
+#define RISCV_PTE_U 4
+#define RISCV_PTE_G 5
+#define RISCV_PTE_A 6
+#define RISCV_PTE_D 7
+#define RISCV_PTE_RSW 8
+#define RISCV_PTE_RSW_w 2
+
+#define RISCV_PTE_FLAG(FLAG) (1 << RISCV_PTE_ ## FLAG)
+
+#define PAGE_DEFAULT_FLAGS (0\
+ | RISCV_PTE_FLAG (V)\
+ | RISCV_PTE_FLAG (R)\
+ | RISCV_PTE_FLAG (W)\
+ | RISCV_PTE_FLAG (X)\
+ )
+#define PAGE_READONLY_FLAGS (0\
+ | RISCV_PTE_FLAG (V)\
+ | RISCV_PTE_FLAG (R)\
+ )
+#define PAGE_READWRITE_FLAGS (0\
+ | RISCV_PTE_FLAG (V)\
+ | RISCV_PTE_FLAG (R)\
+ | RISCV_PTE_FLAG (W)\
+ )
+#define PAGE_PRESENT_FLAGS (0\
+ | RISCV_PTE_FLAG (V)\
+ )
#define PAGE_NONPRESENT_FLAGS 0

-#define INVALID_PHYS_ADDR (~0UL)
+#define INVALID_PHYS_ADDR (~0ul)

-#define TEMPORARY_MAPPING_BASE 0x0000008000000000UL
+#define TEMPORARY_MAPPING_BASE (~0ul << (RISCV_PAGE_SV_DEPTH - 1))
#define NUM_TEMPORARY_PAGES 16

-#define REMAP_BASE 0xffffff8000000000UL
#define NUM_REMAP_BITMAP_PAGES 4
+/*
+ * REMAP_BASE hast to be described by the same top-level PTE as JAILHOUSE_BASE,
+ * otherwise it won't be added to non-CPU 0 page tables automatically
+ */
+#define REMAP_BASE\
+ (JAILHOUSE_BASE & ~0ul << (RISCV_HV_PAGE_SV_DEPTH - 9))

-#define CELL_ROOT_PT_PAGES 1
+#if REMAP_BASE + (NUM_REMAP_BITMAP_PAGES * 1ul << /*
+ */ 12 /* ld bytes/page
+ */ + 3 /* ld bits/byte
+ */ + 12 /* each bit represents a page
+ */) > JAILHOUSE_BASE
+
+# error Overlap between REMAP area and JAILHOUSE_BASE!
+#endif
+
+#define CELL_ROOT_PT_PAGES (1 << 2)
+
+#define ATP_MODE_SHIFT 60
+#define ATP_MODE_SV39 0x8
+#define ATP_MODE_SV48 0x9
+#define ATP_MODE_SV57 0xa

#ifndef __ASSEMBLY__

-typedef u64 *pt_entry_t;
+#include <jailhouse/string.h>
+
+typedef size_t *pt_entry_t;

-static inline void arch_paging_flush_page_tlbs(unsigned long page_addr)
+/* MMU mode for Jailhouse (S-Mode) */
+extern unsigned char hv_atp_mode;
+
+static inline void arch_paging_flush_page_tlbs(size_t page_addr)
{
+ asm volatile("sfence.vma /* rd, */ zero, %[addr]" :
+ : [addr] "r" (page_addr));
}

-static inline void arch_paging_flush_cpu_caches(void *addr, long size)
+/*
+ * In RISC-V, the MMU accesses page tables through the caches (the MMU is a
+ * coherent agent)
+ */
+static inline void arch_paging_flush_cpu_caches(void const *addr, size_t size)
{
}

+#define ENABLE_MMU(NAME, REG) \
+static inline void enable_mmu_##NAME(u8 mode, unsigned long pt) \
+{ \
+ u64 atp; \
+ \
+ atp = (u64)mode << ATP_MODE_SHIFT | (u64)pt >> PAGE_SHIFT; \
+ asm volatile("sfence.vma\n" \
+ "csrw %0, %1\n" \
+ "sfence.vma\n" \
+ : : "i"(REG), "rK"(atp) : "memory"); \
+}
+
+ENABLE_MMU(satp, CSR_SATP)
+ENABLE_MMU(hgatp, CSR_HGATP)
+
+struct paging_structures;
+struct cell;
+
+void riscv_paging_vcpu_init(struct paging_structures *pg_structs);
+int riscv_paging_cell_init(struct cell *const cell);
+
#endif /* !__ASSEMBLY__ */

#endif /* !_JAILHOUSE_ASM_PAGING_H */
diff --git a/hypervisor/arch/riscv/include/asm/paging_modes.h b/hypervisor/arch/riscv/include/asm/paging_modes.h
index e69de29b..b7beb168 100644
--- a/hypervisor/arch/riscv/include/asm/paging_modes.h
+++ b/hypervisor/arch/riscv/include/asm/paging_modes.h

@@ -0,0 +1,18 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See

+ * the COPYING file in the top-level directory.
+ */
+
+#include <jailhouse/paging.h>
+
+extern const struct paging riscv_Sv39[];
+extern const struct paging riscv_Sv48[];
+extern const struct paging riscv_Sv39x4[];
+extern const struct paging riscv_Sv48x4[];
diff --git a/hypervisor/include/jailhouse/header.h b/hypervisor/include/jailhouse/header.h
index 518bc5cb..c5285262 100644
--- a/hypervisor/include/jailhouse/header.h
+++ b/hypervisor/include/jailhouse/header.h
@@ -10,6 +10,9 @@

* the COPYING file in the top-level directory.
*/

+#ifndef _JAILHOUSE_HEADER_H
+#define _JAILHOUSE_HEADER_H
+
#include <asm/jailhouse_header.h>

#define JAILHOUSE_SIGNATURE "JAILHOUS"
@@ -94,3 +97,5 @@ struct jailhouse_header {
};

#endif /* !__ASSEMBLY__ */
+

+#endif /* _JAILHOUSE_HEADER_H */
--
2.36.1

[Ralf Ramsauer]

For the moment, only support the fast&easy output path that is always
available: SBI. No UARTs needed for the hypervisor, SBI suffices atm.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/dbg-write.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/hypervisor/arch/riscv/dbg-write.c b/hypervisor/arch/riscv/dbg-write.c
index 6bd75887..187d4517 100644
--- a/hypervisor/arch/riscv/dbg-write.c
+++ b/hypervisor/arch/riscv/dbg-write.c
@@ -4,14 +4,24 @@

* Copyright (c) Siemens AG, 2020
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

*/

#include <jailhouse/printk.h>
+#include <asm/sbi.h>

-void arch_dbg_write_init(void)
+static void riscv_dbg_write_sbi(const char *msg)
{
+ char ch;
+
+ while ((ch = *msg++))
+ sbi_console_putchar_legacy0_1(ch);
+}
+
+void arch_dbg_write_init (void)
+{
+ arch_dbg_write = riscv_dbg_write_sbi;
}
--
2.36.1

[Ralf Ramsauer]

Now we have to think of Jailhouse's memory layout.

RISC-V has a bunch of MMU variants SVxx (SV39, SV48, SV57), while xx
denotes the number of virtually adressable bits. And there's an
exceptional case: If bit xx-1 is set in a vaddr, then all bits >= XX
must be 1.

For the virtual HV location, we need a virtual address that is valid for
each MMU type. Let's virtually place the hypervisor at the topmost 16
MiB of a SV39. This results in address 0xffff_ffd0_0000_0000.

On RISC-V, Linux could map the hypervisor to any virtual location, so we
need to setup bootstrap page tables that help us to jump to our link
location. Now this gets funny, as we need to support ALL MMU types for
the trampoline, as Linux might map us to a very high address.

The position-independent startup/trampoline code sets up a trampoline
that consists of two entries:
- Virtual Linux Load Location -> physical HV memory
- 0xffff_ffd0_0000_0000 -> physical HV memory

The startup code will create and activate this table, and jump to the
destination address where it was linked. From now on, we can use
position-dependant code.

Later, those trampoline page tables will not be destroyed, as we can
reuse them when disabling the hypervisor, or when errors occur during
startup.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/Kbuild | 3 +-
hypervisor/arch/riscv/entry.S | 349 +++++++++++++++++-
hypervisor/arch/riscv/exception.S | 91 +++++
.../arch/riscv/include/asm/jailhouse_header.h | 24 +-
hypervisor/arch/riscv/include/asm/setup.h | 14 +
hypervisor/arch/riscv/setup.c | 10 +-
6 files changed, 481 insertions(+), 10 deletions(-)
create mode 100644 hypervisor/arch/riscv/exception.S
create mode 100644 hypervisor/arch/riscv/include/asm/setup.h

diff --git a/hypervisor/arch/riscv/Kbuild b/hypervisor/arch/riscv/Kbuild
index 4213194f..7809007c 100644
--- a/hypervisor/arch/riscv/Kbuild
+++ b/hypervisor/arch/riscv/Kbuild
@@ -14,4 +14,5 @@

always-y := lib.a

-lib-y := entry.o setup.o dbg-write.o control.o ivshmem.o paging.o pci.o traps.o lib.o
+lib-y := entry.o exception.o setup.o dbg-write.o control.o ivshmem.o paging.o
+lib-y += pci.o traps.o lib.o
diff --git a/hypervisor/arch/riscv/entry.S b/hypervisor/arch/riscv/entry.S
index def55516..524270cf 100644
--- a/hypervisor/arch/riscv/entry.S
+++ b/hypervisor/arch/riscv/entry.S
@@ -2,16 +2,355 @@

* Jailhouse, a Linux-based partitioning hypervisor

*

* Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.
*/

- .text
- .globl arch_entry
+#include <asm/asm-defines.h>
+#include <asm/csr64.h>
+#include <asm/paging.h>
+
+#define SSTATUS_INITIAL (SR_SPIE | SR_SPP | SR_FS_DIRTY | \
+ SR_VS_DIRTY | SR_XS_DIRTY | SR_UXL_64)
+
+#define PAGING_FLAGS_NEXT_LEVEL RISCV_PTE_FLAG(G) | RISCV_PTE_FLAG(V)
+#define PAGING_FLAGS_BOOTSTRAP PAGE_DEFAULT_FLAGS | RISCV_PTE_FLAG(G)
+
+/*
+ * hideleg needs to be set early, otherwise vsip/vsie are forced to 0. Bits set
+ * here causes aliasing of VSIP/HIP and VSIE/HIE
+ */
+#define INITIAL_HIDELEG \
+ ((IE_SIE << VSIP_TO_HVIP_SHIFT) | \
+ (IE_TIE << VSIP_TO_HVIP_SHIFT) | \
+ (IE_EIE << VSIP_TO_HVIP_SHIFT))
+
+#define INITIAL_SIE (IE_SIE | IE_TIE | IE_EIE)
+
+.macro csr_from_csr vsreg, sreg, t
+ csrr \t, \sreg
+ csrw \vsreg, \t
+.endm
+
+.macro load_csr csr, temp, value
+ li \temp, \value
+ csrw \csr, \temp
+.endm
+
+.macro access_config op, reg, offset
+ \op \reg, \offset (t4)
+.endm
+
+.macro ld_constant reg, constant
+ ld \reg, %lo (pool_\constant - constant_pool) (t6)
+.endm
+
+.macro bootstrap_page macro
+ .irp page,\
+ bt_tbl_l0,\
+ bt_tbl_l1_0,\
+ bt_tbl_l2_0,\
+ bt_tbl_l3_0,\
+ bt_tbl_l1_1,\
+ bt_tbl_l2_1,\
+ bt_tbl_l3_1
+ \macro \page
+ .endr
+.endm
+.globl bt_tbl_l0
+
+.macro define_page page
+ .p2align RISCV_PAGE_WIDTH
+ \page:
+ .skip (1 << RISCV_PAGE_WIDTH)
+.endm
+
+.macro pool_constant symbol
+ .p2align 3
+ pool_\symbol:
+ .8byte \symbol
+.endm
+
+/*
+ * level: 0 for 4 KiB leaves
+ * virt: virtual address to map
+ * pte: pte contents: phys addr for leaves, sub node otherwise; clobbered
+ * table: node to write pte to
+ * temp: clobberable register
+ * flags: PTE flag bits
+ */
+.macro riscv_pt_write_pte level, virt, pte, table, temp, flags
+ ori \pte, \pte, \flags
+ /* RISC-V uses 12-bit _signed_ immediates, so extracting the
+ * least-significant 12 bits can't be done in a single instruction of
+ * the base ABI
+ */
+ srli \temp, \virt,\
+ (RISCV_PAGE_WIDTH + (\level) * RISCV_PTES_PER_NODE)
+ andi \temp, \temp, 1 << (RISCV_PAGE_WIDTH - RISCV_PTE_SIZE) - 1
+ slli \temp, \temp, RISCV_PTE_SIZE
+ add \temp, \temp, \table
+ sd \pte, 0 (\temp)
+.endm
+
+.macro riscv_pt_write_address level, virt, addr, table, temp, flags
+ srli \addr, \addr, 2
+ riscv_pt_write_pte \level, \virt, \addr, \table, \temp, \flags
+.endm
+
+.macro write_table lvl, table, dst, flags
+ la t0, \table
+ ld_constant t3, \dst
+ add t4, a2, t3
+ riscv_pt_write_address \lvl, t2, t4, t0, t5, \flags
+.endm
+
+.macro sv57_map_2M l0, l1, l2, l3, phys
+ write_table 4, \l0, \l1, PAGING_FLAGS_NEXT_LEVEL
+ write_table 3, \l1, \l2, PAGING_FLAGS_NEXT_LEVEL
+ write_table 2, \l2, \l3, PAGING_FLAGS_NEXT_LEVEL
+ write_table 1, \l3, \phys, PAGING_FLAGS_BOOTSTRAP
+.endm
+
+.macro sv48_map_2M l0, l1, l2, phys
+ write_table 3, \l0, \l1, PAGING_FLAGS_NEXT_LEVEL
+ write_table 2, \l1, \l2, PAGING_FLAGS_NEXT_LEVEL
+ write_table 1, \l2, \phys, PAGING_FLAGS_BOOTSTRAP
+.endm
+
+.macro sv39_map_2M l0, l1, phys
+ write_table 2, \l0, \l1, PAGING_FLAGS_NEXT_LEVEL
+ write_table 1, \l1, \phys, PAGING_FLAGS_BOOTSTRAP
+.endm
+
+/* place large constants into here */
+.section .rodata
+pool_constant virtual_arch_entry
+bootstrap_page pool_constant
+
+.p2align 3
+constant_pool:
+.p2align 3
+pool_jailhouse_base:
+ .8byte JAILHOUSE_BASE
+
+.section .data
+ bootstrap_page define_page
+ .purgem define_page
+
+.text
+.globl arch_entry
arch_entry:
- li a0,-55
- ret
+ /*
+ * a0: cpuid
+ * t5: hypervisor_header
+ * t6: __page_pool
+ */
+ la t5, hypervisor_header
+ la t6, __page_pool
+
+ /*
+ * t1: max_cpus
+ * t2: sizeof(struct per_cpu)
+ * Note: this would also be available directly: sizeof$struct$per_cpu
+ */
+ lw t1, HEADER_MAX_CPUS (t5)
+ lw t2, HEADER_PERCPU_SIZE (t5)
+
+ /* t4: configuration area */
+ mul t2, t2, t1
+ add t4, t2, t6
+
+ /* t6: constant_pool */
+ la t6, constant_pool
+
+ /*
+ * these pages are used during the initialization phase currently, they
+ * are not reclaimed afterwards. Create a "bootstrap" paging table
+ * that maps the the hypervisor image to the logical/physical addresses
+ * it was linked and also where it was currently linked.
+ *
+ * In case of any MMU variant, we always need to map 2M-Pages.
+ * Depending on the MMU variant, this may require different 'n' levels
+ * of page tables.
+ * SV39: 2 levels
+ * SV48: 3 levels
+ * SV57: 4 levels
+ *
+ * We always require one root table, and then (n-1)*2 subtables.
+ * SV39: 1+1*2 = 3 pages
+ * SV48: 1+2*2 = 5 pages
+ * SV57: 1+3*2 = 7 pages
+ *
+ * The problem is the location of those page tables. For the worst
+ * case, SV57, which we need to prepare for, we would need to reserve 7
+ * pages somewhere. If we then land on a SV39 system, we waste 4 pages
+ * (16KiB).
+ *
+ * So for the moment, simply always statically allocate 7 pages that
+ * are dedicated for the bootstrap page tables. We can't reclaim
+ * memory, as we need those bootstrap tables for the rollback
+ * trampoline, when disabling the hypervisor.
+ *
+ * An alternative I could think of is to allocate those pages at the
+ * very end of hypervisor_mem. We could fill pages, and mark those
+ * pages as used during arch_paging_init. This way, we wouldn't waste
+ * any memory, but it's more complex. For the moment, let's implement
+ * the simple variant, and allocate 7 pages.
+ *
+ * Just some further notes:
+ * If Linux chose to select, for example, SV48, the the bootstrap
+ * tables MUST also select SV48. This is, because there's a 99% chance
+ * that Linux chose an address to remap Jailhouse that is only
+ * adressable with a SV48. Hence, the trampolin page tables must also
+ * be at least SV48. Later, when setting up the final tables, we could
+ * switch down to SV39. This is recommendable, as it requires less page
+ * table lookups.
+ *
+ * t0: Lx VA bootstrap_table_l0
+ * t3: JH VA of bootstrap_table_hypervisor_l1
+ */
+ ld_constant t2, jailhouse_base
+ /* a3: jailhouse_memory.phys_start */
+ access_config ld, a3, CFG_PHYS_START
+ /* a2: virt_to_phys; phys = virt_to_phys + virt */
+ sub a2, a3, t2
+
+ /* Get MMU-Variant that Linux currently uses */
+ csrr a4, satp
+ srli a4, a4, ATP_MODE_SHIFT
+ /* And choose the same variant for the bootstrap tables */
+ li t2, ATP_MODE_SV39
+ beq a4, t2, sv39
+ li t2, ATP_MODE_SV48
+ beq a4, t2, sv48
+ li t2, ATP_MODE_SV57
+ beq a4, t2, sv57
+
+ /* Unknown MMU type if we reach that */
+1: wfi
+ j 1b
+
+ /* virt must be held in t2 */
+sv39:
+ ld_constant t2, jailhouse_base
+ sv39_map_2M bt_tbl_l0, bt_tbl_l1_0, jailhouse_base
+ la t2, arch_entry
+ sv39_map_2M bt_tbl_l0, bt_tbl_l1_1, jailhouse_base
+
+ li t0, ATP_MODE_SV39
+ j setup_satp
+
+sv48:
+ ld_constant t2, jailhouse_base
+ sv48_map_2M bt_tbl_l0, bt_tbl_l1_0, bt_tbl_l2_0, jailhouse_base
+ la t2, arch_entry
+ sv48_map_2M bt_tbl_l0, bt_tbl_l1_1, bt_tbl_l2_1, jailhouse_base
+
+ li t0, ATP_MODE_SV48
+ j setup_satp
+
+sv57:
+ ld_constant t2, jailhouse_base
+ sv57_map_2M bt_tbl_l0, bt_tbl_l1_0, bt_tbl_l2_0, bt_tbl_l3_0, \
+ jailhouse_base
+ la t2, arch_entry
+ sv57_map_2M bt_tbl_l0, bt_tbl_l1_1, bt_tbl_l2_1, bt_tbl_l3_1, \
+ jailhouse_base
+
+ li t0, ATP_MODE_SV57
+ j setup_satp
+
+setup_satp:
+ ld_constant a3, bt_tbl_l0
+ add a3, a2, a3
+ srli a3, a3, RISCV_PAGE_WIDTH
+ slli t0, t0, ATP_MODE_SHIFT
+ or a3, a3, t0
+
+ sfence.vma zero, zero
+ csrrw a4, satp, a3
+ csrw CSR_VSATP, a4
+
+ ld_constant t1, virtual_arch_entry
+ /* leave pos independant code with this jump */
+ jalr t0, t1, 0
+
+ # a0 ... cpuid
+ # a2 ... physical - virtual address offset
+ # a3 ... bootstrap satp
+ # a4 ... linux page table root (currently unused)
+ # ra ... linux return address
+ # t0 ... return address for call from arch_entry
+virtual_arch_entry:
+ # set up stack pointer -- this is hart-dependent
+
+ # percpu data = pool + cpuid * percpu_size
+ la t1, __page_pool
+
+ la t6, hypervisor_header
+ lw t2, HEADER_PERCPU_SIZE (t6)
+
+ mul t2, t2, a0
+ add a1, t2, t1
+ # a1 ... percpu data
+
+ mv t6, sp # need to restore this later
+
+ li t1, PCPU_GUEST_REGS
+ add sp, a1, t1
+
+ sd t6, REG_SP (sp)
+ sd ra, REG_RA (sp)
+ sd t0, REG_T0 (sp)
+ sd gp, REG_GP (sp)
+ sd tp, REG_TP (sp)
+ sd s0, REG_S0 (sp)
+ sd s1, REG_S1 (sp)
+ sd s2, REG_S2 (sp)
+ sd s3, REG_S3 (sp)
+ sd s4, REG_S4 (sp)
+ sd s5, REG_S5 (sp)
+ sd s6, REG_S6 (sp)
+ sd s7, REG_S7 (sp)
+ sd s8, REG_S8 (sp)
+ sd s9, REG_S9 (sp)
+ sd s10, REG_S10 (sp)
+ sd s11, REG_S11 (sp)
+ # fp is s0 -> fp is restored
+
+ # vsstatus receives value of sstatus
+ csr_from_csr CSR_VSSTATUS, sstatus, t1
+
+ li t1, SSTATUS_INITIAL
+ csrw sstatus, t1
+
+ load_csr CSR_HIDELEG, t1, INITIAL_HIDELEG
+
+ csr_from_csr CSR_VSIE, sie, t1
+
+ load_csr sie, t1, INITIAL_SIE
+
+
+ csr_from_csr CSR_VSCAUSE, scause, t1
+ csr_from_csr CSR_VSTVAL, stval, t1
+
+ csr_from_csr CSR_VSSCRATCH, sscratch, t1
+ csrw sscratch, sp
+
+ csr_from_csr CSR_VSTVEC, stvec, t1
+ la t1, nested_exception_handler
+ csrw stvec, t1
+
+ jal ra, entry
+ /* return from entry means an error occurred */
+
+ mv a1, a0
+ mv a0, sp
+ j riscv_deactivate_vmm
diff --git a/hypervisor/arch/riscv/exception.S b/hypervisor/arch/riscv/exception.S
new file mode 100644
index 00000000..0c623602
--- /dev/null
+++ b/hypervisor/arch/riscv/exception.S
@@ -0,0 +1,91 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:

+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *

+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <asm/asm-defines.h>
+
+.macro load_csr csr, temp, value
+ li \temp, \value
+ csrw \csr, \temp
+.endm
+
+.macro context_save_all cmd
+ .irp reg 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, \
+ 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
+ \cmd x\reg, 8*(\reg)(sp)
+ .endr
+.endm
+
+/*
+ * This entry point is used when the hypervisor has been entered already. The
+ * existing stack is reused.
+ */
+.align 4
+.globl nested_exception_handler
+nested_exception_handler:
+ /*
+ * We reuse the old S-Mode stack, so make some place for union
+ * registers. We must not acces this_cpu()-related stuff, as this will
+ * point to the old register set. But it's not necessary, as nested
+ * exception only occur in rare situations, and only one of them (hlvx)
+ * is recoverable. Performance isn't an optimisation criteria, as a
+ * nested exception of hlvx will lead to a crash of the cell in any
+ * case.
+ */
+ addi sp, sp, -REGISTERS_SIZE
+ context_save_all sd
+ mv a0, sp
+ addi a0, a0, REGISTERS_SIZE
+ sd a0, REG_SP (sp)
+ addi a0, a0, -REGISTERS_SIZE
+
+ jal arch_handle_fault
+
+ context_save_all ld
+ addi sp, sp, REGISTERS_SIZE
+ sret
+
+/*
+ * This entry point is used when the guest is running. A new stack needs to be
+ * established (and is taken from sscratch).
+ */
+.align 4
+.globl exception_handler
+exception_handler:
+ csrrw sp, sscratch, sp
+ context_save_all sd
+
+ csrr t1, sscratch
+ sd t1, REG_SP (sp)
+
+ la a0, nested_exception_handler
+ csrw stvec, a0
+
+ mv a0, sp
+ jal arch_handle_trap
+
+.globl vmreturn
+vmreturn:
+ la a0, exception_handler
+ csrw stvec, a0
+
+ context_save_all ld
+ csrrw sp, sscratch, sp
+ sret
+
+.p2align 12
+.globl riscv_park_loop
+riscv_park_loop:
+ wfi
+ j riscv_park_loop
+.p2align 12
diff --git a/hypervisor/arch/riscv/include/asm/jailhouse_header.h b/hypervisor/arch/riscv/include/asm/jailhouse_header.h
index a0c22083..c4e1c24c 100644
--- a/hypervisor/arch/riscv/include/asm/jailhouse_header.h
+++ b/hypervisor/arch/riscv/include/asm/jailhouse_header.h
@@ -4,11 +4,29 @@
* Copyright (C) Siemens AG, 2020

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.
*/

-#define JAILHOUSE_BASE __JH_CONST_UL(0xffffffdfff000000)
-#define JAILHOUSE_BORROW_ROOT_PT 1
+/*
+ * RISC-V defined architectural constraints on the address space:
+ * Address immediates are sign extended.
+ *
+ * lui+jalr can jump to absolute addresses +-2 GiB around zero
+ * --> this best left free for millicode routines (or boot ROM),
+ * at least on 64-bit machines.
+ * This is the range 0xffff_ffff_8000_000--0x0000_0000_7fff_ffff
+ *
+ * Legal virtual addresses are sign extensions of the most-significant bit
+ * mapped by the virtual addressing mode; e.g., in Sv39, bits 63:39 of an
+ * address must be a copy of bit 38. Hence, the lowest (signed) address
+ * available is 0xffff_ffc0_0000_0000.
+ *
+ * Linux normally maps itself at 0xffff_ffe0_0000_0000 and modules at
+ * 0xffff_ffd0_0000_0000.
+ */
+
+/* this provides 16 MB of space for the Jailhouse core */
+#define JAILHOUSE_BASE 0xffffffdfff000000
diff --git a/hypervisor/arch/riscv/include/asm/setup.h b/hypervisor/arch/riscv/include/asm/setup.h
new file mode 100644
index 00000000..a5b6f564
--- /dev/null
+++ b/hypervisor/arch/riscv/include/asm/setup.h
@@ -0,0 +1,14 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *

+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+void __attribute__((noreturn))
+riscv_deactivate_vmm(union registers *regs, int errcode);
diff --git a/hypervisor/arch/riscv/setup.c b/hypervisor/arch/riscv/setup.c
index 53cdce80..985f27e2 100644
--- a/hypervisor/arch/riscv/setup.c
+++ b/hypervisor/arch/riscv/setup.c
@@ -4,13 +4,15 @@

* Copyright (c) Siemens AG, 2020
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.
*/

#include <jailhouse/entry.h>
+#include <asm/processor.h>
+#include <asm/setup.h>

int arch_init_early(void)
{
@@ -22,6 +24,12 @@ int arch_cpu_init(struct per_cpu *cpu_data)
return -ENOSYS;
}

+void __attribute__((noreturn))
+riscv_deactivate_vmm(union registers *regs, int errcode)
+{
+ while (1);
+}
+
void __attribute__((noreturn)) arch_cpu_activate_vmm(void)
{
while (1);
--
2.36.1

[Ralf Ramsauer]

On RISC-V, there's no possibility to parameterise IPIs resp. enrich them
with some userdata. Great. We need to do it in software...

We differentiate between two IPI types in software: IPI_CAUSE_MGMT, and
IPI_CAUSE_GUEST.

When a guest sends an IPI to a HART within the cell, the HV will trap,
and we forward the IPI call to the SBI. Before, we set the cause to
IPI_CAUSE_GUEST.

Follow the same logic for management IPIs accordingly.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/control.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)

diff --git a/hypervisor/arch/riscv/control.c b/hypervisor/arch/riscv/control.c
index b848994b..263708a7 100644
--- a/hypervisor/arch/riscv/control.c
+++ b/hypervisor/arch/riscv/control.c
@@ -11,6 +11,8 @@
*/

#include <jailhouse/control.h>
+#include <jailhouse/printk.h>
+#include <asm/sbi.h>

int arch_cell_create(struct cell *cell)
{
@@ -68,4 +70,16 @@ void arch_park_cpu(unsigned int cpu_id)

void arch_send_event(struct public_per_cpu *target_data)
{
+ struct sbiret result;
+
+ target_data->ipi_cause = IPI_CAUSE_MGMT;
+ memory_barrier();
+
+ result = sbi_send_ipi(1UL << (target_data->phys_id % BITS_PER_LONG),
+ target_data->phys_id / BITS_PER_LONG);
+ if (result.error != SBI_SUCCESS) {
+ printk("IPI send to HART %lu failed: %ld\n",
+ target_data->phys_id, result.error);
+ panic_stop();
+ }
}
--
2.36.1

[Ralf Ramsauer]

Export offsets that we will later need in assembly.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/asm-defines.c | 27 ++++++++++++++++++++++
hypervisor/include/jailhouse/gen-defines.h | 8 ++++++-
2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/hypervisor/arch/riscv/asm-defines.c b/hypervisor/arch/riscv/asm-defines.c
index 17c2f256..0143d5c9 100644
--- a/hypervisor/arch/riscv/asm-defines.c
+++ b/hypervisor/arch/riscv/asm-defines.c
@@ -11,9 +11,36 @@
*/

#include <jailhouse/gen-defines.h>
+#include <jailhouse/percpu.h>

void common(void);

void common(void)
{
+ OFFSETU(REG_SP, registers, sp);
+ OFFSETU(REG_RA, registers, ra);
+ OFFSETU(REG_T0, registers, t0);
+ OFFSETU(REG_TP, registers, tp);
+ OFFSETU(REG_GP, registers, gp);
+ OFFSETU(REG_S0, registers, s0);
+ OFFSETU(REG_S1, registers, s1);
+ OFFSETU(REG_S2, registers, s2);
+ OFFSETU(REG_S3, registers, s3);
+ OFFSETU(REG_S4, registers, s4);
+ OFFSETU(REG_S5, registers, s5);
+ OFFSETU(REG_S6, registers, s6);
+ OFFSETU(REG_S7, registers, s7);
+ OFFSETU(REG_S8, registers, s8);
+ OFFSETU(REG_S9, registers, s9);
+ OFFSETU(REG_S10, registers, s10);
+ OFFSETU(REG_S11, registers, s11);
+
+ OFFSET(HEADER_MAX_CPUS, jailhouse_header, max_cpus);
+ OFFSET(HEADER_PERCPU_SIZE, jailhouse_header, percpu_size);
+
+ OFFSET(CFG_PHYS_START, jailhouse_system, hypervisor_memory.phys_start);
+
+ OFFSET(PCPU_GUEST_REGS, per_cpu, guest_regs);
+
+ DEFINE(REGISTERS_SIZE, sizeof(union registers));
}
diff --git a/hypervisor/include/jailhouse/gen-defines.h b/hypervisor/include/jailhouse/gen-defines.h
index d42fe255..7b348761 100644
--- a/hypervisor/include/jailhouse/gen-defines.h
+++ b/hypervisor/include/jailhouse/gen-defines.h
@@ -22,8 +22,14 @@

#define BLANK() asm volatile("\n=>" : : )

+#define OFF(type, sym, str, mem) \
+ DEFINE(sym, __builtin_offsetof(type str, mem))
+
#define OFFSET(sym, str, mem) \
- DEFINE(sym, __builtin_offsetof(struct str, mem))
+ OFF(struct, sym, str, mem)
+
+#define OFFSETU(sym, str, mem) \
+ OFF(union , sym, str, mem)

#define COMMENT(x) \
asm volatile("\n=>#" x)
--
2.36.1

[Ralf Ramsauer]

Specifically on RISC-V, and fix some typos.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

Documentation/hypervisor-interfaces.txt | 16 ++++++++++++++--
Documentation/memory-layout.txt | 20 ++++++++++++--------
2 files changed, 26 insertions(+), 10 deletions(-)

diff --git a/Documentation/hypervisor-interfaces.txt b/Documentation/hypervisor-interfaces.txt
index d888dab7..fe493e46 100644
--- a/Documentation/hypervisor-interfaces.txt
+++ b/Documentation/hypervisor-interfaces.txt
@@ -13,8 +13,8 @@ Detection
---------

This interface is useful for cell code that should work not only inside a
-Jailhouse cell. The ABI is architecture specific. So far, it is only available
-for x86.
+Jailhouse cell. The ABI is architecture specific and not available for
+all processor architectures.


x86 ABI
@@ -53,6 +53,18 @@ Result in EAX: 0
EDX: 0


+RISC-V ABI
+- - - - - -
+
+Jailhouse adhers to the Supervisor Binary Interface (SBI) defined in
+https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc
+That is, it provides the SBI to guests (as well as using the system
+provided SBI).
+
+The sbi_get_impl_id() function returns 'JHOU', 0x4a484f55.
+The get_sbi_impl_version() function returns the version of the Jailhouse SBI.
+
+
Hypercalls
----------

diff --git a/Documentation/memory-layout.txt b/Documentation/memory-layout.txt
index 82029f17..7e8281af 100644
--- a/Documentation/memory-layout.txt
+++ b/Documentation/memory-layout.txt
@@ -6,10 +6,10 @@ visible hypervisor memory, globally visible I/O memory remappings and CPU-
specific remappings of hypervisor memory as well as cell memory pages.

Cells are not mapped as a whole into the hypervisor address space. Only
-explicitely shared pages and pages that are temporarily mapped, e.g. during
-MMIO instruction parsing, are visible by the hypervisor during runtime.
-Furthermore, the visibility is limited to the CPU that create it because they
-are only added to the CPU-specifc mapping.
+explicitly shared pages and pages that are temporarily mapped, e.g. during
+MMIO instruction parsing, are visible to the hypervisor during runtime.
+Furthermore, their visibility is limited to the CPU that creates the mapping
+because they are only added to the CPU-specific mapping.


Common memory region
@@ -23,9 +23,9 @@ Prior to enabling the hypervisor and after disabling it, this region is also
mapped into the address space of Linux that acts as root cell. The virtual
address of this mapping is identical to the one used by the hypervisor when the
architecture set JAILHOUSE_BORROW_ROOT_PT (currently x86 and ARM), on other,
-this addressed can differ.
+this address can differ.

-The commom memory region contains an array of per-CPU data structures, one for
+The common memory region contains an array of per-CPU data structures, one for
each configured CPU. Each per-CPU data structure consists of a private part and
a public part. The public part will remain visible for all CPUs throughout the
hypervisor operation. The private part, however, is only visible during setup
@@ -112,15 +112,19 @@ Size: PAGE_SIZE * NUM_REMAP_BITMAP_PAGES * PAGE_SIZE * 8
| |
+--------------------------------------+ - higher address

+The architecture-independent code assumes that the I/O mememory remapping
+region is described by the same top-level PTE as the common memory region
+(JAILHOUSE_BASE).
+

CPU-specific remapping region
-----------------------------

-This region is differently mapped for each CPU. It consistes of a virtual
+This region is mapped differently for each CPU. It consists of a virtual
address range that is used for temporarily mapping individual pages of the cell
that runs on the same CPU.

-Futhermore, the private per-CPU data which is hidden from the common memory
+Furthermore, the private per-CPU data which is hidden from the common memory
region is made available at fixed virtual address here. This allows to
dereference CPU local data quickly and generically. Moreover, it hides cell-
private data that the hypervisor may collect on VM exit, e.g. register content,
--
2.36.1

[Ralf Ramsauer]

This commit add the stubs for two exception handlers: arch_handle_trap
and arch_handle_fault.

Rationale:

On RISC-V, there's a scratch register (CSR), which is typically user by
the supervisor to store its stack pointer. On traps, the guest/user's SP
is atomically exchanged with the scratch register, to set up the
supervisor's stack.

Now, in case of faults when we're inside Jailhouse, we would have to do
the same trick, but, in case of double faults we're doomed anyway.

So instead of preparing a new scratch register when entering the
Hypervisor, I chose to simply redirect the trap vector and reuse the
stack, while the hypervisor is active.

So if we're trapping the hypervisor, we will later end up in
arch_handle_trap. If we get a fault while we're inside the hypervisor,
we will end up in arch_handle_fault.

These routines will be called from assembly.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/Kbuild | 2 +-
hypervisor/arch/riscv/traps.c | 28 ++++++++++++++++++++++++++++
2 files changed, 29 insertions(+), 1 deletion(-)
create mode 100644 hypervisor/arch/riscv/traps.c

diff --git a/hypervisor/arch/riscv/Kbuild b/hypervisor/arch/riscv/Kbuild
index dd114a6c..4213194f 100644
--- a/hypervisor/arch/riscv/Kbuild
+++ b/hypervisor/arch/riscv/Kbuild
@@ -14,4 +14,4 @@

always-y := lib.a

-lib-y := entry.o setup.o dbg-write.o control.o ivshmem.o paging.o pci.o lib.o
+lib-y := entry.o setup.o dbg-write.o control.o ivshmem.o paging.o pci.o traps.o lib.o
diff --git a/hypervisor/arch/riscv/traps.c b/hypervisor/arch/riscv/traps.c
new file mode 100644
index 00000000..f61dfc1a
--- /dev/null
+++ b/hypervisor/arch/riscv/traps.c
@@ -0,0 +1,28 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+

+#include <asm/processor.h>
+
+void arch_handle_trap(union registers *regs);
+void arch_handle_fault(union registers *regs);
+
+void arch_handle_trap(union registers *regs)
+{
+ for (;;)
+ cpu_relax();
+}
+
+void arch_handle_fault(union registers *regs)
+{
+ for (;;)
+ cpu_relax();
+}
--
2.36.1

[Ralf Ramsauer]

From: Konrad Schwarz <konrad....@gmail.com>

While we need support for SV39, SV48 AND SV57 in the bootstrapping
pagetables, we can later grade down to SV39.

For hypervisor paging, I chose to use SV39, as it comes with less levels
than the other ones.

For G-Stage (guest) paging, I chose SV39 as well: It is very unlikely
that we need a GP address that is very very high. Independant of the
G-Stage paging, the guest may use any available paging method that it
wants. This means, Linux may enable SV57 paging on top of a SV39
G-stage.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/paging.c | 262 +++++++++++++++++++++++++-
hypervisor/include/jailhouse/paging.h | 2 +-
2 files changed, 260 insertions(+), 4 deletions(-)

diff --git a/hypervisor/arch/riscv/paging.c b/hypervisor/arch/riscv/paging.c
index 294ea958..99a842ea 100644
--- a/hypervisor/arch/riscv/paging.c
+++ b/hypervisor/arch/riscv/paging.c
@@ -2,22 +2,278 @@

* Jailhouse, a Linux-based partitioning hypervisor

*
* Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

*/

+#include <jailhouse/control.h>
#include <jailhouse/paging.h>
+#include <jailhouse/percpu.h>
+#include <asm/csr64.h>
+
+#define HV_PAGING riscv_Sv39
+#define HV_ATP_MODE ATP_MODE_SV39
+#define CELL_PAGING riscv_Sv39x4
+#define CELL_ATP_MODE ATP_MODE_SV39
+
+#define PAGE_BITS 12
+#define WORD_BITS 3 /* 1 << WORD_BITS == sizeof (void *) */
+#define MAX_FLAG 10
+#define FLAG_MASK ((1 << MAX_FLAG) - 1)
+
+#define PAGE_LEVEL_BITS (PAGE_BITS - WORD_BITS)
+#define PAGE_LEVEL_MASK(ROOT) ((1 << (PAGE_LEVEL_BITS + 2 * !!(ROOT))) - 1)
+
+#define UNTRANSLATED_BITS(LEVEL) \
+ ((LEVEL) * PAGE_LEVEL_BITS + PAGE_BITS)
+
+#define PAGE_TERMINAL_FLAGS \
+ (RISCV_PTE_FLAG(R) | RISCV_PTE_FLAG(W) | RISCV_PTE_FLAG(X))
+
+unsigned char hv_atp_mode;
+static unsigned char cell_atp_mode;
+
+static inline unsigned long pte2phys(unsigned long pte)
+{
+ return (pte & ~FLAG_MASK) << (PAGE_BITS - MAX_FLAG);
+}
+
+static inline unsigned long phys2pte(unsigned long phys)
+{
+ return phys >> (PAGE_BITS - MAX_FLAG);
+}
+
+#define DEF_GET_ENTRY(NAME, LEVEL, ROOT) \
+static pt_entry_t \
+sv## NAME ##_vpn## LEVEL ##_get_entry(page_table_t pt, \
+ unsigned long virt) \
+{ \
+ return pt + ((virt >> UNTRANSLATED_BITS(LEVEL)) & \
+ PAGE_LEVEL_MASK(ROOT)); \
+}
+
+DEF_GET_ENTRY(X, 0, false)
+DEF_GET_ENTRY(X, 1, false)
+DEF_GET_ENTRY(X, 2, false)
+DEF_GET_ENTRY(X, 3, false)
+
+static bool svX_entry_valid(pt_entry_t pte, unsigned long flags)
+{
+ /*
+ * We must not check flags, we only need to check for the V-bit. PTEs
+ * are valid, if V is set. If R/W/X is set, then it means that the PTE
+ * is a leaf.
+ */
+ return !!(*pte & RISCV_PTE_FLAG(V));
+}
+
+#define DEF_SET_TERMINAL(NAME, FLAGS) \
+static void sv## NAME ##_vpnX_set_terminal(pt_entry_t pte, \
+ unsigned long phys, \
+ unsigned long flags) \
+{ \
+ /* \
+ * set A and D flas pre-emptively, to avoid page-faults \
+ * exceptions when the hardware does not set A and D by itself \
+ */ \
+ *pte = FLAGS | RISCV_PTE_FLAG(D) | RISCV_PTE_FLAG(A) | flags | \
+ phys2pte(phys); \
+}
+
+DEF_SET_TERMINAL(X, 0)
+
+#define DEF_GET_PHYS(LEVEL) \
+static unsigned long \
+svX_vpn## LEVEL ##_get_phys (pt_entry_t pte, unsigned long virt)\
+{ \
+ unsigned long entry = *pte; \
+ if (!(RISCV_PTE_FLAG(V) & entry) || \
+ !(PAGE_TERMINAL_FLAGS & (entry))) \
+ return INVALID_PHYS_ADDR; \
+ return pte2phys(entry) | \
+ (((1UL << UNTRANSLATED_BITS(LEVEL)) - 1) & virt);\
+}
+
+DEF_GET_PHYS(0)
+DEF_GET_PHYS(1)
+DEF_GET_PHYS(2)
+DEF_GET_PHYS(3)
+
+static unsigned long svX_get_flags(pt_entry_t pte)
+{
+ return *pte & FLAG_MASK;
+}
+
+#define DEF_SET_NEXT(NAME, FLAGS) \
+static void \
+sv## NAME ##_vpnX_set_next_pt(pt_entry_t pte, unsigned long next_pt) \
+{ \
+ *pte = FLAGS | RISCV_PTE_FLAG(V) | phys2pte(next_pt); \
+}
+
+DEF_SET_NEXT(X, RISCV_PTE_FLAG(G))
+
+static unsigned long svX_vpnX_get_next_pt(pt_entry_t pte)
+{
+ return pte2phys(*pte);
+}
+
+static void svX_clear_entry(pt_entry_t pte)
+{
+ *pte = 0;
+}
+
+static inline bool _svX_page_table_empty(page_table_t page_table,
+ unsigned long len)
+{
+ unsigned long *page_table_end = page_table + len;
+
+ for (; page_table_end > page_table; ++page_table)
+ if (RISCV_PTE_FLAG (V) & *page_table)
+ return false;
+ return true;
+}
+
+static bool svX_page_table_empty(page_table_t page_table)
+{
+ return _svX_page_table_empty(page_table, 1 << PAGE_LEVEL_BITS);
+}
+
+#define RISCV_SVX_PAGING_LEVEL(LEVEL) \
+ { \
+ .page_size = 1UL << UNTRANSLATED_BITS(LEVEL), \
+ .get_entry = svX_vpn ## LEVEL ## _get_entry, \
+ .entry_valid = svX_entry_valid, \
+ .set_terminal = svX_vpnX_set_terminal, \
+ .get_phys = svX_vpn## LEVEL ##_get_phys, \
+ .get_flags = svX_get_flags, \
+ .set_next_pt = svX_vpnX_set_next_pt, \
+ .get_next_pt = svX_vpnX_get_next_pt, \
+ .clear_entry = svX_clear_entry, \
+ .page_table_empty = svX_page_table_empty, \
+ }
+
+/* sequence is from root to leaves */
+const struct paging riscv_Sv39[] = {
+ RISCV_SVX_PAGING_LEVEL(2),
+ RISCV_SVX_PAGING_LEVEL(1),
+ RISCV_SVX_PAGING_LEVEL(0),
+};
+
+const struct paging riscv_Sv48[] = {
+ RISCV_SVX_PAGING_LEVEL(3),
+ RISCV_SVX_PAGING_LEVEL(2),
+ RISCV_SVX_PAGING_LEVEL(1),
+ RISCV_SVX_PAGING_LEVEL(0),
+};
+
+/* 4K*2 for level 2, in case of SV39, and for level 3, ind case of SV48 */
+DEF_GET_ENTRY(39x4, 2, true)
+DEF_GET_ENTRY(48x4, 3, true)
+
+/* For the rest (non-root tbls), reuse svX routines */
+#define sv39x4_vpn0_get_entry svX_vpn0_get_entry
+#define sv39x4_vpn1_get_entry svX_vpn1_get_entry
+
+#define sv48x4_vpn0_get_entry svX_vpn0_get_entry
+#define sv48x4_vpn1_get_entry svX_vpn1_get_entry
+#define sv48x4_vpn2_get_entry svX_vpn2_get_entry
+
+#define sv39x4_vpn0_get_phys svX_vpn0_get_phys
+#define sv48x4_vpn0_get_phys svX_vpn0_get_phys
+
+#define sv39x4_vpn1_get_phys svX_vpn1_get_phys
+#define sv48x4_vpn1_get_phys svX_vpn1_get_phys
+
+#define sv39x4_vpn2_get_phys svX_vpn2_get_phys
+#define sv48x4_vpn2_get_phys svX_vpn2_get_phys
+
+#define sv48x4_vpn3_get_phys svX_vpn3_get_phys
+
+DEF_SET_TERMINAL(Xx4, RISCV_PTE_FLAG(U))
+
+DEF_SET_NEXT(Xx4, 0)
+
+static bool svXx4_root_page_table_empty(page_table_t page_table)
+{
+ return _svX_page_table_empty(page_table, 2 << (2 + PAGE_LEVEL_BITS));
+}
+
+#define RISCV_SVXx4_PAGING_LEVEL(WIDTH, LEVEL, ROOT) \
+ { \
+ 1UL << UNTRANSLATED_BITS(LEVEL), \
+ sv ## WIDTH ## x4_vpn ## LEVEL ## _get_entry, \
+ svX_entry_valid, \
+ svXx4_vpnX_set_terminal, \
+ sv ## WIDTH ## x4_vpn ## LEVEL ## _get_phys, \
+ svX_get_flags, \
+ svXx4_vpnX_set_next_pt, \
+ svX_vpnX_get_next_pt, \
+ svX_clear_entry, \
+ (ROOT)? svXx4_root_page_table_empty: \
+ svX_page_table_empty, \
+ }
+
+/* sequence is from root to leaves */
+const struct paging riscv_Sv39x4[] = {
+ RISCV_SVXx4_PAGING_LEVEL(39, 2, true),
+ RISCV_SVXx4_PAGING_LEVEL(39, 1, false),
+ RISCV_SVXx4_PAGING_LEVEL(39, 0, false),
+};
+
+const struct paging riscv_Sv48x4[] = {
+ RISCV_SVXx4_PAGING_LEVEL(48, 3, true),
+ RISCV_SVXx4_PAGING_LEVEL(48, 2, false),
+ RISCV_SVXx4_PAGING_LEVEL(48, 1, false),
+ RISCV_SVXx4_PAGING_LEVEL(48, 0, false),
+};

void arch_paging_init(void)
{
+ /*
+ * Basically, any MMU mode can be used here. Let's choose SV39 for two
+ * reasons:
+ * - It only requires three levels for 4K pages, whereas SV57 requires
+ * five levels
+ * - In Jailhouse, we typically use 1:1 mappings, and any real
+ * hardware should be mapable with a SV39. No need for SV48 or SV57.
+ * Guests may still use whatever they want.
+ *
+ * Same arguments apply for cell paging (G-stage paging).
+ */
+ hv_paging_structs.root_paging = HV_PAGING;
+ hv_atp_mode = HV_ATP_MODE;
+ cell_atp_mode = CELL_ATP_MODE;
}

-// Might be misplaced
-unsigned long arch_paging_gphys2phys(unsigned long gphys, unsigned long flags)
+unsigned long arch_paging_gphys2phys (unsigned long gphys, unsigned long flags)
{
+ return paging_virt2phys(&this_cell()->arch.mm, gphys, flags);
+}

+
+void riscv_paging_vcpu_init(struct paging_structures *pg_structs)

+{
+ unsigned long table;
+
+ table = paging_hvirt2phys(pg_structs->root_table);
+ enable_mmu_hgatp(ATP_MODE_SV39, table);
+}
+
+int riscv_paging_cell_init(struct cell *cell)
+{
+ cell->arch.mm.hv_paging = 0;
+ cell->arch.mm.root_paging = CELL_PAGING;
+
+ cell->arch.mm.root_table =
+ page_alloc_aligned(&mem_pool, CELL_ROOT_PT_PAGES);
+ if (!cell->arch.mm.root_table)
+ return -ENOMEM;
+
return 0;
}
diff --git a/hypervisor/include/jailhouse/paging.h b/hypervisor/include/jailhouse/paging.h
index d592abad..8841b09e 100644
--- a/hypervisor/include/jailhouse/paging.h
+++ b/hypervisor/include/jailhouse/paging.h
@@ -96,7 +96,7 @@ typedef pt_entry_t page_table_t;
struct paging {
/** Page size of terminal entries in this level or 0 if none are
* supported. */
- unsigned int page_size;
+ unsigned long page_size;

/**
* Get entry in given table corresponding to virt address.
--
2.36.1

[Ralf Ramsauer]

---
hypervisor/arch/riscv/control.c | 213 ++++++++++++++++++--
hypervisor/arch/riscv/include/asm/control.h | 13 ++
2 files changed, 209 insertions(+), 17 deletions(-)

diff --git a/hypervisor/arch/riscv/control.c b/hypervisor/arch/riscv/control.c
index 263708a7..fa9921e6 100644
--- a/hypervisor/arch/riscv/control.c
+++ b/hypervisor/arch/riscv/control.c
@@ -2,70 +2,230 @@

* Jailhouse, a Linux-based partitioning hypervisor
*
* Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.
*/

#include <jailhouse/control.h>
+#include <jailhouse/paging.h>
#include <jailhouse/printk.h>
+#include <jailhouse/string.h>
+#include <asm/control.h>
+#include <asm/csr64.h>
#include <asm/sbi.h>

-int arch_cell_create(struct cell *cell)
+extern void __attribute__((noreturn)) riscv_park_loop(void);
+
+static void riscv_cpu_reset(unsigned long pc, unsigned long a0, unsigned long a1)
+{
+ union registers *regs;
+
+ regs = &this_cpu_data()->guest_regs;
+
+ memset(regs, 0, sizeof(union registers));
+ regs->pc = pc;
+ regs->a0 = a0;
+ regs->a1 = a1;
+
+ /* TBD: Check if we need to clear more registers */
+ csr_write(sepc, regs->pc);
+ csr_write(CSR_VSATP, 0);
+ csr_write(CSR_HIE, 0);
+ csr_write(CSR_HIP, 0);
+ csr_write(CSR_VSSTATUS, 0);
+
+ csr_set(sstatus, SR_SPP); /* Return to VS-Mode */
+}
+
+void arch_check_events(void)
{
- return -ENOSYS;
+ struct public_per_cpu *pcpu;
+
+ pcpu = this_cpu_public();
+
+ spin_lock(&pcpu->control_lock);
+
+ if (pcpu->suspend_cpu) {
+ pcpu->cpu_suspended = true;
+
+ spin_unlock(&pcpu->control_lock);
+
+ while (pcpu->suspend_cpu)
+ cpu_relax();
+
+ spin_lock(&pcpu->control_lock);
+ }
+
+ pcpu->cpu_suspended = false;
+
+ if (pcpu->wait_for_power_on) {
+ pcpu->wait_for_power_on = false;
+ goto out;
+ }
+
+ if (pcpu->hsm.state == START_PENDING) { /* We could also check if START_PENDING is set... */
+ pcpu->reset = false;
+ riscv_cpu_reset(pcpu->hsm.start_addr, pcpu->phys_id, pcpu->hsm.opaque);
+ riscv_paging_vcpu_init(&this_cell()->arch.mm);
+ pcpu->hsm.state = STARTED;
+ } else if (pcpu->park) {
+ riscv_park_cpu();
+ }
+
+ if (pcpu->reset) {
+ pcpu->reset = false;
+ riscv_cpu_reset(0x0, pcpu->phys_id, 0);
+ riscv_paging_vcpu_init(&this_cell()->arch.mm);
+ }
+
+out:
+ spin_unlock(&pcpu->control_lock);
}

int arch_map_memory_region(struct cell *cell,
const struct jailhouse_memory *mem)
{
- return -ENOSYS;
+ int result = paging_create(
+ &cell->arch.mm,
+ mem->flags & JAILHOUSE_MEM_COMM_REGION ?
+ paging_hvirt2phys (&cell->comm_page) : mem->phys_start,
+ mem->size, mem->virt_start,
+ /* guests require U access */
+ RISCV_PTE_FLAG(V) | RISCV_PTE_FLAG(U) |
+ (mem->flags & JAILHOUSE_MEM_READ ? RISCV_PTE_FLAG(R) : 0) |
+ (mem->flags & JAILHOUSE_MEM_WRITE ? RISCV_PTE_FLAG(W) : 0) |
+ (mem->flags & JAILHOUSE_MEM_EXECUTE ? RISCV_PTE_FLAG(X) : 0),
+ PAGING_COHERENT |
+ (mem->flags & JAILHOUSE_MEM_NO_HUGEPAGES ? 0 : PAGING_HUGE));
+
+ return result;
}

-int arch_unmap_memory_region(struct cell *cell,
- const struct jailhouse_memory *mem)
+int arch_unmap_memory_region(struct cell *const cell,
+ const struct jailhouse_memory *mem_original)
{
- return -ENOSYS;
+ return paging_destroy (&cell->arch.mm,
+ mem_original->virt_start,
+ mem_original->size,
+ PAGING_COHERENT);
}

-void arch_check_events(void)
+void arch_flush_cell_vcpu_caches(struct cell *const cell)
{
+ /* the necessary TLB invalidation has already been performed
+ in the map/unmap routines */
+ /* doing it here would require the entire cell's TLB to be
+ flushed, because this function does not receive information
+ about the memory segment to invalidate. That would be overkill
+ (although the current unmap_region implementation does this
+ nonetheless, owing to other API shortcomings).
+ */
}

-void arch_flush_cell_vcpu_caches(struct cell *cell)
+int arch_cell_create(struct cell *const cell)
{
+ struct public_per_cpu *ppc;

+ unsigned int cpu;
+

+ cell->arch.mm.root_paging = riscv_Sv39x4;
+ cell->arch.mm.root_table = page_alloc_aligned(&mem_pool, CELL_ROOT_PT_PAGES);
+
+ for_each_cpu(cpu, &cell->cpu_set) {
+ ppc = public_per_cpu(cpu);
+ ppc->wait_for_power_on = false;
+ ppc->park = true;
+ ppc->reset = false;
+ }
+

+ if (!cell->arch.mm.root_table)
+ return -ENOMEM;
+

+ return 0;
}

-void arch_cell_destroy(struct cell *cell)
+void arch_cell_destroy(struct cell *const cell)
{
+ unsigned int cpu;
+ struct public_per_cpu *ppc;
+
+ page_free(&mem_pool, cell->arch.mm.root_table, CELL_ROOT_PT_PAGES);
+
+ for_each_cpu(cpu, &cell->cpu_set) {
+ ppc = public_per_cpu(cpu);
+ ppc->wait_for_power_on = false;
+ ppc->park = true;
+ ppc->reset = false;
+ }
}

-void arch_cell_reset(struct cell *cell)
+void arch_cell_reset(struct cell *const cell)
{
+ unsigned int first = first_cpu(&cell->cpu_set);
+ struct public_per_cpu *pcpu;
+ unsigned int cpu;
+
+ /*
+ * All CPUs except the first one shall not be started, they shall park
+ */
+ pcpu = public_per_cpu(first);
+ pcpu->wait_for_power_on = false;
+ pcpu->park = false;
+ for_each_cpu_except(cpu, &cell->cpu_set, first)
+ public_per_cpu(cpu)->wait_for_power_on = true;
}

-void arch_prepare_shutdown(void)
+void arch_reset_cpu(unsigned int const cpu_id)
{
+ public_per_cpu(cpu_id)->reset = true;
+
+ resume_cpu(cpu_id);
}

-void __attribute__((noreturn)) arch_panic_stop(void)
+void arch_park_cpu(unsigned int const cpu_id)
{
- while (1);
+ struct public_per_cpu *pc = public_per_cpu(cpu_id);
+
+ if (pc->hsm.state == STOPPED)
+ return;
+
+ if (pc->cpu_suspended == true)
+ return;
+
+ spin_lock(&pc->control_lock);
+ pc->park = true;
+ arch_send_event(pc);
+ spin_unlock(&pc->control_lock);
+
+ while (pc->hsm.state != STOPPED)
+ cpu_relax();
+
}

-void arch_panic_park(void)
+void arch_prepare_shutdown(void)
{
}

-void arch_reset_cpu(unsigned int const cpu_id)
+void __attribute__((noreturn)) arch_panic_stop(void)
{
+ /* No need to check return code here */
+ sbi_hart_stop();
+
+ /*
+ * If this happens, which should never be the case, then let the CPU
+ * execute the park loop.
+ */
+ riscv_park_loop();
}

-void arch_park_cpu(unsigned int cpu_id)
+void arch_panic_park(void)
{
+ riscv_park_cpu();
}

void arch_send_event(struct public_per_cpu *target_data)
@@ -83,3 +243,22 @@ void arch_send_event(struct public_per_cpu *target_data)
panic_stop();
}
}
+
+void riscv_park_cpu(void)
+{
+ struct public_per_cpu *pcpu = this_cpu_public();
+
+ pcpu->hsm.state = STOPPED;
+ /*
+ * BUG FIXME: The Timer IRQ might be pending, and we're
+ * effectively busy waiting here. Apparently, Linux doesn't
+ * shut down the timer before offlining the CPU. Check this!
+ *
+ * Second, do we need to manually disable all external PLIC IRQs of the
+ * cell? Actually, yes.
+ */
+ sbi_set_timer(-1);
+
+ riscv_paging_vcpu_init(&parking_pt);
+ riscv_cpu_reset(0, 0, 0);
+}
diff --git a/hypervisor/arch/riscv/include/asm/control.h b/hypervisor/arch/riscv/include/asm/control.h
index e69de29b..466311f8 100644
--- a/hypervisor/arch/riscv/include/asm/control.h
+++ b/hypervisor/arch/riscv/include/asm/control.h
@@ -0,0 +1,13 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+void riscv_park_cpu(void);
--
2.36.1

[Ralf Ramsauer]

We need this for RISC-V. Makes life easier.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

driver/main.c | 1 +
hypervisor/include/jailhouse/header.h | 2 ++
2 files changed, 3 insertions(+)

diff --git a/driver/main.c b/driver/main.c
index ef1c2a90..d5c823f2 100644
--- a/driver/main.c
+++ b/driver/main.c
@@ -513,6 +513,7 @@ static int jailhouse_cmd_enable(struct jailhouse_system __user *arg)

header = (struct jailhouse_header *)hypervisor_mem;
header->max_cpus = config_header.root_cell.num_cpus;
+ header->initial_load_address = (unsigned long)hypervisor_mem;

#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
header->arm_linux_hyp_vectors = virt_to_phys(*__hyp_stub_vectors_sym);
diff --git a/hypervisor/include/jailhouse/header.h b/hypervisor/include/jailhouse/header.h
index c5285262..574aae3f 100644
--- a/hypervisor/include/jailhouse/header.h
+++ b/hypervisor/include/jailhouse/header.h
@@ -77,6 +77,8 @@ struct jailhouse_header {
* @note Filled at build time */
void *gcov_info_head;

+ unsigned long initial_load_address;
+
/** Configured maximum logical CPU ID + 1.
* @note Filled by Linux loader driver before entry. */
unsigned int max_cpus;
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/spinlock.h | 52 +++++++++++++++++++-
1 file changed, 50 insertions(+), 2 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/spinlock.h b/hypervisor/arch/riscv/include/asm/spinlock.h
index a5e707d7..db9b33bc 100644
--- a/hypervisor/arch/riscv/include/asm/spinlock.h
+++ b/hypervisor/arch/riscv/include/asm/spinlock.h
@@ -4,7 +4,7 @@

* Copyright (c) Siemens AG, 2020

*
* Authors:
- * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.

@@ -13,15 +13,63 @@
#ifndef _JAILHOUSE_ASM_SPINLOCK_H
#define _JAILHOUSE_ASM_SPINLOCK_H

+#define RISCV_USE_LR_SC 1
+
typedef struct {
+ int unsigned spin; /* has to have offset 0 */
} spinlock_t;

-static inline void spin_lock(spinlock_t *lock)
+static inline void spin_init(spinlock_t *lock)
{
+ lock->spin = 0;
}

static inline void spin_unlock(spinlock_t *lock)
{
+ __asm__ ("\n\
+ .if %[use_lr_sc]\n\
+ fence rw, w\n\
+ sw zero, %[spin]\n\
+ .else\n\
+ amoswap.w.rl x0, x0, %[spin]\n\
+ .endif\n"
+ : [spin] "=&m" (lock->spin):
+ [use_lr_sc] "n" (RISCV_USE_LR_SC):
+ "memory");
+}
+
+static inline void spin_lock(spinlock_t *lock)
+{
+ /* test and test and set */
+ __asm__ ("\n\
+ .if %[use_lr_sc]\n\
+\n\
+ la t2, 1\n\
+\n\
+1: lw t1, %[spin]\n\
+ bnez t1, 1b\n\
+\n\
+2: lr.w.aq t1, %[spin]\n\
+ bnez t1, 1b\n\
+ sc.w t1, t2, %[spin]\n\
+ bnez t1, 1b\n\
+\n\
+ .else\n\
+\n\
+ # see figure 8.2 in the RISC-V Unprivileged ISA\n\
+ li t2, 1\n\
+\n\
+3: lw t1, %[spin]\n\
+ bnez t1, 3b\n\
+\n\
+ amoswap.w.aq t1, t2, %[spin]\n\
+ bnez t1, 3b\n\
+\n\
+ .endif\n\
+" :
+ [spin] "=&m" (lock->spin):
+ [use_lr_sc] "n" (RISCV_USE_LR_SC):
+ "t1", "t2", "memory");
}

#endif /* !_JAILHOUSE_ASM_SPINLOCK_H */
--
2.36.1

[Ralf Ramsauer]

Same demo as for other architectures.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

inmates/demos/riscv/Makefile | 3 +-
inmates/demos/riscv/timer-demo.c | 57 +++++++++++++++
inmates/lib/riscv/Makefile | 2 +-
inmates/lib/riscv/header.S | 29 ++++++++
inmates/lib/riscv/include/inmate.h | 27 +++++++-
inmates/lib/riscv/inmate.lds.S | 3 +
inmates/lib/riscv/irq.c | 107 +++++++++++++++++++++++++++++
7 files changed, 225 insertions(+), 3 deletions(-)
create mode 100644 inmates/demos/riscv/timer-demo.c
create mode 100644 inmates/lib/riscv/irq.c

diff --git a/inmates/demos/riscv/Makefile b/inmates/demos/riscv/Makefile
index 2f0bac84..e371a29b 100644
--- a/inmates/demos/riscv/Makefile
+++ b/inmates/demos/riscv/Makefile
@@ -12,8 +12,9 @@

include $(INMATES_LIB)/Makefile.lib

-INMATES := tiny-demo.bin
+INMATES := tiny-demo.bin timer-demo.bin

tiny-demo-y := tiny-demo.o
+timer-demo-y := timer-demo.o

$(eval $(call DECLARE_TARGETS,$(INMATES)))
diff --git a/inmates/demos/riscv/timer-demo.c b/inmates/demos/riscv/timer-demo.c
new file mode 100644
index 00000000..748219b7
--- /dev/null
+++ b/inmates/demos/riscv/timer-demo.c
@@ -0,0 +1,57 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *

+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <inmate.h>
+
+#include <jailhouse/hypercall.h>
+
+#define TIMEBASE_FREQ 0x989680
+#define TIMER_DELTA (TIMEBASE_FREQ / 6)
+
+unsigned long timer_next;
+
+static inline u64 timer_ticks_to_ns(u64 ticks)
+{
+ return (1000UL * 1000UL * 1000UL * ticks) / TIMEBASE_FREQ;
+}
+
+unsigned long timer_handler(void)
+{
+ static u64 min_delta = ~0ULL, max_delta = 0;
+ unsigned long delta;
+
+ delta = get_cycles() - timer_next;
+ if (delta < min_delta)
+ min_delta = delta;
+ if (delta > max_delta)
+ max_delta = delta;
+
+ printk("Timer fired, jitter: %6ld ns, min: %6ld ns, max: %6ld ns\n",
+ (long)timer_ticks_to_ns(delta),
+ (long)timer_ticks_to_ns(min_delta),
+ (long)timer_ticks_to_ns(max_delta));
+
+ timer_next += TIMER_DELTA;
+
+ return timer_next;
+}
+
+void inmate_main(void)
+{
+ printk("Initializing the timer...\n");
+ timer_next = get_cycles() + 0x400000;
+ sbi_set_timer(timer_next);
+ timer_enable();
+ enable_irqs();
+
+ halt();
+}
diff --git a/inmates/lib/riscv/Makefile b/inmates/lib/riscv/Makefile
index 9ff57721..3b796dc3 100644
--- a/inmates/lib/riscv/Makefile
+++ b/inmates/lib/riscv/Makefile
@@ -41,6 +41,6 @@ include $(INMATES_LIB)/Makefile.lib
always-y := lib.a inmate.lds

lib-y := $(common-objs-y)
-lib-y += header.o printk.o setup.o uart.o
+lib-y += header.o irq.o printk.o setup.o uart.o

lib-y += ../cmdline.o ../printk.o ../setup.o ../string.o ../uart-8250.o
diff --git a/inmates/lib/riscv/header.S b/inmates/lib/riscv/header.S
index 3a12f51f..5ddffe79 100644
--- a/inmates/lib/riscv/header.S
+++ b/inmates/lib/riscv/header.S
@@ -36,9 +36,38 @@

* THE POSSIBILITY OF SUCH DAMAGE.
*/

+#define STRUCT_REGISTERS_SIZE (32 * 8)
+
+.macro context cmd

+ .irp reg 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, \
+ 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

+ \cmd x\reg, 8*(\reg - 1)(sp)
+ .endr
+.endm
+
.section ".boot", "ax"

.globl __reset_entry
__reset_entry:
la sp, __stack_top
+ addi sp, sp, -STRUCT_REGISTERS_SIZE
+
+ la a5 , __ex_stack_top
+ csrw sscratch, a5
+
+ la a5, exception_handler
+ csrw stvec, a5
+
j c_entry
+
+.text
+.align 4

+exception_handler:
+ csrrw sp, sscratch, sp

+ context sd
+
+ jal arch_handle_trap
+
+ context ld

+ csrrw sp, sscratch, sp
+ sret

diff --git a/inmates/lib/riscv/include/inmate.h b/inmates/lib/riscv/include/inmate.h
index dac9f146..57338d25 100644
--- a/inmates/lib/riscv/include/inmate.h
+++ b/inmates/lib/riscv/include/inmate.h
@@ -1,7 +1,7 @@
/*

* Jailhouse, a Linux-based partitioning hypervisor
*

- * Copyright (c) OTH Regensburg

+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:

* Ralf Ramsauer <ralf.r...@oth-regensburg.de>
@@ -60,6 +60,9 @@ typedef unsigned long long u64;

#define SR_SIE 0x00000002UL

+#define IRQ_S_TIMER 5
+#define IE_TIE (0x1UL << IRQ_S_TIMER)
+
#define csr_read(csr) \
({ \
register unsigned long __v; \
@@ -93,11 +96,31 @@ typedef unsigned long long u64;
: "memory"); \
})

+static inline unsigned long get_cycles(void)
+{
+ return csr_read(time);
+}
+
static inline void disable_irqs(void)
{
csr_clear(sstatus, SR_SIE);
}

+static inline void enable_irqs(void)
+{
+ csr_set(sstatus, SR_SIE);
+}
+
+static inline void timer_enable(void)
+{
+ csr_set(sie, IE_TIE);
+}
+
+static inline void timer_disable(void)
+{
+ csr_clear(sie, IE_TIE);
+}
+
static inline void cpu_relax(void)
{
int dummy;
@@ -152,6 +175,8 @@ static inline void mmio_write64(void *address, u64 value)
*(volatile u64 *)address = value;
}

+unsigned long timer_handler(void);
+
#include <inmate_common.h>

#endif /* !_JAILHOUSE_INMATE_H */
diff --git a/inmates/lib/riscv/inmate.lds.S b/inmates/lib/riscv/inmate.lds.S
index cb6e7a76..91bc8f22 100644
--- a/inmates/lib/riscv/inmate.lds.S
+++ b/inmates/lib/riscv/inmate.lds.S
@@ -69,6 +69,9 @@ SECTIONS {
. = ALIGN(4096);
. = . + 0x1000;
__stack_top = .;
+
+ . = . + 0x1000;
+ __ex_stack_top = .;
}

ENTRY(__reset_entry)
diff --git a/inmates/lib/riscv/irq.c b/inmates/lib/riscv/irq.c
new file mode 100644
index 00000000..8f6d0085
--- /dev/null
+++ b/inmates/lib/riscv/irq.c
@@ -0,0 +1,107 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *

+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <inmate.h>
+
+#define CAUSE_IRQ_FLAG (1UL << (__riscv_xlen - 1))
+
+void arch_handle_trap(void);
+
+static inline bool is_irq(u64 cause)
+{
+ return !!(cause & CAUSE_IRQ_FLAG);
+}
+
+static inline unsigned long to_irq(unsigned long cause)
+{
+ return cause & ~CAUSE_IRQ_FLAG;
+}
+
+static int handle_irq(unsigned int irq)
+{
+ int err;
+ struct sbiret ret;
+ unsigned long tval;
+
+ switch (irq) {
+ case IRQ_S_TIMER:
+ tval = timer_handler();
+ ret = sbi_set_timer(tval);
+ err = ret.error;
+ break;
+
+ default:
+ err = -1;
+ break;
+ }
+ return err;
+}
+
+/*
+ * Any positive value will reprogramm the timer, -1 will halt the timer.
+ */
+unsigned long __attribute__((weak)) timer_handler(void)
+{
+ return -1;
+}
+
+void arch_handle_trap(void)
+{
+ unsigned long scause = csr_read(scause);
+ int err;
+
+ if (is_irq(scause)) {
+ err = handle_irq(to_irq(scause));

+ goto out;
+ }
+

+ switch (scause) {
+ default:
+ /* We don't have any exception handlers at the moment */
+ printk("Unhandled exception %lu occured.\n", scause);
+ err = -1;
+ break;
+ }
+
+out:
+ if (err) {
+ printk("FATAL INMATE ERROR. HALTED.\n");
+ stop();
+
+ }
+}
--
2.36.1

[Ralf Ramsauer]

We need this macro on RISC-V.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/arm-common/irqchip.c | 5 -----
hypervisor/include/jailhouse/control.h | 5 +++++
2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/hypervisor/arch/arm-common/irqchip.c b/hypervisor/arch/arm-common/irqchip.c
index 59e5dd53..c93117d1 100644
--- a/hypervisor/arch/arm-common/irqchip.c
+++ b/hypervisor/arch/arm-common/irqchip.c
@@ -24,11 +24,6 @@
#include <asm/irqchip.h>
#include <asm/smccc.h>

-#define for_each_irqchip(chip, config, counter) \
- for ((chip) = jailhouse_cell_irqchips(config), (counter) = 0; \
- (counter) < (config)->num_irqchips; \
- (chip)++, (counter)++)
-
spinlock_t dist_lock;

void *gicd_base;
diff --git a/hypervisor/include/jailhouse/control.h b/hypervisor/include/jailhouse/control.h
index 2e2446f0..ddb9d039 100644
--- a/hypervisor/include/jailhouse/control.h
+++ b/hypervisor/include/jailhouse/control.h
@@ -106,6 +106,11 @@ unsigned int next_cpu(unsigned int cpu, struct cpu_set *cpu_set,
(counter) < (config)->num_memory_regions; \
(mem)++, (counter)++)

+#define for_each_irqchip(chip, config, counter) \
+ for ((chip) = jailhouse_cell_irqchips(config), (counter) = 0; \
+ (counter) < (config)->num_irqchips; \
+ (chip)++, (counter)++)
+
/**
* Check if the CPU is assigned to the specified cell.
* @param cell Cell the CPU may belong to.
--
2.36.1

[Ralf Ramsauer]

And do some heavy liftig.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/setup.c | 198 ++++++++++++++++++++++++++++++++--
1 file changed, 190 insertions(+), 8 deletions(-)

diff --git a/hypervisor/arch/riscv/setup.c b/hypervisor/arch/riscv/setup.c
index 985f27e2..dadffc70 100644
--- a/hypervisor/arch/riscv/setup.c
+++ b/hypervisor/arch/riscv/setup.c
@@ -2,37 +2,219 @@

* Jailhouse, a Linux-based partitioning hypervisor
*

* Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022
*
* Authors:

* Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

*/

+#include <jailhouse/control.h>
#include <jailhouse/entry.h>
-#include <asm/processor.h>
+#include <jailhouse/paging.h>
+#include <jailhouse/percpu.h>
+#include <asm/csr64.h>
#include <asm/setup.h>

+extern unsigned long bt_tbl_l0[PAGE_SIZE / sizeof(unsigned long)];
+
+void riscv_park_loop(void);
+
int arch_init_early(void)
{
- return -ENOSYS;
+ int err;
+
+ err = riscv_paging_cell_init(&root_cell);
+ if (err)
+ return err;
+
+ parking_pt.root_paging = root_cell.arch.mm.root_paging;
+
+ err = paging_create(&parking_pt, paging_hvirt2phys(riscv_park_loop),
+ PAGE_SIZE, 0, PAGE_DEFAULT_FLAGS | RISCV_PTE_FLAG(G) | RISCV_PTE_FLAG(U),
+ PAGING_COHERENT | PAGING_NO_HUGE);
+
+ return 0;
}

int arch_cpu_init(struct per_cpu *cpu_data)
{
- return -ENOSYS;
+ struct public_per_cpu *ppc = &cpu_data->public;
+ unsigned long final_pt;
+
+ spin_init(&ppc->control_lock);
+

+ ppc->reset = false;

+ ppc->park = false;

+ ppc->wait_for_power_on = false;
+

+ ppc->phys_id =
+ jailhouse_cell_cpus(root_cell.config)[cpu_data->public.cpu_id]
+ .phys_id;
+ ppc->hsm.state = STARTED;
+
+ final_pt = paging_hvirt2phys(&ppc->root_table_page);
+ enable_mmu_satp(hv_atp_mode, final_pt);
+
+ return 0;
}

-void __attribute__((noreturn))
-riscv_deactivate_vmm(union registers *regs, int errcode)
+void __attribute__ ((noreturn)) arch_cpu_activate_vmm(void)
{
- while (1);
+ union registers *regs;
+ unsigned long tmp;

+
+ regs = &this_cpu_data()->guest_regs;
+

+ /* VSBE = 0 -> VS-Mode mem accesses are LE */
+ csr_write(CSR_HSTATUS,
+ HSTATUS_SPV | /* Return to VS-Mode */
+ (2ULL << HSTATUS_VSXL_SHIFT)); /* xlen = 64 */
+
+ csr_write(CSR_HEDELEG,
+ (1UL << EXC_INST_MISALIGNED) |
+ (1UL << EXC_INST_ACCESS) |
+ (1UL << EXC_INST_ILLEGAL) |
+ (1UL << EXC_BREAKPOINT) |
+ (1UL << EXC_LOAD_ACCESS_MISALIGNED) |
+ (1UL << EXC_LOAD_ACCESS) |
+ (1UL << EXC_AMO_ADDRESS_MISALIGNED) |
+ (1UL << EXC_STORE_ACCESS) |
+ (1UL << EXC_SYSCALL) |
+ (1UL << EXC_INST_PAGE_FAULT) |
+ (1UL << EXC_LOAD_PAGE_FAULT) |
+ (1UL << EXC_STORE_PAGE_FAULT));
+
+ csr_write(CSR_HGEIE, 0);
+ csr_write(CSR_HCOUNTEREN, SCOUNTEREN_TM);
+ csr_write(CSR_HTIMEDELTA, 0);
+
+ tmp = csr_read(sip);
+ csr_write(sip, tmp); /* clear pending */
+ csr_write(CSR_HVIP, tmp); /* reinject pending */
+
+ riscv_paging_vcpu_init(&this_cell()->arch.mm);
+
+ /* Return value */
+ regs->a0 = 0;
+
+ csr_write(sepc, regs->ra); /* We will use sret, so move ra->sepc */
+
+ tmp = csr_swap(sscratch, regs->sp);
+ asm volatile("mv sp, %0\n"
+ "j vmreturn\n" : : "r"(tmp));
+
+ __builtin_unreachable();
}

-void __attribute__((noreturn)) arch_cpu_activate_vmm(void)
+static unsigned long symbol_offset(const void *addr)
{
- while (1);
+ return (unsigned long)addr - (unsigned long)&hypervisor_header;
+}
+
+void __attribute__((noreturn)) riscv_deactivate_vmm(union registers *regs, int errcode)
+{
+ unsigned long linux_tables_offset, bootstrap_table_phys;
+ register union registers *_regs;
+ register int _errcode;
+ u8 atp_mode;
+
+ linux_tables_offset = symbol_offset((void*)hypervisor_header.initial_load_address);
+
+ /* Do not return to VS-mode, rather return to S-Mode */
+ csr_clear(CSR_HSTATUS, HSTATUS_SPV);
+
+ /*
+ * We don't know which page table is currently active. So in any case,
+ * just jump back to the bootstrap tables, as they contain the old
+ * Linux mapping of Jailhouse
+ */
+ bootstrap_table_phys = system_config->hypervisor_memory.phys_start +
+ symbol_offset(&bt_tbl_l0);
+ /* Take Linux's MMU mode */
+ atp_mode = csr_read(CSR_VSATP) >> ATP_MODE_SHIFT;
+ enable_mmu_satp(atp_mode, bootstrap_table_phys);
+
+ /* next access to regs will be under Linux's old page table, so amend the address */
+ _regs = (void*)regs + linux_tables_offset;
+ _errcode = errcode;
+
+ /* Before switching back, we need to jump to original load location */
+ asm volatile(
+ "add sp, sp, %0\n" /* Get stack under control */
+ "jalr zero, 0(%1)\n" /* Jump to pre-virtual paging location */
+ :
+ : "r"(linux_tables_offset),
+ "r"(&&linux_tables + linux_tables_offset)
+ :);
+linux_tables:
+ /*
+ * From now on, we can safely access stack variables, but we must not
+ * use any absolute addresses
+ */
+ csr_from_csr(satp, CSR_VSATP);
+ asm volatile("sfence.vma");
+ /* From here on, Linux's paging is active. */
+
+ /* Restore S-mode CSRs from VS-mode */
+ csr_from_csr(stval, CSR_VSTVAL);
+ csr_from_csr(scause, CSR_VSCAUSE);
+ csr_from_csr(sscratch, CSR_VSSCRATCH);
+ csr_from_csr(sie, CSR_VSIE);
+ csr_from_csr(stvec, CSR_VSTVEC);
+ csr_from_csr(sstatus, CSR_VSSTATUS);
+
+ /*
+ * We came from an ecall, so we may clobber a0-a7. That's just fine, we
+ * can use them as scratch.
+ */
+ asm volatile(
+ /* scratch setup */
+ "mv a2, %0\n" /* holds registers */
+ "csrr a1, sepc\n" /* a1 hold return address */
+ "addi a1, a1, 4\n" /* we came from an ecall */
+
+ /* restore original sepc */
+ //"csrr a0, csr_vsepc\n"
+ "csrr a0, %2\n"
+ "csrw sepc, a0\n"
+
+ /* set return code */
+ "mv a0, %1\n"
+
+ /* restore registers */
+ "ld ra, 8(a2)\n"
+ "ld sp, 16(a2)\n"
+ "ld gp, 24(a2)\n"
+ "ld tp, 32(a2)\n"
+ "ld t0, 40(a2)\n"
+ "ld t1, 48(a2)\n"
+ "ld t2, 56(a2)\n"
+ "ld s0, 64(a2)\n"
+ "ld s1, 72(a2)\n"
+ /* Skip clobbers a0 - a7 */
+ "ld s2, 144(a2)\n"
+ "ld s3, 152(a2)\n"
+ "ld s4, 160(a2)\n"
+ "ld s5, 168(a2)\n"
+ "ld s6, 176(a2)\n"
+ "ld s7, 184(a2)\n"
+ "ld s8, 192(a2)\n"
+ "ld s9, 200(a2)\n"
+ "ld s10, 208(a2)\n"
+ "ld s11, 216(a2)\n"
+ "ld t3, 224(a2)\n"
+ "ld t4, 232(a2)\n"
+ "ld t5, 240(a2)\n"
+ "ld t6, 248(a2)\n"
+ /* And we're done. */
+ "jalr zero, a1, 0\n"
+ :: "r"(_regs), "r"(_errcode), "i"(CSR_VSEPC) :);
+
+ __builtin_unreachable();
}

void arch_cpu_restore(unsigned int cpu_id, int return_code)
--
2.36.1

[Ralf Ramsauer]

The boot protocol is simple:
- a0 hold the hart id
- a1 hold the DTB

That's it.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

inmates/tools/riscv/Makefile | 19 +++++++++++++++++++
inmates/tools/riscv/linux-loader.c | 25 +++++++++++++++++++++++++
2 files changed, 44 insertions(+)
create mode 100644 inmates/tools/riscv/linux-loader.c

diff --git a/inmates/tools/riscv/Makefile b/inmates/tools/riscv/Makefile
index e69de29b..4a722777 100644
--- a/inmates/tools/riscv/Makefile
+++ b/inmates/tools/riscv/Makefile
@@ -0,0 +1,19 @@
+#
+# Jailhouse, a Linux-based partitioning hypervisor
+#
+# Copyright (c) Siemens AG, 2013-2015
+#
+# Authors:
+# Jan Kiszka <jan.k...@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2. See
+# the COPYING file in the top-level directory.
+#
+
+include $(INMATES_LIB)/Makefile.lib
+
+INMATES := linux-loader.bin
+
+linux-loader-y := linux-loader.o
+
+$(eval $(call DECLARE_TARGETS,$(INMATES)))
diff --git a/inmates/tools/riscv/linux-loader.c b/inmates/tools/riscv/linux-loader.c
new file mode 100644
index 00000000..1434cb0a
--- /dev/null
+++ b/inmates/tools/riscv/linux-loader.c
@@ -0,0 +1,25 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ * Stefan Huber <stefan...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <inmate.h>
+

+void inmate_main(void)
+{
+ void (*entry)(u64 hartid, u64 dtb);
+ unsigned long dtb;
+
+ entry = (void *)cmdline_parse_int("kernel", 0);
+ dtb = cmdline_parse_int("dtb", 0);
+
+ entry(hart_id, dtb);
+}
--
2.36.1

[Ralf Ramsauer]

We will need the hartid for the linux-loader later.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

inmates/lib/riscv/header.S | 3 +++
inmates/lib/riscv/include/inmate.h | 2 ++
inmates/lib/riscv/setup.c | 2 ++
3 files changed, 7 insertions(+)

diff --git a/inmates/lib/riscv/header.S b/inmates/lib/riscv/header.S
index 5ddffe79..e4b21ce5 100644
--- a/inmates/lib/riscv/header.S
+++ b/inmates/lib/riscv/header.S
@@ -58,6 +58,9 @@ __reset_entry:
la a5, exception_handler
csrw stvec, a5

+ la a5, hart_id
+ sd a0, 0(a5)
+
j c_entry

.text
diff --git a/inmates/lib/riscv/include/inmate.h b/inmates/lib/riscv/include/inmate.h
index 57338d25..468d2218 100644
--- a/inmates/lib/riscv/include/inmate.h
+++ b/inmates/lib/riscv/include/inmate.h
@@ -58,6 +58,8 @@ typedef unsigned int u32;
typedef signed long long s64;

typedef unsigned long long u64;

+extern unsigned long hart_id;
+
#define SR_SIE 0x00000002UL

#define IRQ_S_TIMER 5
diff --git a/inmates/lib/riscv/setup.c b/inmates/lib/riscv/setup.c
index 705c5630..a87e8c50 100644
--- a/inmates/lib/riscv/setup.c
+++ b/inmates/lib/riscv/setup.c
@@ -38,6 +38,8 @@

#include <inmate.h>

+unsigned long hart_id;
+
void arch_init_early(void)
{
}
--
2.36.1

[Ralf Ramsauer]

The design of the PLIC is poor: We basically need to trap & moderate
each access.

The strategy is as follows:

On IRQ arrival in S-Mode, we directly acknowledge the IRQ, save it in a
shadow register, and reinject it to the VS-Mode guest. Now disable IRQs
for S-Mode, until the guest has claimed and acknowledged the IRQ.

After the guest acknowledged the IRQ, reenable IRQs in S-Mode again.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/Kbuild | 2 +-
hypervisor/arch/riscv/include/asm/plic.h | 3 +
hypervisor/arch/riscv/plic.c | 576 +++++++++++++++++++++++
3 files changed, 580 insertions(+), 1 deletion(-)
create mode 100644 hypervisor/arch/riscv/plic.c

diff --git a/hypervisor/arch/riscv/Kbuild b/hypervisor/arch/riscv/Kbuild
index 7809007c..9fd2456f 100644
--- a/hypervisor/arch/riscv/Kbuild
+++ b/hypervisor/arch/riscv/Kbuild
@@ -15,4 +15,4 @@
always-y := lib.a

lib-y := entry.o exception.o setup.o dbg-write.o control.o ivshmem.o paging.o
-lib-y += pci.o traps.o lib.o
+lib-y += plic.o pci.o traps.o lib.o
diff --git a/hypervisor/arch/riscv/include/asm/plic.h b/hypervisor/arch/riscv/include/asm/plic.h
index 04cdfa63..c5414e9e 100644
--- a/hypervisor/arch/riscv/include/asm/plic.h
+++ b/hypervisor/arch/riscv/include/asm/plic.h
@@ -15,4 +15,7 @@

#define PLIC_MAX_IRQS 1024

+extern int plic_set_pending(void);
+bool irqchip_irq_in_cell(struct cell *cell, unsigned int irq);
+
#endif /* _PLIC_H */
diff --git a/hypervisor/arch/riscv/plic.c b/hypervisor/arch/riscv/plic.c
new file mode 100644
index 00000000..84f95c0b
--- /dev/null
+++ b/hypervisor/arch/riscv/plic.c
@@ -0,0 +1,576 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *

+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+

+#include <jailhouse/paging.h>
+#include <jailhouse/cell.h>
+#include <jailhouse/string.h>
+#include <jailhouse/unit.h>
+#include <jailhouse/control.h>
+#include <jailhouse/processor.h>
+#include <jailhouse/printk.h>
+#include <asm/csr64.h>
+
+#define PLIC_PRIO_BASE 0x0
+#define PLIC_PENDING_BASE 0x1000
+#define PLIC_ENABLE_BASE 0x2000
+#define PLIC_ENABLE_OFF 0x80
+#define PLIC_ENABLE_END 0x1f2000
+#define PLIC_CTX_BASE 0x200000
+#define PLIC_CTX_PRIO_TH 0x0
+#define PLIC_CTX_CLAIM 0x4
+#define PLIC_CTX_SZ 0x1000
+#define PLIC_CTX_END 0x4000000
+
+#define REG_SZ 4
+#define REG_RANGE(A, B) (A)...((B) - REG_SZ)
+
+#define PLIC_BITS_PER_REG (REG_SZ * 8)
+#define IRQ_BIT(irq) ((irq) % PLIC_BITS_PER_REG)
+#define IRQ_MASK(irq) (1 << IRQ_BIT(irq))
+
+
+/* Could also be used for arm-common/irqchip.c */
+#define IRQCHIP_PINS \
+ (sizeof(((struct jailhouse_irqchip *)0)->pin_bitmap) * 8)
+
+#define IRQ_BITMAP_PINS \
+ (sizeof(((struct cell *)0)->arch.irq_bitmap) * 8)
+
+static void *plic_base;
+static unsigned long pending[MAX_CPUS];
+
+static inline unsigned long plic_phys(void)
+{
+ return system_config->platform_info.riscv.plic.base_address;
+}
+
+static inline s16 hart_to_context(unsigned int hartid)
+{
+ if (hartid > ARRAY_SIZE(
+ system_config->platform_info.riscv.plic.hart_to_context))
+ return -1;
+
+ return system_config->platform_info.riscv.plic.hart_to_context[hartid];
+}
+
+static inline u16 plic_max_irq(void)
+{
+ return system_config->platform_info.riscv.plic.max_irq;
+}
+
+static inline u16 plic_max_priority(void)
+{
+ return system_config->platform_info.riscv.plic.max_priority;
+}
+
+static inline unsigned int plic_size(void)
+{
+ return system_config->platform_info.riscv.plic.size;
+}
+
+static inline void ext_enable(void)
+{
+ csr_set(sie, IE_EIE);
+}
+
+static inline void ext_disable(void)
+{
+ csr_clear(sie, IE_EIE);
+}
+
+static inline void guest_clear_ext(void)
+{
+ csr_clear(CSR_HVIP, (1 << IRQ_S_EXT) << VSIP_TO_HVIP_SHIFT);
+}
+
+static inline void guest_inject_ext(void)
+{
+ csr_set(CSR_HVIP, (1 << IRQ_S_EXT) << VSIP_TO_HVIP_SHIFT);
+}
+
+static inline bool guest_ext_pending(void)
+{
+ return !!(csr_read(CSR_HVIP) &
+ ((1 << IRQ_S_EXT) << VSIP_TO_HVIP_SHIFT));
+}
+
+static inline u32 plic_read(u32 reg)
+{
+ return mmio_read32(plic_base + reg);
+}
+
+static inline void plic_write(u32 reg, u32 value)
+{
+ mmio_write32(plic_base + reg, value);
+}
+
+static inline u32 plic_read_context(u32 context, u32 off)
+{
+ return plic_read(PLIC_CTX_BASE + context * PLIC_CTX_SZ + off);
+}
+
+static inline u32 plic_read_claim(u32 context)
+{
+ return plic_read_context(context, PLIC_CTX_CLAIM);
+}
+
+static inline u32 plic_en_reg(s16 context, unsigned int irq)
+{
+ u32 reg;
+
+ reg = PLIC_ENABLE_BASE + (context * PLIC_ENABLE_OFF) +
+ (irq / PLIC_BITS_PER_REG) * REG_SZ;
+
+ return reg;
+}
+
+static inline u32 plic_read_enabled(s16 context, unsigned int irq)
+{
+ return plic_read(plic_en_reg(context, irq));
+}
+
+static inline void plic_write_enabled(s16 context, unsigned int irq, u32 val)
+{
+ plic_write(plic_en_reg(context, irq), val);
+}
+
+static inline bool plic_irq_is_enabled(s16 context, unsigned int irq)
+{
+ u32 en = plic_read_enabled(context, irq);
+
+ return !!(en & IRQ_MASK(irq));
+}
+
+static inline void plic_enable_irq(s16 context, unsigned int irq)
+{
+ u32 val;
+
+ val = plic_read_enabled(context, irq) | IRQ_MASK(irq);
+ plic_write_enabled(context, irq, val);
+}
+
+static inline void plic_disable_irq(s16 context, unsigned int irq)
+{
+ u32 val;
+
+ val = plic_read_enabled(context, irq) & ~IRQ_MASK(irq);
+ plic_write_enabled(context, irq, val);
+}
+
+static bool irq_bitmap_test(u32 *bitmap, unsigned int irq)
+{
+ u32 val;
+
+ if (irq >= plic_max_irq())
+ return false;
+
+ val = bitmap[irq / PLIC_BITS_PER_REG];
+
+ return !!(val & IRQ_MASK(irq));
+}
+
+static inline void irq_bitmap_set(u32 *bitmap, unsigned int irq)
+{
+ bitmap[irq / PLIC_BITS_PER_REG] |= IRQ_MASK(irq);
+}
+
+static inline void irq_bitmap_clear(u32 *bitmap, unsigned int irq)
+{
+ bitmap[irq / PLIC_BITS_PER_REG] &= ~IRQ_MASK(irq);
+}
+
+inline bool irqchip_irq_in_cell(struct cell *cell, unsigned int irq)
+{
+ return irq_bitmap_test(cell->arch.irq_bitmap, irq);
+}
+
+int plic_set_pending(void)
+{
+ int my_context;
+ u32 irq;
+ unsigned int cpuid;
+
+ this_cpu_public()->stats[JAILHOUSE_CPU_STAT_VMEXITS_VIRQ]++;
+
+ cpuid = phys_processor_id();
+
+ /* Assume that phys_processor_id always returns something < 64 */
+ my_context = hart_to_context(cpuid);
+ if (my_context < 0)
+ return -ENOSYS;
+
+ irq = plic_read_claim(my_context);
+ if (irq == 0) /* spurious IRQ, should not happen */
+ return -EINVAL;
+
+ if (irq > plic_max_irq())
+ return -EINVAL;
+
+ pending[cpuid] = irq;
+ /*
+ * We can directly inject the IRQ into the guest if the IRQ is not
+ * pending, because we know that the IRQ is enabled, otherwise we
+ * wouldn't have received it
+ */
+ guest_inject_ext();
+
+ /*
+ * Don't claim complete! This must be done by the guest. We will handle
+ * that in plic_handler(). In the meanwhile we simply deactivate S-Mode
+ * External IRQs, and reenable them when the guest claims it. In this
+ * way, we only need to store one pending IRQ per hart.
+ */
+ ext_disable();
+
+ return 0;
+}
+
+static inline void plic_passthru(const struct mmio_access *access)
+{
+ plic_write(access->address, access->value);
+}
+
+static inline enum mmio_result
+plic_handle_context_claim(struct mmio_access *access, unsigned long hart)
+{
+ if (!access->is_write) {
+ access->value = pending[hart];
+ return MMIO_HANDLED;
+ }
+
+ /* claim write case */
+ if (access->value != pending[hart]) {
+ printk("FATAL: Guest acknowledged non-pending IRQ %lu\n",
+ access->value);
+ return MMIO_ERROR;
+ }
+
+ plic_write(access->address, access->value);
+
+ /* Check if there's another physical IRQ pending */
+ /* TODO: This is where we would need to prioritise vIRQs */
+ pending[hart] = plic_read(access->address);
+ if (pending[hart])
+ return MMIO_HANDLED;
+
+ guest_clear_ext();
+ ext_enable();
+
+ return MMIO_HANDLED;
+}
+
+static enum mmio_result plic_handle_context(struct mmio_access *access)
+{
+ unsigned int cpu;
+ unsigned long hart;
+ int ctx;
+ u64 addr;
+
+ addr = access->address - PLIC_CTX_BASE;
+ ctx = addr / PLIC_CTX_SZ;
+
+ /*
+ * It is clear that a hart is allowed to access its own context.
+ * But we also need to allow accesses to context to neighboured
+ * harts within the cell.
+ *
+ * In (probably) 99% of all cases, the current active CPU will access
+ * its own context. So do this simple check first, and check other
+ * contexts of the cell (for loop) later. This results in a bit more
+ * complex code, but results in better performance.
+ */
+ hart = phys_processor_id();
+ if (hart_to_context(hart) == ctx)
+ goto allowed;
+
+ for_each_cpu_except(cpu, &this_cell()->cpu_set, this_cpu_id())
+ if (hart_to_context(public_per_cpu(cpu)->phys_id) == ctx)
+ goto allowed;
+
+ return trace_error(MMIO_ERROR);
+
+allowed:
+ addr -= ctx * PLIC_CTX_SZ;
+ if (addr == PLIC_CTX_CLAIM) {
+ return plic_handle_context_claim(access, hart);
+ } else if (addr == PLIC_CTX_PRIO_TH) {
+ /* We land here if we permit the access */
+ if (access->is_write)
+ plic_passthru(access);
+ else
+ access->value = plic_read(access->address);
+ } else {
+ return MMIO_ERROR;
+ }
+
+ return MMIO_HANDLED;
+}
+
+static enum mmio_result plic_handle_prio(struct mmio_access *access)
+{
+ unsigned int irq;
+ unsigned int prio = access->value;
+
+ irq = access->address / REG_SZ;
+
+ if (!irqchip_irq_in_cell(this_cell(), irq))
+ return MMIO_ERROR;
+
+ /*
+ * Maybe we can abandon this check. The cell should know the max
+ * allowed value, so simply allow any value?
+ */
+ if (prio > plic_max_priority())
+ return MMIO_ERROR;
+
+ plic_passthru(access);
+ return MMIO_HANDLED;
+}
+
+static enum mmio_result plic_handle_enable(struct mmio_access *access)
+{
+ struct public_per_cpu *pc;
+ u32 irq_allowed_bitmap;
+ unsigned int idx, cpu;
+ short int ctx;
+
+ ctx = (access->address - PLIC_ENABLE_BASE) / PLIC_ENABLE_OFF;
+
+ /* Does the context even belong to one of the cell's CPUs? */
+ for_each_cpu(cpu, &this_cell()->cpu_set) {
+ pc = public_per_cpu(cpu);
+ if (hart_to_context(pc->phys_id) == ctx)
+ goto allowed;
+ }
+
+ /*
+ * FIXME: Why does Linux read non-allowed ctxs? This seems to be an
+ * actual bug in Linux. When we remove a CPU from Linux, and we later
+ * change the affinity of the IRQ, then Linux will try to access
+ * Contexts which it is not in charge of any longer. While Linux
+ * disables IRQs, it does not adjust smp_affinities when removing CPUs.
+ *
+ * For the moment, and as a workaround, simply report any read as 0,
+ * and forbid writes != 0.
+ *
+ * ... Okay, we really have a Linux bug here.
+ * (a) Linux doesn't remove the affinity from removed CPUs
+ * (b) Linux allows to set affinity to non-present CPUs
+ *
+ * Actually, we should always return MMIO_ERROR here.
+ */
+
+#if 1
+ if (!access->is_write) {
+ access->value = 0;
+ } else if (access->value != 0)
+ return MMIO_ERROR;
+ return MMIO_HANDLED;
+#else
+ return MMIO_ERROR;
+#endif
+
+allowed:
+ /*
+ * Now we have to check if we have a read or write access. In case of
+ * reads, simply return the real value of the PLIC.
+ *
+ * In case of writes, compare against the irq_bitmap, if we're allowed
+ * to perform the write.
+ */
+ idx = ((access->address - PLIC_ENABLE_BASE) % PLIC_ENABLE_OFF)
+ * 8 / PLIC_BITS_PER_REG;
+
+ if (!access->is_write) {
+ access->value = plic_read(access->address);
+ return MMIO_HANDLED;
+ }
+
+ /* write case */
+ irq_allowed_bitmap = this_cell()->arch.irq_bitmap[idx];
+
+ if (access->value & ~irq_allowed_bitmap) {
+ printk("FATAL: Cell enabled non-assigned IRQ\n");
+ return MMIO_ERROR;
+ }
+
+ plic_passthru(access);
+
+ return MMIO_HANDLED;
+}
+
+static enum mmio_result plic_handler(void *arg, struct mmio_access *access)
+{
+ /* only allow 32bit access */
+ if (access->size != REG_SZ)
+ return MMIO_ERROR;
+
+ switch (access->address) {
+ case REG_RANGE(PLIC_PRIO_BASE, PLIC_PENDING_BASE):
+ return plic_handle_prio(access);
+ break;
+
+ case REG_RANGE(PLIC_ENABLE_BASE, PLIC_ENABLE_END):
+ return plic_handle_enable(access);
+ break;
+
+ case REG_RANGE(PLIC_CTX_BASE, PLIC_CTX_END):
+ if (access->address < plic_size())
+ return plic_handle_context(access);

+ break;
+
+ default:

+ break;
+ }
+
+ return MMIO_ERROR;
+}
+
+static int plic_cell_init(struct cell *cell)
+{
+ const struct jailhouse_irqchip *chip;
+ unsigned int n, pos;
+
+ mmio_region_register(cell, plic_phys(), plic_size(), plic_handler,
+ cell);
+
+ for_each_irqchip(chip, cell->config, n) {
+ /* Only support one single PLIC at the moment */
+ if (chip->address !=
+ system_config->platform_info.riscv.plic.base_address)
+ return trace_error(-EINVAL);
+
+ if (chip->pin_base % 32 != 0 ||
+ chip->pin_base + IRQCHIP_PINS > IRQ_BITMAP_PINS)
+ return trace_error(-EINVAL);
+
+ for (pos = 0; pos < ARRAY_SIZE(chip->pin_bitmap); pos++)
+ cell->arch.irq_bitmap[chip->pin_base / 32 + pos] |=
+ chip->pin_bitmap[pos];
+ }
+
+ /* This logic is shared with arm-common */
+ if (cell == &root_cell)
+ return 0;
+
+ for_each_irqchip(chip, cell->config, n) {
+ // TODO: Check if IRQs are disabled before removing them
+ for (pos = 0; pos < ARRAY_SIZE(chip->pin_bitmap); pos++)
+ root_cell.arch.irq_bitmap[chip->pin_base / 32 + pos] &=
+ ~chip->pin_bitmap[pos];
+ }
+
+ return 0;
+}
+
+static int plic_init(void)
+{
+ unsigned int cpu, irq;
+ s16 context;
+
+ plic_base = paging_map_device(plic_phys(), plic_size());
+ if (!plic_base)
+ return -ENOMEM;
+
+ plic_cell_init(&root_cell);
+
+ /*
+ * If we check during early initialisation if all enabled IRQs belong
+ * to the root cell, then we don't need to check if an IRQ belongs to a
+ * cell on arrival.
+ */
+ for_each_cpu(cpu, &root_cell.cpu_set) {
+ context = hart_to_context(cpu);
+ for (irq = 0; irq < plic_max_irq(); irq++) {
+ if (plic_irq_is_enabled(context, irq) &&
+ !irqchip_irq_in_cell(&root_cell, irq)) {
+ printk("Error: IRQ %u active in root cell\n",
+ irq);
+ return trace_error(-EINVAL);
+ }
+ }
+ }
+
+ return 0;
+}
+
+static void plic_cell_exit(struct cell *cell)
+{
+ const struct jailhouse_irqchip *chip;
+ unsigned int n, pos;
+
+ mmio_region_unregister(cell, plic_phys());
+
+ /* set all pins of the old cell in the root cell */
+ for_each_irqchip(chip, cell->config, n)
+ for (pos = 0; pos < ARRAY_SIZE(chip->pin_bitmap); pos++)
+ root_cell.arch.irq_bitmap[chip->pin_base / 32 + pos] |=
+ chip->pin_bitmap[pos];
+
+ /* mask out pins again that actually didn't belong to the root cell */
+ for_each_irqchip(chip, root_cell.config, n)
+ for (pos = 0; pos < ARRAY_SIZE(chip->pin_bitmap); pos++)
+ root_cell.arch.irq_bitmap[chip->pin_base / 32 + pos] &=
+ chip->pin_bitmap[pos];
+}
+
+static void plic_shutdown(void)
+{
+ if (plic_base)
+ paging_unmap_device(plic_phys(), plic_base, plic_size());
+}
+
+static unsigned int plic_mmio_count_regions(struct cell *cell)
+{
+ return 1;
+}
+
+static void plic_config_commit(struct cell *cell)
+{
+ unsigned int irq, n;
+ s16 ctx;
+
+ if (!cell)
+ return;
+
+ if (cell == &root_cell)
+ return;
+
+ for (irq = 0; irq < plic_max_irq(); irq++) {
+ /*
+ * Three possibilities:
+ * 1. IRQ belongs to root cell and was removed (cell
+ * creation)
+ * 2. IRQ belonged to non-root cell and was assigned back to
+ * non-root cell (cell destruction)
+ * 3. IRQ belonged to non-root cell and is simply gone
+ * (belongs to no one)
+ *
+ * IRQ-Bitmaps are already updated accordingly. All we have to
+ * do is to ensure that the IRQ is disabled. That's all.
+ */
+ if (!irqchip_irq_in_cell(cell, irq))
+ continue;
+
+ /* Disable the IRQ for each defined PLIC context. */
+ for (n = 0; n < MAX_CPUS; n++) {
+ ctx = hart_to_context(n);
+ if (ctx == -1)
+ // or break? Are there non-contiguous harts?
+ continue;
+
+ plic_disable_irq(ctx, irq);
+ }
+ }
+}
+
+DEFINE_UNIT (plic, "RISC-V PLIC");
--
2.36.1

[Ralf Ramsauer]

From: Stefan Huber <stefan...@oth-regensburg.de>

add necessary files for developing tiny-demo inmate. Consists of
modified copies from x86/arm.

Signed-off-by: Stefan Huber <stefan...@oth-regensburg.de>

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

.gitignore | 1 +
inmates/demos/riscv/Makefile | 19 ++++
inmates/demos/riscv/tiny-demo.c | 19 ++++
inmates/lib/include/inmate_common.h | 1 +
inmates/lib/riscv/Makefile | 46 ++++++++
inmates/lib/riscv/Makefile.lib | 64 ++++++++++++
inmates/lib/riscv/header.S | 44 ++++++++
inmates/lib/riscv/include/inmate.h | 157 ++++++++++++++++++++++++++++
inmates/lib/riscv/inmate.lds.S | 74 +++++++++++++
inmates/lib/riscv/printk.c | 61 +++++++++++
inmates/lib/riscv/setup.c | 43 ++++++++
inmates/lib/riscv/uart.c | 47 +++++++++
12 files changed, 576 insertions(+)
create mode 100644 inmates/demos/riscv/tiny-demo.c
create mode 100644 inmates/lib/riscv/Makefile.lib
create mode 100644 inmates/lib/riscv/header.S
create mode 100644 inmates/lib/riscv/include/inmate.h
create mode 100644 inmates/lib/riscv/inmate.lds.S
create mode 100644 inmates/lib/riscv/printk.c
create mode 100644 inmates/lib/riscv/setup.c
create mode 100644 inmates/lib/riscv/uart.c

diff --git a/.gitignore b/.gitignore
index 245733cb..80224289 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@ include/jailhouse/config.h
hypervisor/hypervisor.lds
inmates/lib/arm/inmate.lds
inmates/lib/arm64/inmate.lds
+inmates/lib/riscv/inmate.lds
pyjailhouse/pci_defs.py
tools/demos/cache-timings
tools/demos/ivshmem-demo
diff --git a/inmates/demos/riscv/Makefile b/inmates/demos/riscv/Makefile
index e69de29b..2f0bac84 100644
--- a/inmates/demos/riscv/Makefile
+++ b/inmates/demos/riscv/Makefile

@@ -0,0 +1,19 @@
+#
+# Jailhouse, a Linux-based partitioning hypervisor
+#

+# Copyright (c) Siemens AG, 2013, 2014

+#
+# Authors:
+# Jan Kiszka <jan.k...@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2. See
+# the COPYING file in the top-level directory.
+#
+
+include $(INMATES_LIB)/Makefile.lib
+

+INMATES := tiny-demo.bin
+
+tiny-demo-y := tiny-demo.o
+
+$(eval $(call DECLARE_TARGETS,$(INMATES)))
diff --git a/inmates/demos/riscv/tiny-demo.c b/inmates/demos/riscv/tiny-demo.c
new file mode 100644
index 00000000..e4af2642
--- /dev/null
+++ b/inmates/demos/riscv/tiny-demo.c
@@ -0,0 +1,19 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ * Stefan Huber <stefan...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+

+#include <inmate.h>
+
+void inmate_main(void)
+{

+ printk("Hello from this tiny cell!\n");
+}
diff --git a/inmates/lib/include/inmate_common.h b/inmates/lib/include/inmate_common.h
index 1c20a0af..8f474264 100644
--- a/inmates/lib/include/inmate_common.h
+++ b/inmates/lib/include/inmate_common.h
@@ -72,6 +72,7 @@
#define MSIX_CTRL_FMASK 0x4000

#ifndef __ASSEMBLY__
+
typedef s8 __s8;
typedef u8 __u8;

diff --git a/inmates/lib/riscv/Makefile b/inmates/lib/riscv/Makefile
index e69de29b..9ff57721 100644
--- a/inmates/lib/riscv/Makefile
+++ b/inmates/lib/riscv/Makefile
@@ -0,0 +1,46 @@

+#
+# Jailhouse, a Linux-based partitioning hypervisor
+#

+# Copyright (c) Siemens AG, 2015, 2016

+#
+# Authors:
+# Jan Kiszka <jan.k...@siemens.com>
+#

+# This work is licensed under the terms of the GNU GPL, version 2. See
+# the COPYING file in the top-level directory.
+#
+# Alternatively, you can use or redistribute this file under the following
+# BSD license:
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+# THE POSSIBILITY OF SUCH DAMAGE.

+#
+
+include $(INMATES_LIB)/Makefile.lib
+

+always-y := lib.a inmate.lds
+
+lib-y := $(common-objs-y)
+lib-y += header.o printk.o setup.o uart.o
+
+lib-y += ../cmdline.o ../printk.o ../setup.o ../string.o ../uart-8250.o
diff --git a/inmates/lib/riscv/Makefile.lib b/inmates/lib/riscv/Makefile.lib
new file mode 100644
index 00000000..9a87d8ff
--- /dev/null
+++ b/inmates/lib/riscv/Makefile.lib
@@ -0,0 +1,64 @@

+#
+# Jailhouse, a Linux-based partitioning hypervisor
+#

+# Copyright (c) Siemens AG, 2013, 2014

+#
+# Authors:
+# Jan Kiszka <jan.k...@siemens.com>
+#

+# This work is licensed under the terms of the GNU GPL, version 2. See
+# the COPYING file in the top-level directory.
+#
+# Alternatively, you can use or redistribute this file under the following
+# BSD license:
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+# THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+include $(ALWAYS_COMPAT_MK)
+
+-include $(GEN_CONFIG_MK)
+
+#KBUILD_CFLAGS += -m64 -mno-red-zone
+LINUXINCLUDE += -I$(INMATES_LIB)/include
+
+define DECLARE_TARGETS =
+ _TARGETS = $(1)
+ always-y := $$(_TARGETS)
+ # $(NAME-y) NAME-linked.o NAME.bin
+ targets += $$(foreach t,$$(_TARGETS:.bin=-y),$$($$t)) \
+ $$(_TARGETS:.bin=-linked.o) $$(_TARGETS)
+endef
+
+# prevent deleting intermediate files which would cause rebuilds
+.SECONDARY: $(addprefix $(obj)/,$(targets))
+
+# obj/NAME-linked.o: ... obj/$(NAME-y) lib/lib[32].a
+.SECONDEXPANSION:
+$(obj)/%-linked.o: $(INMATES_LIB)/inmate.lds $$(addprefix $$(obj)/,$$($$*-y)) \
+ $(INMATES_LIB)/$$(if $$($$*_32),lib32.a,lib.a) FORCE
+ $(call if_changed,ld)
+
+$(obj)/%.bin: $(obj)/%-linked.o FORCE
+ $(call if_changed,objcopy)
diff --git a/inmates/lib/riscv/header.S b/inmates/lib/riscv/header.S
new file mode 100644
index 00000000..3a12f51f
--- /dev/null
+++ b/inmates/lib/riscv/header.S
@@ -0,0 +1,44 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+.section ".boot", "ax"
+
+ .globl __reset_entry
+__reset_entry:
+ la sp, __stack_top
+ j c_entry
diff --git a/inmates/lib/riscv/include/inmate.h b/inmates/lib/riscv/include/inmate.h
new file mode 100644
index 00000000..dac9f146
--- /dev/null
+++ b/inmates/lib/riscv/include/inmate.h
@@ -0,0 +1,157 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *
+ * Copyright (c) OTH Regensburg

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+#ifndef _JAILHOUSE_INMATE_H
+#define _JAILHOUSE_INMATE_H
+
+#define __stringify_1(x...) #x
+#define __stringify(x...) __stringify_1(x)
+
+#define COMM_REGION_BASE 0x100000
+
+#define PAGE_SIZE (4 * 1024ULL)
+
+typedef signed char s8;
+typedef unsigned char u8;
+
+typedef signed short s16;
+typedef unsigned short u16;
+
+typedef signed int s32;
+typedef unsigned int u32;
+
+typedef signed long long s64;
+typedef unsigned long long u64;
+
+#define SR_SIE 0x00000002UL
+
+#define csr_read(csr) \
+({ \
+ register unsigned long __v; \
+ __asm__ __volatile__ ("csrr %0, " __stringify(csr) \
+ : "=r" (__v) : \
+ : "memory"); \
+ __v; \
+})
+
+#define csr_write(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrw " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+#define csr_clear(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrc " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+#define csr_set(csr, val) \
+({ \
+ unsigned long __v = (unsigned long)(val); \
+ __asm__ __volatile__ ("csrs " __stringify(csr) ", %0" \
+ : : "rK" (__v) \
+ : "memory"); \
+})
+
+static inline void disable_irqs(void)
+{
+ csr_clear(sstatus, SR_SIE);
+}
+
+static inline void cpu_relax(void)
+{
+ int dummy;
+
+ asm volatile ("div %0, %0, zero" : "=r"(dummy));
+ asm volatile ("" ::: "memory");
+}
+
+static inline void __attribute__((noreturn)) halt(void)
+{
+ while (1)
+ asm volatile ("wfi" : : : "memory");
+}
+
+static inline u8 mmio_read8(void *address)
+{
+ return *(volatile u8 *)address;
+}
+
+static inline u16 mmio_read16(void *address)
+{
+ return *(volatile u16 *)address;
+}
+
+static inline u32 mmio_read32(void *address)
+{
+ return *(volatile u32 *)address;
+}
+
+static inline u64 mmio_read64(void *address)
+{
+ return *(volatile u64 *)address;
+}
+
+static inline void mmio_write8(void *address, u8 value)
+{
+ *(volatile u8 *)address = value;
+}
+
+static inline void mmio_write16(void *address, u16 value)
+{
+ *(volatile u16 *)address = value;
+}
+
+static inline void mmio_write32(void *address, u32 value)
+{
+ *(volatile u32 *)address = value;
+}
+
+static inline void mmio_write64(void *address, u64 value)
+{
+ *(volatile u64 *)address = value;
+}
+
+#include <inmate_common.h>
+
+#endif /* !_JAILHOUSE_INMATE_H */
diff --git a/inmates/lib/riscv/inmate.lds.S b/inmates/lib/riscv/inmate.lds.S
new file mode 100644
index 00000000..cb6e7a76
--- /dev/null
+++ b/inmates/lib/riscv/inmate.lds.S
@@ -0,0 +1,74 @@
+/*
+ * Jailhouse RISC-V support
+ *
+ * Copyright (C) 2022 OTH Regensburg

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+#include <inmate_common.h>
+
+SECTIONS {
+ . = CONFIG_INMATE_BASE;
+ .boot : { *(.boot) }
+
+ . = CONFIG_INMATE_BASE + 0x100;
+ .cmdline : {
+ *(.cmdline)
+ BYTE(0); /* empty string in case no buffer is provided */
+ }
+
+ bss_start = .;
+ .bss : {
+ *(.bss)
+ }
+
+ . = ALIGN(4);
+ .text : {
+ *(.text)
+ }
+
+ .rodata : {
+ *(.rodata)
+ }
+
+ .data : {
+ *(.data)
+ }
+
+ . = ALIGN(4096);

+ . = . + 0x1000;

+ __stack_top = .;
+}
+
+ENTRY(__reset_entry)
diff --git a/inmates/lib/riscv/printk.c b/inmates/lib/riscv/printk.c
new file mode 100644
index 00000000..5eb31537
--- /dev/null
+++ b/inmates/lib/riscv/printk.c
@@ -0,0 +1,61 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) Siemens AG, 2013

+ *
+ * Authors:

+ * Jan Kiszka <jan.k...@siemens.com>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+#include <inmate.h>

+#include <uart.h>
+
+static void reg_out_mmio8(struct uart_chip *chip, unsigned int reg, u32 value)
+{
+ mmio_write8(chip->base + reg, value);
+}
+
+static u32 reg_in_mmio8(struct uart_chip *chip, unsigned int reg)
+{
+ return mmio_read8(chip->base + reg);
+}
+
+void arch_console_init(struct uart_chip *chip)
+{
+ struct jailhouse_console *console = &comm_region->console;
+
+ if (cmdline_parse_bool("con-regdist-1",
+ CON_USES_REGDIST_1(console->flags))) {
+ chip->reg_out = reg_out_mmio8;
+ chip->reg_in = reg_in_mmio8;
+ }
+}
diff --git a/inmates/lib/riscv/setup.c b/inmates/lib/riscv/setup.c
new file mode 100644
index 00000000..705c5630
--- /dev/null
+++ b/inmates/lib/riscv/setup.c
@@ -0,0 +1,43 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2012

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+#include <inmate.h>
+

+void arch_init_early(void)
+{
+}
diff --git a/inmates/lib/riscv/uart.c b/inmates/lib/riscv/uart.c
new file mode 100644
index 00000000..4c394dee
--- /dev/null
+++ b/inmates/lib/riscv/uart.c
@@ -0,0 +1,47 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2018-2022

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+#include <inmate.h>

+#include <uart.h>
+
+DECLARE_UART(8250);
+
+struct uart_chip *uart_array[] = {
+ &UART_OPS_NAME(8250),
+ NULL
+};
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/traps.c | 784 +++++++++++++++++++++++++++++++++-
1 file changed, 779 insertions(+), 5 deletions(-)

diff --git a/hypervisor/arch/riscv/traps.c b/hypervisor/arch/riscv/traps.c
index f61dfc1a..8f59a675 100644
--- a/hypervisor/arch/riscv/traps.c
+++ b/hypervisor/arch/riscv/traps.c
@@ -10,19 +10,793 @@

* the COPYING file in the top-level directory.

*/

+#include <jailhouse/control.h>
+#include <jailhouse/printk.h>

+#include <jailhouse/entry.h>
+#include <jailhouse/percpu.h>
+#include <jailhouse/processor.h>
+#include <jailhouse/utils.h>
+#include <asm/control.h>
#include <asm/processor.h>
+#include <asm/csr64.h>
+#include <asm/sbi.h>
+#include <asm/setup.h>

void arch_handle_trap(union registers *regs);
void arch_handle_fault(union registers *regs);

-void arch_handle_trap(union registers *regs)
+static const char *causes[] = {
+ [EXC_INST_MISALIGNED] = "Instruction Address Misaligned",
+ [EXC_INST_ACCESS] = "Instruction Address Fault",
+ [EXC_INST_ILLEGAL] = "Illegal Instruction",
+ [EXC_BREAKPOINT] = "Breakpoint",
+ [EXC_LOAD_ACCESS_MISALIGNED] = "Load Accesss Misaligned",
+ [EXC_LOAD_ACCESS] = "Load Access Fault",
+ [EXC_AMO_ADDRESS_MISALIGNED] = "AMO Address Misaligned",
+ [EXC_STORE_ACCESS] = "Store Access Fault",
+ [EXC_SYSCALL] = "Env Call From U-Mode",
+ [EXC_HYPERVISOR_SYSCALL] = "Env Call From S-Mode",
+ [EXC_SUPERVISOR_SYSCALL] = "Env Call From VS-Mode",
+ [11] = "Env Call From M-Mode",
+ [EXC_INST_PAGE_FAULT] = "Instruction Pagefault",
+ [EXC_LOAD_PAGE_FAULT] = "Load Pagefault",
+ [14] = "Reserved",
+ [EXC_STORE_PAGE_FAULT] = "Store Pagefault",
+ [16 ... 19] = "Reserved",
+ [EXC_INST_GUEST_PAGE_FAULT] = "Inst Guest Pagefault",
+ [EXC_LOAD_GUEST_PAGE_FAULT] = "Load Guest Pagefault",
+ [EXC_VIRTUAL_INST_FAULT] = "Virtual Instruction Fault",
+ [EXC_STORE_GUEST_PAGE_FAULT] = "Store Guest Pagefault",
+};
+

+static inline bool is_irq(u64 cause)
+{
+ return !!(cause & CAUSE_IRQ_FLAG);

+}
+

+static inline unsigned long to_irq(unsigned long cause)
+{
+ return cause & ~CAUSE_IRQ_FLAG;
+}
+

+static void dump_regs(union registers *r)
+{
+ struct public_per_cpu *pc = this_cpu_public();

+ unsigned long scause = csr_read(scause);

+ const char *cause_str = "UNKNOWN";
+
+ /* We might want to later exchange it with panic_printk, but for the
+ * moment of development, printk is more helpful. */
+ void (*printer)(const char *, ...) = panic_printk;
+
+ if (scause <= 23)
+ cause_str = causes[scause];
+ else if (is_irq(scause))
+ cause_str = "IRQ";
+
+ printer("Fatal: Exception on CPU %u (HART %u). Cause: %lu (%s)\n",
+ pc->cpu_id, pc->phys_id, to_irq(scause), cause_str);
+ printer("scause: 0x%016lx, stval: 0x%016lx, htval<<2: 0x%016lx\n",
+ scause, csr_read(stval), csr_read(CSR_HTVAL) << 2);
+ printer(" PC: 0x%016llx RA: 0x%016llx SP: 0x%016llx\n", r->pc, r->ra, r->sp);
+ printer(" GP: 0x%016llx TP: 0x%016llx T0: 0x%016llx\n", r->gp, r->tp, r->t0);
+ printer(" T1: 0x%016llx T2: 0x%016llx S0: 0x%016llx\n", r->t1, r->t2, r->s0);
+ printer(" S1: 0x%016llx A0: 0x%016llx A1: 0x%016llx\n", r->s1, r->a0, r->a1);
+ printer(" A2: 0x%016llx A3: 0x%016llx A4: 0x%016llx\n", r->a2, r->a3, r->a4);
+ printer(" A5: 0x%016llx A6: 0x%016llx A7: 0x%016llx\n", r->a5, r->a6, r->a7);
+ printer(" S2: 0x%016llx S3: 0x%016llx S4: 0x%016llx\n", r->s2, r->s3, r->s4);
+ printer(" S5: 0x%016llx S6: 0x%016llx S7: 0x%016llx\n", r->s5, r->s6, r->s7);
+ printer(" S8: 0x%016llx S9: 0x%016llx S10: 0x%016llx\n", r->s8, r->s9, r->s10);
+ printer("S11: 0x%016llx T3: 0x%016llx T4: 0x%016llx\n", r->s11, r->t3, r->t4);
+ printer(" T5: 0x%016llx T6: 0x%016llx\n", r->t5, r->t6);
+}
+
+static int handle_timer(void)
+{
+ this_cpu_public()->stats[JAILHOUSE_CPU_STAT_VMEXITS_TIMER]++;
+ sbi_set_timer(-1);
+
+ /* inject timer to VS */
+ csr_set(CSR_HVIP, (1 << IRQ_S_TIMER) << VSIP_TO_HVIP_SHIFT);
+
+ return 0;
+}
+
+static inline void riscv_guest_inject_ipi(void)
+{
+ /* inject IPI to VS */
+ csr_set(CSR_HVIP, (1 << IRQ_S_SOFT) << VSIP_TO_HVIP_SHIFT);
+}
+
+static int handle_ipi(void)

+{
+ struct public_per_cpu *pcpu = this_cpu_public();

+ bool check_events = false;
+ u32 *stats = pcpu->stats;
+ int err = 0;
+
+ /* Take the control lock */
+ spin_lock(&pcpu->control_lock);
+
+ /*
+ * Next, we have to check who's the recipient of the IPI. Is it the
+ * hypervisor, or is it the guest?
+ */
+ switch (pcpu->ipi_cause) {
+ case IPI_CAUSE_GUEST:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_IPI]++;
+ riscv_guest_inject_ipi();
+ break;
+
+ case IPI_CAUSE_MGMT:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_MANAGEMENT]++;
+ check_events = true;
+ break;
+
+ case IPI_CAUSE_NONE:
+ default:
+ printk("Fatal: Invalid IPI cause on CPU %u\n", this_cpu_id());
+ err = -EINVAL;
+ break;
+ }
+
+ this_cpu_public()->ipi_cause = IPI_CAUSE_NONE;
+ /*
+ * IPI must be acknowledged. Clear the IPI by clearing sip.SSIP.
+ * Alternatively, IPI could be cleared via SBI. This is deprecated and
+ * unperformant.
+ */
+ csr_clear(sip, (1 << IRQ_S_SOFT));
+
+ /*
+ * Unlock the control lock. Another CPU may us now send some further
+ * events, until we enter arch_check_events. It is important that the
+ * IPI is acknowledged here, as from now on, further IPIs might already
+ * be sent by remote CPUs.
+ */
+ spin_unlock(&pcpu->control_lock);
+
+ if (check_events)
+ arch_check_events();
+

+ return err;
+}
+

+static int handle_irq(u64 irq)
+{
+ int err;
+

+ switch (irq) {
+ case IRQ_S_TIMER:

+ err = handle_timer();
+ break;
+
+ case IRQ_S_SOFT:
+ err = handle_ipi();
+ break;
+
+ case IRQ_S_EXT:
+ err = plic_set_pending();

+ break;
+
+ default:

+ err = -ENOSYS;
+ break;
+ }
+
+ return err;
+}
+
+static int sbi_ext_time(struct sbiret *ret, unsigned int fid, u64 stime)
+{
+ if (fid != SBI_EXT_TIME_SET_TIMER)
+ return -ENOSYS;
+
+ /* Clear pending IRQs */
+ csr_clear(CSR_HVIP, ((1 << IRQ_S_TIMER) << VSIP_TO_HVIP_SHIFT));
+
+ *ret = sbi_set_timer(stime);
+
+ return 0;
+}
+
+static inline void skip_emulated_instruction(union registers *regs, unsigned int b)
+{
+ regs->pc += b;

+ csr_write(sepc, regs->pc);
+}

+
+static bool sbi_permitted_target_mask(unsigned long mask, unsigned long base)

+{
+ unsigned int cpu;

+ unsigned long phys;
+
+ for_each_cpu(cpu, &this_cell()->cpu_set) {
+ phys = public_per_cpu(cpu)->phys_id;
+ if (phys < base)
+ continue;
+
+ mask &= ~(1UL << (phys - base));
+ if (!mask)

+ return true;
+ }
+

+ return false;
+}
+

+static bool sbi_permitted_hart(unsigned long hartid)

+{
+ unsigned int cpu;
+

+ for_each_cpu(cpu, &this_cell()->cpu_set)
+ if (public_per_cpu(cpu)->phys_id == hartid)
+ return true;

+ return false;
+}
+

+static void cpu_queue_ipi(unsigned int cpu)
+{
+ struct public_per_cpu *pcpu;
+ pcpu = public_per_cpu(cpu);
+
+retry:
+ spin_lock(&pcpu->control_lock);
+ if (pcpu->ipi_cause == IPI_CAUSE_NONE)
+ pcpu->ipi_cause = IPI_CAUSE_GUEST;
+ else {
+ spin_unlock(&pcpu->control_lock);
+ goto retry;

+ }
+ spin_unlock(&pcpu->control_lock);
+}
+

+static int handle_sbi_send_ipi(struct sbiret *ret, unsigned long mask,
+ unsigned long base)
+{
+ unsigned long phys;

+ unsigned int cpu;
+

+ /* targets in cell? Check privileges. */
+ if (!sbi_permitted_target_mask(mask, base))
+ return -EPERM;
+
+ for (phys = 0; phys < sizeof(unsigned long) * 8; phys++) {
+ if (!(mask & (1UL << phys)))
+ continue;
+
+ cpu = cpu_by_phys_processor_id(base + phys);
+ if (cpu == this_cpu_id()) {
+ mask &= ~(1UL << phys);
+ riscv_guest_inject_ipi();
+ continue;
+ }
+
+ cpu_queue_ipi(cpu);
+ }
+
+ /* If we land here, we can forward the SBI call */
+
+ /* Only forward the IPI, if in mask is anything left */
+ if (mask)
+ *ret = sbi_send_ipi(mask, base);
+ else /* Just a single self-IPI */
+ ret->value = ret->error = 0;

+
+ return 0;
+}
+
+

+static inline int handle_sbi_rfence(struct sbiret *ret, unsigned int fid,
+ unsigned long mask, unsigned long base,
+ unsigned long a2, unsigned long a3,
+ unsigned long a4)
+{
+ if (!sbi_permitted_target_mask(mask, base))
+ return -EPERM;
+
+ *ret = sbi_ecall(SBI_EXT_RFENCE, fid, mask, base, a2, a3, a4, 0);

+ return 0;
+}
+

+static int riscv_unpark(struct sbiret *ret, unsigned long hartid, unsigned long start_addr, unsigned long opaque)
+{
+ struct public_per_cpu *pcpu;
+ unsigned int cpu;
+ struct sbiret tmp;
+ int err = 0;
+
+ cpu = cpu_by_phys_processor_id(hartid);
+ pcpu = public_per_cpu(cpu);
+
+ spin_lock(&pcpu->control_lock);
+ if (pcpu->hsm.state == STARTED || pcpu->hsm.state == START_PENDING) {
+ ret->value = 0;
+ ret->error = SBI_ERR_ALREADY_AVAILABLE;
+ goto unlock_out;
+ }
+
+ pcpu->hsm.start_addr = start_addr;
+ pcpu->hsm.opaque = opaque;
+ pcpu->hsm.state = START_PENDING;
+ pcpu->ipi_cause = IPI_CAUSE_MGMT;
+ // fence?
+
+ tmp = sbi_send_ipi(1UL << hartid, 0);
+ if (tmp.error != SBI_SUCCESS) {
+ err = -EINVAL;
+ goto unlock_out;
+ }
+
+ ret->error = SBI_SUCCESS;
+ ret->value = 0;
+
+unlock_out:
+ spin_unlock(&pcpu->control_lock);

+ return err;
+}
+

+static int handle_sbi_hsm(struct sbiret *ret, unsigned int fid, unsigned long a0,
+ unsigned long a1, unsigned long a2)
+{
+ int err = 0;
+
+ switch (fid) {
+ case SBI_EXT_HSM_HART_STOP:
+ riscv_park_cpu();
+ break;
+
+ case SBI_EXT_HSM_HART_START:
+ ret->value = 0;
+ if (!sbi_permitted_hart(a0) ||
+ a0 == this_cpu_public()->phys_id) {
+ ret->error = SBI_ERR_INVALID_PARAM;
+ break;
+ }
+
+ /* FIXME: Check if remote CPU is really parked! */
+ err = riscv_unpark(ret, a0, a1, a2);
+ break;
+
+ case SBI_EXT_HSM_HART_STATUS:
+ if (sbi_permitted_hart(a0)) {
+ ret->error = SBI_SUCCESS;
+ ret->value = public_per_cpu(cpu_by_phys_processor_id(a0))->hsm.state;
+ } else {
+ ret->error = SBI_ERR_INVALID_PARAM;
+ ret->value = 0;
+ }

+ break;
+
+ default:

+ printk("Unknown HSM Fid: %u\n", fid);
+ err = -EINVAL;
+ break;
+ }
+
+ return err;
+}
+
+static int sbi_ext_base(struct sbiret *ret, unsigned long fid, unsigned long a0)
+{
+ int err = 0;
+
+ switch (fid) {
+ case SBI_EXT_BASE_GET_SPEC_VERSION:
+ case SBI_EXT_BASE_GET_IMP_ID:
+ case SBI_EXT_BASE_GET_IMP_VERSION:
+ case SBI_EXT_BASE_PROBE_EXT: /* FIXME Rethink of this one. We could decline HSM for non-root cells */
+ case SBI_EXT_BASE_GET_MVENDORID:
+ case SBI_EXT_BASE_GET_MARCHID:
+ case SBI_EXT_BASE_GET_MIMPID:
+ *ret = sbi_ecall(SBI_EXT_BASE, fid, a0, 0, 0, 0, 0, 0);

+ break;
+
+ default:

+ err = -ENOSYS;
+ break;
+ }
+
+ return err;
+}
+
+static int handle_ecall(union registers *regs)
+{
+ /*
+ * Spec: In the name of compatibility, SBI extension IDs (EIDs) and SBI
+ * function IDs (FIDs) are encoded as signed 32-bit integers. When
+ * passed in registers these follow the standard above calling
+ * convention rules.
+ */
+ u32 *stats = this_cpu_public()->stats;
+ unsigned int eid, fid;
+ struct sbiret ret;
+ int err = -ENOSYS;
+
+ eid = regs->a7;
+ fid = regs->a6;
+ ret.value = 0;
+
+ switch (eid) {
+ /* Treat putchar like a hypercall. Accounts for a hypercall. */
+ case SBI_EXT_0_1_CONSOLE_PUTCHAR:
+ ret.error = hypercall(JAILHOUSE_HC_DEBUG_CONSOLE_PUTC,
+ regs->a0, 0);
+ err = 0;
+ break;
+
+ case SBI_EXT_0_1_CONSOLE_GETCHAR: /* FIXME: Only allow for root cell? */
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ ret = sbi_console_getchar_legacy_0_1();
+ err = 0;
+ break;
+
+ case SBI_EXT_BASE:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ err = sbi_ext_base(&ret, fid, regs->a0);
+ break;
+
+ case SBI_EXT_TIME: /* since SBI v0.2 */
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ err = sbi_ext_time(&ret, fid, regs->a0);
+ break;
+
+ case SBI_EXT_SPI:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ err = handle_sbi_send_ipi(&ret, regs->a0, regs->a1);
+ break;
+
+ case SBI_EXT_RFENCE:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ err = handle_sbi_rfence(&ret, fid, regs->a0, regs->a1, regs->a2,
+ regs->a3, regs->a4);
+ break;
+
+ case SBI_EXT_HSM:
+ stats[JAILHOUSE_CPU_STAT_VMEXITS_SBI]++;
+ err = handle_sbi_hsm(&ret, fid, regs->a0, regs->a1, regs->a2);
+ break;
+
+ case JAILHOUSE_EID:
+ ret.error = hypercall(fid, regs->a0, regs->a1);
+ if (fid == JAILHOUSE_HC_DISABLE && !ret.error) {
+ riscv_deactivate_vmm(regs, 0);
+ }
+ /* If the HC failed, don't crash the CPU, rather propagate the error */
+ err = 0;

+ break;
+
+ default:

+ printk("Unknown SBI call EID: 0x%x FID: 0x%x\n", eid, fid);
+ return -EINVAL;
+ break;
+ }
+

+ if (err)
+ return err;
+

+ /* If we came from stop, don't propagate error codes */
+ if (eid == SBI_EXT_HSM && fid == SBI_EXT_HSM_HART_STOP)
+ return 0;
+
+ regs->a0 = ret.error;
+ regs->a1 = ret.value;
+ skip_emulated_instruction(regs, 4);

+
+ return 0;
+}
+

+static inline int gmem_read16(unsigned long addr, u16 *val)
+{
+ register unsigned long err asm("a0");
+ register unsigned long _val asm("a1") = (unsigned long)val;
+ register unsigned long _addr asm("a2") = addr;
+
+ /*
+ * hlvx.hu may fault and throw an exception. This is, so far, the only
+ * known S-Mode exception that is recoverable and must not crash the
+ * hypervisor. In this case, the hypervisor will catch the exception,
+ * and set a0 properly. If a0 remains zero, then no exception occured
+ * and everything is fine. We need to save sstatus and hstatus before,
+ * as nested S-Mode exceptions will mess up those registers.
+ */
+ asm volatile("addi %2, zero, 0\n"
+ "csrr a3, sstatus\n"
+ "csrr a4, %3\n"
+ /* Haha, I wonder what Jan will say to this hack... */
+ "HLVX:"
+ ".word 0x643645f3\n" /* hlvx.hu a1, (a2) */
+ "csrw sstatus, a3\n"
+ "csrw %3, a4\n"
+ : "=r"(_val)
+ : "r"(_addr), "r"(err), "i"(CSR_HSTATUS)
+ : "memory", "a3", "a4");
+ *val = _val;
+
+ /* err will be set, if an exception has been thrown */
+ return err;
+}
+
+static inline u64 sext(u32 w)
+{
+ u64 ret;
+ asm volatile("sext.w %0, %0" : "=r"(ret) : "r"(w) :);
+ return ret;
+}
+
+#define COMP_RX_OFF 8
+
+union instruction {
+ struct {
+ u32 type:2;
+ u32 opcode:5;
+ u32 dest:5;
+ u32 width:3;
+ u32 base:5;
+ u32 offset:12;
+ } load __attribute__((packed));
+ struct {
+ u32 type:2;
+ u32 opcode:5;
+ u32 offset40:5;
+ u32 width:3;
+ u32 base:5;
+ u32 src:5;
+ u32 offset115:7;
+ } store __attribute__((packed));
+ struct {
+ u32 type:2;
+ u32 opcode:5;
+ u32 rsvd:25;
+ } generic __attribute__((packed));
+ u32 raw;
+} __attribute__((packed));
+
+union cinstruction {
+ struct {
+ u16 opcode:2;
+ u16 src_dst:3;
+ u16 off1:2;
+ u16 base:3;
+ u16 off2:3;
+ u16 funct3:3;
+ } load_store __attribute__((packed));
+ struct {
+ u16 opcode:2;
+ u16 rsvd:14;
+ } generic __attribute__((packed));
+ u16 raw;
+} __attribute__((packed));
+
+struct riscv_mmio_inst {
+ unsigned char reg;
+ unsigned char width;
+ bool sign_extended;
+};
+
+static int riscv_decode_compressed_instruction(struct riscv_mmio_inst *mmio_inst, u16 inst)
{
- for (;;)
- cpu_relax();
+ union cinstruction i;
+ int err = -ENOSYS;
+
+ i.raw = inst;
+ /* SW */
+ if (i.generic.opcode == 0 && i.load_store.funct3 == 6) {
+ mmio_inst->width = 4;
+ mmio_inst->sign_extended = true;
+ mmio_inst->reg = i.load_store.src_dst + COMP_RX_OFF;
+ err = 0;
+ /* LW */
+ } else if (i.generic.opcode == 0 && i.load_store.funct3 == 2) {
+ mmio_inst->width = 4;
+ mmio_inst->sign_extended = true;
+ mmio_inst->reg = i.load_store.src_dst + COMP_RX_OFF;
+ err = 0;
+ }
+

+ return err;
+}
+

+static int riscv_decode_instruction(struct riscv_mmio_inst *mmio_inst, u32 inst, bool is_compressed)
+{
+ union instruction i;
+ int err = -ENOSYS;
+
+ if (is_compressed)
+ return riscv_decode_compressed_instruction(mmio_inst, inst);
+
+ i.raw = inst;
+ if (i.generic.type != 0x3)
+ return err;
+
+ /* LB, LH, LW */
+ if (i.generic.opcode == 0x0) {
+ if (i.load.width > 2)
+ return err;
+ mmio_inst->width = 1 << i.load.width;
+ mmio_inst->sign_extended = true;
+ mmio_inst->reg = i.load.dest;
+ err = 0;
+ /* SB, SW, SH */
+ } else if (i.generic.opcode == 0x8) {
+ if (i.store.width > 2)
+ return err;
+ mmio_inst->width = 1 << i.store.width;
+ mmio_inst->sign_extended = true;
+ mmio_inst->reg = i.store.src;
+ err = 0;
+ }
+

+ return err;
+}
+

+static int handle_fault(union registers *regs, bool is_write)
+{
+ struct riscv_mmio_inst mmio_inst;
+ struct mmio_access mmio;
+ enum mmio_result result;
+ bool is_compressed;
+ u32 instruction;
+ size_t *reg;
+ u16 tmp;
+ int err;
+
+ this_cpu_public()->stats[JAILHOUSE_CPU_STAT_VMEXITS_MMIO]++;
+
+ mmio.is_write = is_write;
+ mmio.address = csr_read(CSR_HTVAL) << 2;
+
+ /*
+ * Potentially, htval might point to address NULL, while pointing to a
+ * valid trap value. However, 0 might also indicate that htval is not
+ * supported by the micro-architecture. Hence, by design, let's say
+ * that address NULL should (a) not be used or (b) must not cause
+ * access faults when being used.
+ *
+ * Here, we assume that the micro-architecture doesn't support htval,
+ * if we read back zero on a fault exception.
+ */
+ if (!mmio.address)
+ return -ENOSYS;
+
+ /* Load remaining lsb two bits from stval */
+ mmio.address |= csr_read(stval) & 0x3;
+
+ /* Load faulting instruction */
+
+ /* check if htinst is available */
+#if 0 /* htinst might hold pseudo-decoded instruction which we don't support yet. */
+ instruction = csr_read(CSR_HTINST);
+#else
+ instruction = 0;
+#endif
+ if (instruction != 0) {
+ is_compressed = (instruction & 0x3) != 0x3;
+ } else { /* if not, load vom guest mem */
+ err = gmem_read16(regs->pc, &tmp);

+ if (err)
+ return err;
+

+ instruction = tmp;
+ if ((instruction & 0x3) == 0x3) {
+ is_compressed = false;
+ err = gmem_read16(regs->pc + 2, &tmp);

+ if (err)
+ return err;

+ instruction |= (u32)tmp << 16;
+ } else
+ is_compressed = true;
+ }
+
+ err = riscv_decode_instruction(&mmio_inst, instruction, is_compressed);
+ if (err)
+ goto unsup;
+
+ if (mmio.is_write)
+ mmio.value = regs->raw.x[mmio_inst.reg];
+ mmio.size = mmio_inst.width;
+
+ result = mmio_handle_access(&mmio);
+ if (result == MMIO_HANDLED) {
+ if (!mmio.is_write) {
+ reg = &regs->raw.x[mmio_inst.reg];
+ *reg = mmio.value;
+ if (mmio_inst.width < 8) {
+ *reg &= ((1UL << mmio_inst.width * 8) - 1);
+ if (mmio_inst.sign_extended)
+ *reg = sext(*reg);
+ }
+ }
+ skip_emulated_instruction(regs, is_compressed ? 2 : 4);

+ return 0;
+ }
+

+ return -ENOSYS;
+
+unsup:
+ printk("Unsupported instruction: 0x%x\n", instruction);
+ return -ENOSYS;
}

void arch_handle_fault(union registers *regs)
{
- for (;;)
- cpu_relax();
+ unsigned long hlvx_location;
+ /*
+ * If we have a Load Guest Page Fault, and the fault came from hlvx,
+ * then set a0 to -EINVAL, skip hlvx instruction, and return.
+ */
+ regs->pc = csr_read(sepc);
+ if (csr_read(scause) == EXC_LOAD_GUEST_PAGE_FAULT) {
+ asm volatile("la %0, HLVX\n" : "=r"(hlvx_location));
+ if (regs->pc != hlvx_location)
+ goto panic_out;
+
+ skip_emulated_instruction(regs, 4);
+ regs->a0 = -EINVAL;
+ return;
+ }
+
+panic_out:
+ panic_printk("FATAL: Unhandled S-Mode exception.\n");
+ panic_printk("Hypervisor registers:\n");
+ dump_regs(regs);
+ panic_stop();
+}

+
+void arch_handle_trap(union registers *regs)
+{

+ unsigned long cause;
+ int err = -1;
+
+ this_cpu_public()->stats[JAILHOUSE_CPU_STAT_VMEXITS_TOTAL]++;
+
+ regs->pc = csr_read(sepc);
+ cause = csr_read(scause);
+
+ if (is_irq(cause)) {
+ err = handle_irq(cause & ~CAUSE_IRQ_FLAG);

+ goto out;
+ }
+

+ switch (cause) {
+ case EXC_INST_ACCESS:
+ case EXC_LOAD_ACCESS:
+ case EXC_STORE_ACCESS:
+ case EXC_INST_PAGE_FAULT:
+ case EXC_LOAD_PAGE_FAULT:
+ case EXC_STORE_PAGE_FAULT:
+ case EXC_INST_MISALIGNED:
+ case EXC_LOAD_ACCESS_MISALIGNED:
+ case EXC_AMO_ADDRESS_MISALIGNED:
+ printk("\nFaulting Address: %016lx\n", csr_read(stval));
+ err = -ENOSYS;
+ break;
+
+ case EXC_SUPERVISOR_SYSCALL:
+ err = handle_ecall(regs);
+ break;
+
+ case EXC_BREAKPOINT:
+ printk("BP occured @ PC: %016lx\n", regs->pc);
+ err = -1;
+ break;
+
+ case EXC_LOAD_GUEST_PAGE_FAULT:
+ err = handle_fault(regs, false);
+ break;
+
+ case EXC_STORE_GUEST_PAGE_FAULT:
+ err = handle_fault(regs, true);
+ break;
+
+ case EXC_INST_ILLEGAL:

+ default:
+ err = -1;
+ break;
+ }
+

+out:
+ if (err) {
+ dump_regs(regs);
+ panic_park();
+ }
}
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/qemu-riscv64-tiny-demo.c | 63 ++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
create mode 100644 configs/riscv/qemu-riscv64-tiny-demo.c

diff --git a/configs/riscv/qemu-riscv64-tiny-demo.c b/configs/riscv/qemu-riscv64-tiny-demo.c
new file mode 100644
index 00000000..076e0ffb
--- /dev/null
+++ b/configs/riscv/qemu-riscv64-tiny-demo.c
@@ -0,0 +1,63 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Minimal configuration for RISC-V demo inmate, 1 CPU, 1MiB RAM
+ *
+ * Copyright (c) OTH Regensburg 2022

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See

+ * the COPYING file in the top-level directory.
+ */
+
+#include <jailhouse/types.h>
+#include <jailhouse/cell-config.h>
+
+struct {
+ struct jailhouse_cell_desc cell;
+ struct jailhouse_cpu cpus[1];
+ struct jailhouse_memory mem_regions[2];
+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,
+ .name = "tiny-demo",
+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG |
+ JAILHOUSE_CELL_TEST_DEVICE |
+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_PERMITTED |
+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE,
+
+ .num_cpus = ARRAY_SIZE(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = 0,
+ .num_pci_devices = 0,
+
+ .console = {
+ .type = JAILHOUSE_CON_TYPE_RISCV_SBI,
+ },
+ },
+
+ .cpus = {
+ {
+ .phys_id = 1,
+ },
+ },
+
+ .mem_regions = {
+ /* RAM */ {
+ .phys_start = 0xbf800000,
+ .virt_start = 0x0,
+ .size = 1 * 1024 * 1024,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },
+ /* communication region */ {
+ .virt_start = 0x00100000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_COMM_REGION,
+ },
+ },
+};
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/dts/noelv-linux-inmate.dts | 130 +++++++++++++++++++++++
configs/riscv/noelv-linux-demo.c | 101 ++++++++++++++++++
2 files changed, 231 insertions(+)
create mode 100644 configs/riscv/dts/noelv-linux-inmate.dts
create mode 100644 configs/riscv/noelv-linux-demo.c

diff --git a/configs/riscv/dts/noelv-linux-inmate.dts b/configs/riscv/dts/noelv-linux-inmate.dts
new file mode 100644
index 00000000..486ea82f
--- /dev/null
+++ b/configs/riscv/dts/noelv-linux-inmate.dts
@@ -0,0 +1,130 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+

+/dts-v1/;
+/ {
+ model = "Jailhouse cell on RISC-V";
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ chosen {
+ stdout-path = "/soc/uart@fc001000:115200n8";
+ bootargs = "console=ttyGR0 earlycon=sbi";
+ linux,initrd-start = <0x82000000>;
+ linux,initrd-end = <0x82200000>;
+ };
+
+ memory@80000000 {
+ device_type = "memory";
+ reg = <0x00000000 0x80000000>, <0x00000000 0x4000000>;
+ };
+
+ cpus {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ timebase-frequency = <0x989680>;
+
+ cpu@0 {
+ device_type = "cpu";
+ reg = <1>;
+ status = "okay";
+ compatible = "riscv";
+ riscv,isa = "rv64imafdcsu";
+ mmu-type = "riscv,sv39";
+
+ cpu0_intc: interrupt-controller {
+ #address-cells = <0>;
+ #interrupt-cells = <1>;
+ interrupt-controller;
+ compatible = "riscv,cpu-intc";
+ };
+ };
+
+ cpu@1 {
+ device_type = "cpu";
+ reg = <2>;
+ status = "okay";
+ compatible = "riscv";
+ riscv,isa = "rv64imafdcsu";
+ mmu-type = "riscv,sv39";
+
+ cpu1_intc: interrupt-controller {
+ #address-cells = <0>;
+ #interrupt-cells = <1>;
+ interrupt-controller;
+ compatible = "riscv,cpu-intc";
+ };
+ };
+
+ cpu@2 {
+ device_type = "cpu";
+ reg = <0>;
+ status = "disabled";
+ compatible = "riscv";
+ riscv,isa = "rv64imafdcsu";
+ mmu-type = "riscv,sv39";
+
+ dummy_intc: interrupt-controller {
+ #address-cells = <0>;
+ #interrupt-cells = <1>;
+ interrupt-controller;
+ compatible = "riscv,cpu-intc";
+ };
+ };
+ };
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+ compatible = "simple-bus";
+ ranges;
+
+ sysclock: sysclock {
+ #clock-cells = <0>;
+ compatible = "fixed-clock";
+ clock-frequency = <100000000>;
+ };
+
+ uart0: uart@fc001000 {
+ compatible = "gaisler,apbuart";
+ reg = <0x00 0xfc001000 0x00 0x100>;
+ clocks = <&sysclock>;
+ current-speed = <115200>;
+ interrupt-parent = <&plic0>;
+ interrupts = <1>;
+ };
+
+ plic0: plic@f8000000 {
+ #address-cells = <0x00>;
+ #interrupt-cells = <0x01>;
+ compatible = "riscv,plic0";
+ interrupt-controller;
+ reg = <0x00 0xf8000000 0x00 0x4000000>;
+ interrupts-extended = <
+ &dummy_intc 11
+ &dummy_intc 9
+ &dummy_intc 8
+ &dummy_intc 10
+ &cpu0_intc 11
+ &cpu0_intc 9
+ &cpu0_intc 8
+ &cpu0_intc 10
+ &cpu1_intc 11
+ &cpu1_intc 9
+ &cpu1_intc 8
+ &cpu1_intc 10
+ >;
+ riscv,ndev = <31>;
+ riscv,max-priority = <7>;
+ };
+ };
+};
diff --git a/configs/riscv/noelv-linux-demo.c b/configs/riscv/noelv-linux-demo.c
new file mode 100644
index 00000000..bd263b78
--- /dev/null
+++ b/configs/riscv/noelv-linux-demo.c
@@ -0,0 +1,101 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Configuration for RISC-V Linux inmate on NOEL-V

+ *
+ * Copyright (c) OTH Regensburg 2022
+ *
+ * Authors:
+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <jailhouse/types.h>
+#include <jailhouse/cell-config.h>
+
+struct {
+ struct jailhouse_cell_desc cell;

+ struct jailhouse_cpu cpus[2];
+ struct jailhouse_memory mem_regions[8];
+ struct jailhouse_irqchip irqchips[1];
+ struct jailhouse_pci_device pci_devices[0];

+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,

+ .name = "linux-demo",

+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG |
+ JAILHOUSE_CELL_TEST_DEVICE |

+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_PERMITTED,

+
+ .num_cpus = ARRAY_SIZE(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),

+ .num_pci_devices = ARRAY_SIZE(config.pci_devices),
+ .num_irqchips = ARRAY_SIZE(config.irqchips),
+
+ /*
+ * This IRQ must be BELOW the .riscv.plic.max_irq, as
+ * Linux won't address any higher IRQ
+ */
+ .vpci_irq_base = 28,

+ },
+
+ .cpus = {
+ {
+ .phys_id = 1,
+ },

+ {
+ .phys_id = 2,

+ },
+ },
+
+ .mem_regions = {

+ /* RAM low */ {
+ .phys_start = 0x3fa00000,
+ .virt_start = 0x0,
+ .size = 0x100000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },
+ /* communication region */ {
+ .virt_start = 0x00100000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_COMM_REGION,
+ },

+ /* RAM high */ {
+ .phys_start = 0x3ba00000,
+ .virt_start = 0x80000000,
+ .size = 0x4000000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },

+ /* uart@fc001000 */ {
+ .phys_start = 0xfc001000,
+ .virt_start = 0xfc001000,
+ .size = 0x1000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,
+ },
+ /* IVSHMEM shared memory regions (networking) */
+ JAILHOUSE_SHMEM_NET_REGIONS(0x3fb00000, 1),
+ },
+
+ .pci_devices = {
+ },
+
+ .irqchips = {
+ /* PLIC */ {
+ .address = 0xf8000000,
+ .pin_base = 0,
+ .pin_bitmap = {
+ (1 << 1), /* UART */
+ 0,
+ 0,
+ 0,
+ },
+ },
+ },
+};
--
2.36.1

[Ralf Ramsauer]

For demonstration purposes only.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

hypervisor/arch/riscv/include/asm/cell.h | 1 +
hypervisor/arch/riscv/include/asm/ivshmem.h | 1 +
hypervisor/arch/riscv/include/asm/percpu.h | 4 +-
hypervisor/arch/riscv/include/asm/plic.h | 7 +
hypervisor/arch/riscv/ivshmem.c | 36 ++++-
hypervisor/arch/riscv/plic.c | 138 +++++++++++++++++++-
hypervisor/arch/riscv/traps.c | 1 +
7 files changed, 181 insertions(+), 7 deletions(-)

diff --git a/hypervisor/arch/riscv/include/asm/cell.h b/hypervisor/arch/riscv/include/asm/cell.h
index 58412414..fcfdaa32 100644
--- a/hypervisor/arch/riscv/include/asm/cell.h
+++ b/hypervisor/arch/riscv/include/asm/cell.h
@@ -23,6 +23,7 @@ struct arch_cell {
struct paging_structures mm;

u32 irq_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)];

+ u32 virq_present_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)];
};

#endif /* !_JAILHOUSE_ASM_CELL_H */
diff --git a/hypervisor/arch/riscv/include/asm/ivshmem.h b/hypervisor/arch/riscv/include/asm/ivshmem.h
index 03251590..8b193947 100644
--- a/hypervisor/arch/riscv/include/asm/ivshmem.h
+++ b/hypervisor/arch/riscv/include/asm/ivshmem.h
@@ -11,4 +11,5 @@
*/

struct arch_ivshmem_irq_cache {
+ u16 id[IVSHMEM_MSIX_VECTORS];
};
diff --git a/hypervisor/arch/riscv/include/asm/percpu.h b/hypervisor/arch/riscv/include/asm/percpu.h
index 4eda15b6..bcafff51 100644
--- a/hypervisor/arch/riscv/include/asm/percpu.h
+++ b/hypervisor/arch/riscv/include/asm/percpu.h
@@ -43,6 +43,8 @@ enum sbi_hart_state {
} hsm; \
bool wait_for_power_on; \
bool reset; \
- bool park;
+ bool park; \
+ u32 virq_enabled_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)]; \
+ u32 virq_pending_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)];

#define ARCH_PERCPU_FIELDS
diff --git a/hypervisor/arch/riscv/include/asm/plic.h b/hypervisor/arch/riscv/include/asm/plic.h
index c5414e9e..016e9b99 100644
--- a/hypervisor/arch/riscv/include/asm/plic.h
+++ b/hypervisor/arch/riscv/include/asm/plic.h
@@ -18,4 +18,11 @@
extern int plic_set_pending(void);
bool irqchip_irq_in_cell(struct cell *cell, unsigned int irq);

+void plic_register_virq(unsigned int irq);
+void plic_unregister_virq(unsigned int irq);
+
+void plic_send_virq(struct cell *cell, unsigned int irq);
+
+void plic_process_pending_virqs(void);

+
#endif /* _PLIC_H */

diff --git a/hypervisor/arch/riscv/ivshmem.c b/hypervisor/arch/riscv/ivshmem.c
index e5dd7973..3c645123 100644
--- a/hypervisor/arch/riscv/ivshmem.c
+++ b/hypervisor/arch/riscv/ivshmem.c
@@ -1,21 +1,35 @@
/*

* Jailhouse, a Linux-based partitioning hypervisor

*
- * Copyright (c) Siemens AG, 2020
+ * Copyright (c) Siemens AG, 2016-2019

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:
* Jan Kiszka <jan.k...@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

*/

-#include <jailhouse/entry.h>
#include <jailhouse/ivshmem.h>
+#include <jailhouse/cell.h>
+#include <asm/processor.h>

void arch_ivshmem_trigger_interrupt(struct ivshmem_endpoint *ive,
unsigned int vector)
{
+ unsigned int irq_id = ive->irq_cache.id[vector];
+
+ if (irq_id) {
+ /*
+ * Ensure that all data written by the sending guest is visible
+ * to the target before triggering the interrupt.
+ */
+ memory_barrier();
+
+ plic_send_virq(ive->device->cell, irq_id);
+ }
}

int arch_ivshmem_update_msix(struct ivshmem_endpoint *ive, unsigned int vector,
@@ -26,4 +40,22 @@ int arch_ivshmem_update_msix(struct ivshmem_endpoint *ive, unsigned int vector,

void arch_ivshmem_update_intx(struct ivshmem_endpoint *ive, bool enabled)
{
+ u8 pin = ive->cspace[PCI_CFG_INT/4] >> 8;
+ struct pci_device *device = ive->device;
+ unsigned int virq;
+
+ /*
+ * Lock used as barrier, ensuring all interrupts triggered after return
+ * use the new setting.
+ */
+ virq = device->cell->config->vpci_irq_base + pin - 1;
+ spin_lock(&ive->irq_lock);
+ if (enabled) {
+ ive->irq_cache.id[0] = virq;
+ plic_register_virq(virq);
+ } else {
+ ive->irq_cache.id[0] = 0;
+ plic_unregister_virq(virq);
+ }
+ spin_unlock(&ive->irq_lock);
}
diff --git a/hypervisor/arch/riscv/plic.c b/hypervisor/arch/riscv/plic.c
index 84f95c0b..1b9e0c3e 100644
--- a/hypervisor/arch/riscv/plic.c
+++ b/hypervisor/arch/riscv/plic.c
@@ -193,6 +193,11 @@ inline bool irqchip_irq_in_cell(struct cell *cell, unsigned int irq)
return irq_bitmap_test(cell->arch.irq_bitmap, irq);
}

+static inline bool irqchip_virq_in_cell(struct cell *cell, unsigned int irq)
+{
+ return irq_bitmap_test(cell->arch.virq_present_bitmap, irq);
+}
+
int plic_set_pending(void)
{
int my_context;
@@ -239,9 +244,38 @@ static inline void plic_passthru(const struct mmio_access *access)
plic_write(access->address, access->value);
}

+static bool plic_inject_pending_virqs(void)

+{
+ struct public_per_cpu *pcpu = this_cpu_public();

+ u32 idx, irq = 0;
+
+ for (idx = 0; idx < ARRAY_SIZE(pcpu->virq_pending_bitmap); idx++) {
+ irq = pcpu->virq_pending_bitmap[idx];
+ if (!irq)
+ continue;
+
+ /*
+ * FIXME: For the moment, simply inject the first pending IRQ.
+ * Later, we need to prioritise those IRQs. Haha. Per call of
+ * this routine, we can only inject ONE single IRQ. That's not
+ * an issue, as the guest will trap again after acknowledging
+ * the last irq. So there will be no misses of pending IRQs.
+ */
+
+ irq = ffsl(irq) + idx * 32;
+
+ pending[pcpu->phys_id] = irq;
+ irq_bitmap_clear(pcpu->virq_pending_bitmap, irq);

+ return true;
+ }
+
+ return false;
+}
+

static inline enum mmio_result

plic_handle_context_claim(struct mmio_access *access, unsigned long hart)
{

+ /* clear pending bit */
if (!access->is_write) {

access->value = pending[hart];

return MMIO_HANDLED;
@@ -254,7 +288,9 @@ plic_handle_context_claim(struct mmio_access *access, unsigned long hart)
return MMIO_ERROR;
}

- plic_write(access->address, access->value);
+ /* TODO: vIRQ could have been disabled before acknowledgement */
+ if (!irq_bitmap_test(this_cell()->arch.virq_present_bitmap, access->value))

+ plic_write(access->address, access->value);

/* Check if there's another physical IRQ pending */

/* TODO: This is where we would need to prioritise vIRQs */

@@ -262,6 +298,11 @@ plic_handle_context_claim(struct mmio_access *access, unsigned long hart)
if (pending[hart])
return MMIO_HANDLED;

+ /* TODO: vIRQ has the lowest prio at the moment */
+ plic_inject_pending_virqs();

+ if (pending[hart])
+ return MMIO_HANDLED;
+

guest_clear_ext();
ext_enable();

@@ -322,6 +363,12 @@ static enum mmio_result plic_handle_prio(struct mmio_access *access)

irq = access->address / REG_SZ;

+ if (irqchip_virq_in_cell(this_cell(), irq)) {
+ // TODO: Allow priorities
+ printk("PLIC: virq priorities not supported!\n");

+ return MMIO_HANDLED;
+ }
+

if (!irqchip_irq_in_cell(this_cell(), irq))
return MMIO_ERROR;

@@ -338,8 +385,8 @@ static enum mmio_result plic_handle_prio(struct mmio_access *access)

static enum mmio_result plic_handle_enable(struct mmio_access *access)
{

+ u32 *virq_enabled, irq_allowed_bitmap, virq_allowed_bitmap;
struct public_per_cpu *pc;
- u32 irq_allowed_bitmap;
unsigned int idx, cpu;
short int ctx;

@@ -389,20 +436,28 @@ allowed:
*/

idx = ((access->address - PLIC_ENABLE_BASE) % PLIC_ENABLE_OFF)

* 8 / PLIC_BITS_PER_REG;
+ // TODO: Should this be locked? virq_allowed_bitmap could be changed
+ // during execution
+ virq_enabled = &pc->virq_enabled_bitmap[idx];

if (!access->is_write) {
- access->value = plic_read(access->address);
+ access->value = plic_read(access->address) | *virq_enabled;
return MMIO_HANDLED;
}

/* write case */

irq_allowed_bitmap = this_cell()->arch.irq_bitmap[idx];

+ virq_allowed_bitmap = this_cell()->arch.virq_present_bitmap[idx];

- if (access->value & ~irq_allowed_bitmap) {
+ if (access->value & ~(irq_allowed_bitmap | virq_allowed_bitmap)) {

printk("FATAL: Cell enabled non-assigned IRQ\n");

return MMIO_ERROR;
}

+ *virq_enabled = access->value & virq_allowed_bitmap;
+
+ /* Only forward physical IRQs to the PLIC */
+ access->value &= irq_allowed_bitmap;
plic_passthru(access);

return MMIO_HANDLED;
@@ -443,6 +498,14 @@ static int plic_cell_init(struct cell *cell)
mmio_region_register(cell, plic_phys(), plic_size(), plic_handler,
cell);

+ /*
+ * TODO: Do we need that, or can we assume that this arrives already
+ * zeroed?
+ */
+ memset(cell->arch.irq_bitmap, 0, sizeof(cell->arch.irq_bitmap));
+ memset(cell->arch.virq_present_bitmap, 0,
+ sizeof(cell->arch.virq_present_bitmap));

+
for_each_irqchip(chip, cell->config, n) {

/* Only support one single PLIC at the moment */

if (chip->address !=
@@ -573,4 +636,71 @@ static void plic_config_commit(struct cell *cell)
}
}

+void plic_process_pending_virqs(void)
+{
+ /*
+ * We can only inject IRQs if there's no other IRQ waiting. No problem:
+ * If other IRQs are currently being handled, the cell must somewhen
+ * acknowledge the interrupt. On acknowledgement, this routine is
+ * called again, so we won't miss the IRQ.
+ */
+ if (guest_ext_pending())
+ return;
+
+ if (!plic_inject_pending_virqs())
+ return;
+
+ ext_disable();
+ guest_inject_ext();
+}
+
+void plic_send_virq(struct cell *cell, unsigned int irq)

+{
+ struct public_per_cpu *pcpu;
+ unsigned int cpu;
+

+ //printk("PLIC: sending vIRQ %u from %s to %s\n", irq, this_cell()->config->name, cell->config->name);
+
+ if (!irq_bitmap_test(cell->arch.virq_present_bitmap, irq)) {
+ printk("vIRQ not present in destination\n");
+ return;
+ }
+
+ // Do we need to lock this section? A vIRQ could be disabled during injection
+ for_each_cpu(cpu, &cell->cpu_set) {
+ pcpu = public_per_cpu(cpu);
+ if (irq_bitmap_test(pcpu->virq_enabled_bitmap, irq)) {
+ irq_bitmap_set(pcpu->virq_pending_bitmap, irq);
+ memory_barrier();
+ arch_send_event(pcpu);

+ break;
+ }
+ }
+}

+
+void plic_register_virq(unsigned int irq)
+{
+ struct cell *cell = this_cell();
+
+ if (irqchip_irq_in_cell(cell, irq)) {
+ printk("FATAL: plic: Unable to register vIRQ %u\n", irq);
+ panic_stop();
+ }
+
+ irq_bitmap_set(cell->arch.virq_present_bitmap, irq);
+}
+
+void plic_unregister_virq(unsigned int irq)
+{
+ struct cell *cell = this_cell();

+ unsigned int cpu;
+

+ if (!irq_bitmap_test(cell->arch.virq_present_bitmap, irq))
+ return;
+
+ irq_bitmap_clear(cell->arch.virq_present_bitmap, irq);
+ for_each_cpu(cpu, &cell->cpu_set)
+ irq_bitmap_clear(public_per_cpu(cpu)->virq_enabled_bitmap, irq);
+}
+
DEFINE_UNIT (plic, "RISC-V PLIC");
diff --git a/hypervisor/arch/riscv/traps.c b/hypervisor/arch/riscv/traps.c
index 8f59a675..6b768c9b 100644
--- a/hypervisor/arch/riscv/traps.c
+++ b/hypervisor/arch/riscv/traps.c
@@ -154,6 +154,7 @@ static int handle_ipi(void)

* IPI is acknowledged here, as from now on, further IPIs might already

* be sent by remote CPUs.

*/
+ plic_process_pending_virqs();
spin_unlock(&pcpu->control_lock);

if (check_events)
--
2.36.1

[Ralf Ramsauer]

---
configs/riscv/noelv-tiny-demo.c | 63 +++++++++++++++++++++++++++++++++

1 file changed, 63 insertions(+)

create mode 100644 configs/riscv/noelv-tiny-demo.c

diff --git a/configs/riscv/noelv-tiny-demo.c b/configs/riscv/noelv-tiny-demo.c
new file mode 100644
index 00000000..4eedbed2
--- /dev/null
+++ b/configs/riscv/noelv-tiny-demo.c

@@ -0,0 +1,63 @@
+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Minimal configuration for RISCV demo inmate, 1 CPU, 1MiB RAM
+ *

+ * Copyright (c) OTH Regensburg 2022
+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <jailhouse/types.h>
+#include <jailhouse/cell-config.h>
+
+struct {
+ struct jailhouse_cell_desc cell;

+ struct jailhouse_cpu cpus[1];
+ struct jailhouse_memory mem_regions[2];

+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,

+ .name = "tiny-demo",

+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG |
+ JAILHOUSE_CELL_TEST_DEVICE |
+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_PERMITTED |

+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE,

+
+ .num_cpus = ARRAY_SIZE(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),

+ .num_irqchips = 0,
+ .num_pci_devices = 0,
+
+ .console = {
+ .type = JAILHOUSE_CON_TYPE_RISCV_SBI,

+ },
+ },
+

+ .cpus = {
+ {
+ .phys_id = 1,
+ },

+ },
+
+ .mem_regions = {

+ /* RAM */ {

+ .phys_start = 0x3fa00000,
+ .virt_start = 0x0,
+ .size = 0x100000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },
+ /* communication region */ {
+ .virt_start = 0x00100000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
+ JAILHOUSE_MEM_COMM_REGION,
+ },

+ },
+};
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/noelv.c | 143 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 143 insertions(+)
create mode 100644 configs/riscv/noelv.c

diff --git a/configs/riscv/noelv.c b/configs/riscv/noelv.c
new file mode 100644
index 00000000..08389cb0
--- /dev/null
+++ b/configs/riscv/noelv.c
@@ -0,0 +1,143 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Configuration for NOEL-V riscv64 target, 1G RAM, 6 cores
+ *
+ * Copyright (c) Siemens AG, 2020
+ * Copyright (c) OTH Regensburg, 2022

+ *
+ * Authors:

+ * Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ *

+ * NOTE: Add "mem=768M" to the kernel command line.

+ */
+
+#include <jailhouse/types.h>
+#include <jailhouse/cell-config.h>
+
+struct {

+ struct jailhouse_system header;
+ struct jailhouse_cpu cpus[6];
+ struct jailhouse_memory mem_regions[7];

+ struct jailhouse_irqchip irqchips[1];
+ struct jailhouse_pci_device pci_devices[0];

+ struct jailhouse_pci_capability pci_caps[0];
+} __attribute__((packed)) config = {
+ .header = {
+ .signature = JAILHOUSE_SYSTEM_SIGNATURE,
+ .revision = JAILHOUSE_CONFIG_REVISION,
+ .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
+ .hypervisor_memory = {
+ .phys_start = 0x3fc00000,
+ .size = 0x00400000,
+ },
+ .platform_info = {
+ .pci_mmconfig_base = 0x40000000,
+ .pci_mmconfig_end_bus = 0,
+ .pci_is_virtual = true,
+ .pci_domain = 0,
+
+ .riscv = {
+ .plic = {
+ .base_address = 0xf8000000,
+ .size = 0x4000000,
+ .max_irq = 31,
+ .max_priority = 7,
+ .hart_to_context = {
+ [0] = 1, /* S-Mode of Hart 0 has PLIC context 1 */
+ [1] = 5, /* S-Mode of Hart 1 has PLIC context 5 */
+ [2] = 9, /* S-Mode of Hart 2 has PLIC context 9 */
+ [3] = 13, /* S-Mode of Hart 3 has PLIC context 13 */
+ [4] = 17, /* S-Mode of Hart 3 has PLIC context 17 */
+ [5] = 21, /* S-Mode of Hart 3 has PLIC context 21 */
+ [6 ... 31] = -1,
+ },
+ },
+ },
+ },
+ .root_cell = {
+ .name = "noelv-riscv64",

+ .num_cpus = ARRAY_SIZE(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),

+ .num_irqchips = ARRAY_SIZE(config.irqchips),
+ .num_pci_devices = ARRAY_SIZE(config.pci_devices),
+ .num_pci_caps = ARRAY_SIZE(config.pci_caps),
+
+ /*

+ * This IRQ must be BELOW the .riscv.plic.max_irq, as
+ * Linux won't address any higher IRQ
+ */
+ .vpci_irq_base = 28,

+ },
+ },
+
+ .cpus = {
+ {
+ .phys_id = 1,
+ },

+ {
+ .phys_id = 0,

+ },
+ {
+ .phys_id = 2,
+ },

+ {
+ .phys_id = 3,
+ },
+ {
+ .phys_id = 4,
+ },
+ {
+ .phys_id = 5,

+ },
+ },
+
+ .mem_regions = {
+ { /* RAM */

+ .phys_start = 0x0,
+ .virt_start = 0x0,
+ .size = 0x3fb00000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_EXECUTE,

+ },
+ { /* uart@fc001000 */
+ .phys_start = 0xfc001000,
+ .virt_start = 0xfc001000,
+ .size = 0x1000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO,
+ },
+ { /* greth@fc084000 */
+ .phys_start = 0xfc084000,
+ .virt_start = 0xfc084000,
+ .size = 0x1000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO,

+ },
+ /* IVSHMEM shared memory regions (networking) */

+ JAILHOUSE_SHMEM_NET_REGIONS(0x3fb00000, 0),

+ },
+
+ .irqchips = {

+ /* plic@f8000000 */ {
+ .address = 0xf8000000,
+ .id = 0, /* PLIC */

+ .pin_base = 0,
+ .pin_bitmap = {

+ (1 << 1) | /* UART */
+ (1 << 5), /* greth */

+ 0,
+ 0,
+ 0,

+ },
+ },

+ },
+
+ .pci_devices = {

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/dts/qemu-linux-inmate.dts | 135 ++++++++++++++++++++++++
configs/riscv/qemu-linux-demo.c | 101 ++++++++++++++++++
2 files changed, 236 insertions(+)
create mode 100644 configs/riscv/dts/qemu-linux-inmate.dts
create mode 100644 configs/riscv/qemu-linux-demo.c

diff --git a/configs/riscv/dts/qemu-linux-inmate.dts b/configs/riscv/dts/qemu-linux-inmate.dts
new file mode 100644
index 00000000..0fdbbac8
--- /dev/null
+++ b/configs/riscv/dts/qemu-linux-inmate.dts
@@ -0,0 +1,135 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Copyright (c) OTH Regensburg, 2022
+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+/dts-v1/;
+/ {
+ model = "Jailhouse cell on RISC-V";
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ chosen {

+ stdout-path = "/soc/uart@10000000:115200n8";
+ bootargs = "console=ttyS0 earlycon=sbi";

+ linux,initrd-start = <0x82000000>;
+ linux,initrd-end = <0x82200000>;
+ };
+
+ memory@80000000 {
+ device_type = "memory";

+ /* 63MiB @ 0x80000000 */
+ reg = <0x00000000 0x80000000>, <0x00000000 0x3f00000>;

+ uart@10000000 {
+ interrupts = <0x0a>;
+ interrupt-parent = <&plic0>;
+ clock-frequency = "\08@";
+ reg = <0x00 0x10000000 0x00 0x100>;
+ compatible = "ns16550a";
+ status = "okay";
+ };
+
+ plic0: plic@c000000 {
+ reg = <0x00 0xc000000 0x00 0x210000>;

+ interrupts-extended = <
+ &dummy_intc 11
+ &dummy_intc 9

+ &cpu0_intc 11
+ &cpu0_intc 9

+ &cpu1_intc 11
+ &cpu1_intc 9

+ >;
+ interrupt-controller;
+ compatible = "sifive,plic-1.0.0\0riscv,plic0";
+ riscv,ndev = <0x35>;
+ #interrupt-cells = <0x01>;
+ #address-cells = <0x00>;
+ };
+
+ pci@30000000 {
+ compatible = "pci-host-ecam-generic";
+ device_type = "pci";
+ bus-range = <0 0>;
+ #interrupt-cells = <1>;
+ reg = <0x0 0x30000000 0x0 0x10000000>;
+ #address-cells = <3>;
+ #size-cells = <2>;
+ ranges = <
+ 0x1000000 0x00 0x00 0x00 0x3000000 0x00 0x10000
+ 0x2000000 0x00 0x40000000 0x00 0x40000000 0x00 0x40000000
+ 0x3000000 0x04 0x00 0x04 0x00 0x04 0x00>;
+
+ interrupt-map-mask = <0>;
+ interrupt-map = <0 0 0 1 &plic0 32 0>;
+ };
+ };
+};
diff --git a/configs/riscv/qemu-linux-demo.c b/configs/riscv/qemu-linux-demo.c
new file mode 100644
index 00000000..e1947125
--- /dev/null
+++ b/configs/riscv/qemu-linux-demo.c
@@ -0,0 +1,101 @@

+/*
+ * Jailhouse, a Linux-based partitioning hypervisor
+ *

+ * Configuration for RISC-V Linux inmate on QEMU
+ *
+ * Copyright (c) OTH Regensburg 2022

+ *
+ * Authors:

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.

+ */
+
+#include <jailhouse/types.h>
+#include <jailhouse/cell-config.h>
+
+struct {

+ struct jailhouse_cell_desc cell;

+ struct jailhouse_cpu cpus[2];
+ struct jailhouse_memory mem_regions[8];

+ struct jailhouse_irqchip irqchips[1];
+ struct jailhouse_pci_device pci_devices[0];

+} __attribute__((packed)) config = {
+ .cell = {
+ .signature = JAILHOUSE_CELL_DESC_SIGNATURE,

+ .revision = JAILHOUSE_CONFIG_REVISION,
+ .name = "linux-demo",

+ .flags = JAILHOUSE_CELL_PASSIVE_COMMREG |
+ JAILHOUSE_CELL_TEST_DEVICE |

+ JAILHOUSE_CELL_VIRTUAL_CONSOLE_PERMITTED,
+

+ .num_cpus = ARRAY_SIZE(config.cpus),
+ .num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = ARRAY_SIZE(config.irqchips),
+ .num_pci_devices = ARRAY_SIZE(config.pci_devices),
+

+ .console = {
+ .type = JAILHOUSE_CON_TYPE_8250,
+ .size = 0x1000,
+ .address = 0x10000000,
+ .flags = JAILHOUSE_CON_ACCESS_MMIO | JAILHOUSE_CON_REGDIST_1,
+ },
+
+ .vpci_irq_base = 32,

+ },
+
+ .cpus = {
+ {
+ .phys_id = 1,
+ },
+ {

+ .phys_id = 2,

+ },
+ },
+
+ .mem_regions = {

+ /* RAM low */ {

+ .phys_start = 0xbf800000,
+ .virt_start = 0x0,

+ .size = 1 * 1024 * 1024,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
+ },
+ /* communication region */ {
+ .virt_start = 0x00100000,
+ .size = 0x00001000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_COMM_REGION,
+ },
+ /* RAM high */ {

+ .phys_start = 0xbb800000,
+ .virt_start = 0x80000000,
+ .size = 64 * 1024 * 1024,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,

+ },
+ /* uart@fc001000 */ {

+ .phys_start = 0x10000000,
+ .virt_start = 0x10000000,

+ .size = 0x1000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,

+ },
+ /* IVSHMEM shared memory regions (networking) */

+ JAILHOUSE_SHMEM_NET_REGIONS(0xbf900000, 1),

+ },
+
+ .pci_devices = {
+ },
+

+ .irqchips = {
+ /* PLIC */ {
+ .address = 0xc000000,

+ .pin_base = 0,
+ .pin_bitmap = {

+ (1 << 0xa), 0, 0, 0,

+ },
+ },
+ },

+};
--
2.36.1

[Ralf Ramsauer]

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/qemu-riscv64.c | 218 +++++++++++++++++++++++++++++++++--
include/jailhouse/console.h | 1 +
2 files changed, 211 insertions(+), 8 deletions(-)

diff --git a/configs/riscv/qemu-riscv64.c b/configs/riscv/qemu-riscv64.c
index 3c68ad71..1ace554b 100644
--- a/configs/riscv/qemu-riscv64.c
+++ b/configs/riscv/qemu-riscv64.c
@@ -1,12 +1,15 @@
/*

* Jailhouse, a Linux-based partitioning hypervisor

*
- * Configuration for QEMU riscv64 virtual target, 1G RAM, 4 cores
+ * Configuration for QEMU riscv64 virtual target, 1G RAM, 6 cores
*

* Copyright (c) Siemens AG, 2020

+ * Copyright (c) OTH Regensburg, 2022

*
* Authors:

* Jan Kiszka <jan.k...@siemens.com>
+ * Konrad Schwarz <konrad....@siemens.com>

+ * Ralf Ramsauer <ralf.r...@oth-regensburg.de>

*

* This work is licensed under the terms of the GNU GPL, version 2. See

* the COPYING file in the top-level directory.

@@ -17,24 +20,65 @@
#include <jailhouse/types.h>
#include <jailhouse/cell-config.h>

+#define IVSHMEM
+
+#define VIRTIO_MMIO(OFFSET) \
+{ \
+ .phys_start = 0x10000000 + (OFFSET), \
+ .virt_start = 0x10000000 + (OFFSET), \

+ .size = 0x00001000, \
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | \

+ JAILHOUSE_MEM_IO, \
+}
+
struct {
struct jailhouse_system header;
- struct jailhouse_cpu cpus[4];
- struct jailhouse_memory mem_regions[1];

+ struct jailhouse_cpu cpus [6];

+ struct jailhouse_memory mem_regions [20];

+ struct jailhouse_irqchip irqchips [1];

+ struct jailhouse_pci_device pci_devices[1];
+ struct jailhouse_pci_capability pci_caps[6];
} __attribute__((packed)) config = {
.header = {
.signature = JAILHOUSE_SYSTEM_SIGNATURE,
.revision = JAILHOUSE_CONFIG_REVISION,
.flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
.hypervisor_memory = {
- .phys_start = 0xbfc00000,
- .size = 0x00400000,
+ /* N.B.: 2 MiB megapage alignment requirement */
+ .phys_start = 0xbfa00000,
+ .size = 0x00600000,
+ },
+
+ .platform_info = {
+ .pci_mmconfig_base = 0x30000000,
+ .pci_mmconfig_end_bus = 0xff, // ??

+ .riscv = {
+ .plic = {

+ .base_address = 0xc000000,
+ .size = 0x600000,
+ .max_irq = 96,

+ .max_priority = 7,
+ .hart_to_context = {
+ [0] = 1,

+ [1] = 3,
+ [2] = 5,
+ [3] = 7,
+ [4] = 9,
+ [5] = 11,

+ [6 ... 31] = -1,
+ }

+ },
+ },
},
.root_cell = {
.name = "qemu-riscv64",
-
.num_cpus = ARRAY_SIZE(config.cpus),

.num_memory_regions = ARRAY_SIZE(config.mem_regions),
+ .num_irqchips = ARRAY_SIZE(config.irqchips),
+ .num_pci_devices = ARRAY_SIZE(config.pci_devices),

+ .num_pci_caps = ARRAY_SIZE(config.pci_caps),

+
+ .vpci_irq_base = 32,

},
},

@@ -51,15 +95,173 @@ struct {
{
.phys_id = 3,
},
+ {
+ .phys_id = 4,
+ },
+ {
+ .phys_id = 5,
+ },
},

.mem_regions = {
/* RAM */ {
.phys_start = 0x80000000,
.virt_start = 0x80000000,
- .size = 0x3fa10000,
+ .size = 0x3fa00000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_EXECUTE,
+ },
+ /* rtc@101000 */ {
+ .phys_start = 0x00101000,
+ .virt_start = 0x00101000,

+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO,
+ },
+ /* uart@10000000 */ {

+ .phys_start = 0x10000000,
+ .virt_start = 0x10000000,

+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO,
+ },
+ /* test@100000 */ {
+ .phys_start = 0x00100000,

+ .virt_start = 0x00100000,
+ .size = 0x00001000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_IO,
+ },
+ /* flash@20000000 */ {
+ .phys_start = 0x20000000,
+ .virt_start = 0x20000000,
+ .size = 0x02000000,

+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |

+ JAILHOUSE_MEM_EXECUTE,
+ },
+ /* flash@22000000, 2nd range */ {
+ .phys_start = 0x22000000,
+ .virt_start = 0x22000000,
+ .size = 0x02000000,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE,
},
- }
+ /* MemRegion: 40040000-4005ffff: e1000e */
+ {
+ .phys_start = 0x40040000,
+ .virt_start = 0x40040000,
+ .size = 0x20000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
+ },
+ /* MemRegion: 40060000-4007ffff: e1000e */
+ {
+ .phys_start = 0x40060000,
+ .virt_start = 0x40060000,
+ .size = 0x20000,
+ .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
+ },
+ /* virtio_mmio@10001000 */
+ VIRTIO_MMIO(0x1000),
+ /* virtio_mmio@10002000 */
+ VIRTIO_MMIO(0x2000),
+ /* virtio_mmio@10003000 */
+ VIRTIO_MMIO(0x3000),
+ /* virtio_mmio@10004000 */
+ VIRTIO_MMIO(0x4000),
+ /* virtio_mmio@10005000 */
+ VIRTIO_MMIO(0x5000),
+ /* virtio_mmio@10006000 */
+ VIRTIO_MMIO(0x6000),
+ /* virtio_mmio@10007000 */
+ VIRTIO_MMIO(0x7000),
+ /* virtio_mmio@10008000 */
+ VIRTIO_MMIO(0x8000),
+

+ /* IVSHMEM shared memory regions (networking) */

+ JAILHOUSE_SHMEM_NET_REGIONS(0xbf900000, 0),

+ },
+
+ .irqchips = {

+ /* plic@c000000 */ {
+ .address = 0xc000000,
+ .id = 0 /* PLIC */,

+ .pin_base = 0,
+ .pin_bitmap = {

+ 0
+ | 1 << 1 /* virtio_mmio@10001000 */
+ | 1 << 2 /* virtio_mmio@10002000 */
+ | 1 << 3 /* virtio_mmio@10003000 */
+ | 1 << 4 /* virtio_mmio@10004000 */
+ | 1 << 5 /* virtio_mmio@10005000 */
+ | 1 << 6 /* virtio_mmio@10006000 */
+ | 1 << 7 /* virtio_mmio@10007000 */
+ | 1 << 8 /* virtio_mmio@10008000 */
+ | 1 << 0xa /* uart@10000000 */
+ | 1 << 0xb /* rtc@101000 */
+ ,
+ 1 << (0x22 - 0x20) /* PCI INT C / slot 0 */
+ ,
+ 0,
+ 0,

+ },
+ },
+ },
+

+ .pci_devices = {
+ { /* e1000e */
+ .type = JAILHOUSE_PCI_TYPE_DEVICE,
+ .domain = 0x0000,
+ .bdf = 0x0010,
+ .bar_mask = {
+ 0xfffe0000, 0xfffe0000, 0xffffffe0,
+ 0xffffc000, 0x00000000, 0x00000000,
+ },
+ .caps_start = 0,
+ .num_caps = 6,
+ .num_msi_vectors = 1,
+ .msi_64bits = 1,
+ .num_msix_vectors = 5,
+ .msix_region_size = 0x1000,
+ .msix_address = 0x40080000,

+ },
+ },
+

+ .pci_caps = {
+ { /* e1000e */
+ .id = PCI_CAP_ID_PM,
+ .start = 0xc8,
+ .len = 8,
+ .flags = JAILHOUSE_PCICAPS_WRITE,
+ },
+ {
+ .id = PCI_CAP_ID_MSI,
+ .start = 0xd0,
+ .len = 14,
+ .flags = JAILHOUSE_PCICAPS_WRITE,
+ },
+ {
+ .id = PCI_CAP_ID_EXP,
+ .start = 0xe0,
+ .len = 20,
+ .flags = JAILHOUSE_PCICAPS_WRITE,
+ },
+ {
+ .id = PCI_CAP_ID_MSIX,
+ .start = 0xa0,
+ .len = 12,
+ .flags = JAILHOUSE_PCICAPS_WRITE,
+ },
+ {
+ .id = PCI_EXT_CAP_ID_ERR | JAILHOUSE_PCI_EXT_CAP,
+ .start = 0x100,
+ .len = 4,
+ .flags = 0,
+ },
+ {
+ .id = PCI_EXT_CAP_ID_DSN | JAILHOUSE_PCI_EXT_CAP,
+ .start = 0x140,
+ .len = 4,
+ .flags = 0,
+ },
+ },
};
diff --git a/include/jailhouse/console.h b/include/jailhouse/console.h
index 34dd7209..0b6c558b 100644
--- a/include/jailhouse/console.h
+++ b/include/jailhouse/console.h
@@ -50,6 +50,7 @@
#define JAILHOUSE_CON_TYPE_SCIFA 0x0007
#define JAILHOUSE_CON_TYPE_IMX 0x0008
#define JAILHOUSE_CON_TYPE_IMX_LPUART 0x0009
+#define JAILHOUSE_CON_TYPE_RISCV_SBI 0x000a

/* Flags: bit 0 is used to select PIO (cleared) or MMIO (set) access */
#define JAILHOUSE_CON_ACCESS_PIO 0x0000
--
2.36.1

[Ralf Ramsauer]

For demonstration purposes only.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

driver/pci.c | 33 +++++++++++++++++++++------------
1 file changed, 21 insertions(+), 12 deletions(-)

diff --git a/driver/pci.c b/driver/pci.c
index da516479..4f52eba8 100644
--- a/driver/pci.c
+++ b/driver/pci.c
@@ -292,19 +292,21 @@ static unsigned int count_ivshmem_devices(struct cell *cell)
return count;
}

-static const struct of_device_id gic_of_match[] = {
+static const struct of_device_id irqchip_of_match[] = {
{ .compatible = "arm,cortex-a15-gic", },
{ .compatible = "arm,cortex-a7-gic", },
{ .compatible = "arm,gic-400", },
{ .compatible = "arm,gic-v3", },
+ { .compatible = "riscv,plic0", },
+ { .compatible = "sifive,plic-1.0.0", },
{},
};

static bool create_vpci_of_overlay(struct jailhouse_system *config)
{
- u32 address_cells, size_cells, gic_address_cells, gic_phandle;
+ u32 address_cells, size_cells, irqchip_address_cells, irqchip_phandle;
struct device_node *vpci_node = NULL;
- struct device_node *root, *gic;
+ struct device_node *root, *irqchip;
struct property *prop = NULL;
unsigned int n, cell;
u64 base_addr;
@@ -322,15 +324,16 @@ static bool create_vpci_of_overlay(struct jailhouse_system *config)

of_node_put(root);

- gic = of_find_matching_node(NULL, gic_of_match);
- if (!gic)
+ irqchip= of_find_matching_node(NULL, irqchip_of_match);
+ if (!irqchip)
return false;

- if (of_property_read_u32(gic, "#address-cells", &gic_address_cells) < 0)
- gic_address_cells = 0;
- gic_phandle = gic->phandle;
+ if (of_property_read_u32(irqchip, "#address-cells",
+ &irqchip_address_cells) < 0)
+ irqchip_address_cells = 0;
+ irqchip_phandle = irqchip->phandle;

- of_node_put(gic);
+ of_node_put(irqchip);

#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0)
if (of_overlay_fdt_apply(__dtb_vpci_template_begin,
@@ -386,7 +389,7 @@ static bool create_vpci_of_overlay(struct jailhouse_system *config)
}

prop = alloc_prop("interrupt-map",
- sizeof(u32) * (8 + gic_address_cells) * 4);
+ sizeof(u32) * (8 + irqchip_address_cells) * 4);
if (!prop)
goto out;

@@ -394,12 +397,18 @@ static bool create_vpci_of_overlay(struct jailhouse_system *config)
for (n = 0, cell = 0; n < 4; n++) {
cell += 3; /* match addr (0) */
prop_val[cell++] = cpu_to_be32(n + 1); /* match addr */
- prop_val[cell++] = cpu_to_be32(gic_phandle);
- cell += gic_address_cells; /* parent addr (0) */
+ prop_val[cell++] = cpu_to_be32(irqchip_phandle);
+ cell += irqchip_address_cells; /* parent addr (0) */
+#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
prop_val[cell++] = cpu_to_be32(GIC_SPI);
prop_val[cell++] =
cpu_to_be32(config->root_cell.vpci_irq_base + n);
prop_val[cell++] = cpu_to_be32(IRQ_TYPE_EDGE_RISING);
+#elif defined(CONFIG_RISCV)
+ prop_val[cell++] =
+ cpu_to_be32(config->root_cell.vpci_irq_base + n);
+ prop_val[cell++] = 0;
+#endif
}

if (of_changeset_add_property(&overlay_changeset, vpci_node, prop) < 0)
--
2.36.1

[Ralf Ramsauer]

For demonstration purposes only.

Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
---

configs/riscv/dts/noelv-linux-inmate.dts | 17 +++++++++++++++++
configs/riscv/noelv-linux-demo.c | 12 +++++++++++-
configs/riscv/noelv.c | 12 +++++++++++-
configs/riscv/qemu-linux-demo.c | 12 +++++++++++-
configs/riscv/qemu-riscv64.c | 12 +++++++++++-
5 files changed, 61 insertions(+), 4 deletions(-)

diff --git a/configs/riscv/dts/noelv-linux-inmate.dts b/configs/riscv/dts/noelv-linux-inmate.dts
index 486ea82f..ca11b134 100644
--- a/configs/riscv/dts/noelv-linux-inmate.dts
+++ b/configs/riscv/dts/noelv-linux-inmate.dts
@@ -126,5 +126,22 @@
riscv,ndev = <31>;
riscv,max-priority = <7>;
};
+
+ pci@40000000 {

+ #address-cells = <3>;
+ #size-cells = <2>;

+ #interrupt-cells = <1>;
+

+ device_type = "pci";
+ bus-range = <0 0>;

+ reg = <0x0 0x40000000 0x0 0x100000>;
+ ranges = <0x2000000 0x0 0x40100000 0x0 0x40100000 0 0x2000>;

+
+ interrupt-map-mask = <0>;

+ interrupt-map = <0 0 0 1 &plic0 28 0>;
+
+ compatible = "pci-host-ecam-generic";
+ status = "okay";

+ };
};
};
diff --git a/configs/riscv/noelv-linux-demo.c b/configs/riscv/noelv-linux-demo.c

index bd263b78..9e62d6be 100644
--- a/configs/riscv/noelv-linux-demo.c
+++ b/configs/riscv/noelv-linux-demo.c
@@ -20,7 +20,7 @@ struct {
struct jailhouse_cpu cpus[2];
struct jailhouse_memory mem_regions[8];
struct jailhouse_irqchip irqchips[1];
- struct jailhouse_pci_device pci_devices[0];

+ struct jailhouse_pci_device pci_devices[1];

} __attribute__((packed)) config = {
.cell = {
.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
@@ -84,6 +84,16 @@ struct {
},

.pci_devices = {
+ { /* IVSHMEM (networking) */
+ .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
+ .domain = 0x0000,
+ .bdf = 0x10 << 3,
+ .bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_INTX,
+ .shmem_regions_start = 4,
+ .shmem_dev_id = 1,
+ .shmem_peers = 2,
+ .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
+ },
},

.irqchips = {
diff --git a/configs/riscv/noelv.c b/configs/riscv/noelv.c
index 08389cb0..5c8b6fa3 100644
--- a/configs/riscv/noelv.c
+++ b/configs/riscv/noelv.c
@@ -25,7 +25,7 @@ struct {
struct jailhouse_cpu cpus[6];
struct jailhouse_memory mem_regions[7];
struct jailhouse_irqchip irqchips[1];
- struct jailhouse_pci_device pci_devices[0];

+ struct jailhouse_pci_device pci_devices[1];

struct jailhouse_pci_capability pci_caps[0];

} __attribute__((packed)) config = {
.header = {

@@ -139,5 +139,15 @@ struct {
},

.pci_devices = {
+ { /* IVSHMEM (networking) */
+ .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
+ .domain = 0x0000,
+ .bdf = 0x10 << 3,
+ .bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_INTX,
+ .shmem_regions_start = 3,
+ .shmem_dev_id = 0,
+ .shmem_peers = 2,
+ .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
+ },
},
};
diff --git a/configs/riscv/qemu-linux-demo.c b/configs/riscv/qemu-linux-demo.c
index e1947125..0ff62549 100644
--- a/configs/riscv/qemu-linux-demo.c
+++ b/configs/riscv/qemu-linux-demo.c
@@ -20,7 +20,7 @@ struct {
struct jailhouse_cpu cpus[2];
struct jailhouse_memory mem_regions[8];
struct jailhouse_irqchip irqchips[1];
- struct jailhouse_pci_device pci_devices[0];

+ struct jailhouse_pci_device pci_devices[1];

} __attribute__((packed)) config = {
.cell = {
.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
@@ -87,6 +87,16 @@ struct {
},

.pci_devices = {
+ { /* IVSHMEM (networking) */
+ .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
+ .domain = 0x0000,
+ .bdf = 0x10 << 3,
+ .bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_INTX,
+ .shmem_regions_start = 4,
+ .shmem_dev_id = 1,
+ .shmem_peers = 2,
+ .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
+ },
},

.irqchips = {
diff --git a/configs/riscv/qemu-riscv64.c b/configs/riscv/qemu-riscv64.c
index 1ace554b..75102bcc 100644
--- a/configs/riscv/qemu-riscv64.c
+++ b/configs/riscv/qemu-riscv64.c
@@ -36,7 +36,7 @@ struct {
struct jailhouse_cpu cpus [6];
struct jailhouse_memory mem_regions [20];
struct jailhouse_irqchip irqchips [1];
- struct jailhouse_pci_device pci_devices[1];
+ struct jailhouse_pci_device pci_devices[2];

struct jailhouse_pci_capability pci_caps[6];
} __attribute__((packed)) config = {
.header = {

@@ -224,6 +224,16 @@ struct {
.msix_region_size = 0x1000,
.msix_address = 0x40080000,
},
+ { /* IVSHMEM (networking) */
+ .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
+ .domain = 0x0000,
+ .bdf = 0x10 << 3,
+ .bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_INTX,
+ .shmem_regions_start = 16,
+ .shmem_dev_id = 0,
+ .shmem_peers = 2,
+ .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
+ },
},

.pci_caps = {
--
2.36.1

[Jan Kiszka]

On 27.06.22 15:28, Ralf Ramsauer wrote:
> Signed-off-by: Ralf Ramsauer <ralf.r...@oth-regensburg.de>
> ---
> configs/riscv/qemu-riscv64.c | 218 +++++++++++++++++++++++++++++++++--
> include/jailhouse/console.h | 1 +
> 2 files changed, 211 insertions(+), 8 deletions(-)
>

...

> diff --git a/include/jailhouse/console.h b/include/jailhouse/console.h
> index 34dd7209..0b6c558b 100644
> --- a/include/jailhouse/console.h
> +++ b/include/jailhouse/console.h
> @@ -50,6 +50,7 @@
> #define JAILHOUSE_CON_TYPE_SCIFA 0x0007
> #define JAILHOUSE_CON_TYPE_IMX 0x0008
> #define JAILHOUSE_CON_TYPE_IMX_LPUART 0x0009
> +#define JAILHOUSE_CON_TYPE_RISCV_SBI 0x000a
>
> /* Flags: bit 0 is used to select PIO (cleared) or MMIO (set) access */
> #define JAILHOUSE_CON_ACCESS_PIO 0x0000

This is likely needed earlier in order to allow building things.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Jan Kiszka]

On 27.06.22 15:29, Ralf Ramsauer wrote:
> diff --git a/hypervisor/arch/riscv/include/asm/percpu.h b/hypervisor/arch/riscv/include/asm/percpu.h
> index 4eda15b6..bcafff51 100644
> --- a/hypervisor/arch/riscv/include/asm/percpu.h
> +++ b/hypervisor/arch/riscv/include/asm/percpu.h
> @@ -43,6 +43,8 @@ enum sbi_hart_state {
> } hsm; \
> bool wait_for_power_on; \
> bool reset; \
> - bool park;
> + bool park; \
> + u32 virq_enabled_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)]; \
> + u32 virq_pending_bitmap[PLIC_MAX_IRQS / (sizeof(u32) * 8)];

Style issue :)

[Jan Kiszka]

On 27.06.22 15:28, Ralf Ramsauer wrote:

> GCC may emit 4x 1 Byte reads in case of our regular mmio_read32
> accessor, when used in combination with -Os. Yes, I've seen it.
>
> Define safe handlers to overcome this issue.
>

The kernel does something similar, right? Worth to point that out then.

[Ralf Ramsauer]

On 27/06/2022 15:43, Jan Kiszka wrote:
> On 27.06.22 15:28, Ralf Ramsauer wrote:
>> GCC may emit 4x 1 Byte reads in case of our regular mmio_read32
>> accessor, when used in combination with -Os. Yes, I've seen it.
>>
>> Define safe handlers to overcome this issue.
>>
>
> The kernel does something similar, right? Worth to point that out then.

Yes, but that's not limited to RISC-V, and there's no particular comment
on that issue in the commit logs in the kernel. Maybe they never
experienced it, as basically all architectures build up their mmio
accessors by using assembly per default.

Ralf

[Ralf Ramsauer]

Hmm, I checked that every commit compiles. So in fact there's no user of
JAILHOUSE_CON_TYPE_RISCV_SBI. This is still missing in dbg-write.c. Will
fix it.

Thanks
Ralf