poweroff without HTIF

[Richard W.M. Jones]

I'm using Michael's riscv-qemu (07146abe22e5ea5) and the Linux
riscv-next branch (418457520fbb). I have a small C program called
‘poweroff’ which simply calls ‘reboot (LINUX_REBOOT_CMD_POWER_OFF);’
but this isn't powering off (exiting) qemu.

So I was interested to find out what actually happens (or at least is
supposed to happen) during power off, since I'd never looked in any
detail before.

The kernel calls the riscv-specific code arch/riscv/kernel/reset.c:

static inline void sbi_shutdown(void)
{
SBI_CALL_0(SBI_SHUTDOWN);
}

void machine_power_off(void)
{
sbi_shutdown();
while (1);
}

which ends up as:

sbi_shutdown();
ffffffe0000183ac: 4501 li a0,0
ffffffe0000183ae: 4581 li a1,0
ffffffe0000183b0: 4601 li a2,0
ffffffe0000183b2: 48a1 li a7,8 // SBI_SHUTDOWN
ffffffe0000183b4: 00000073 ecall
while (1);
ffffffe0000183b8: a001 j ffffffe0000183b8 <machine_power_off+0x12>

ecall calls from S-mode into M-mode, and I believe from there we end
up in riscv-pk machine/mtrap.c:

void poweroff(uint16_t code)
{
printm("Power off\n");
finisher_exit(code);
if (htif) {
htif_poweroff();
} else {
while (1);
}
}

This is a problem because this version of qemu isn't emulating HTIF.

Now at this point I've no idea. Is there some other way planned to do
power off? (I guess without ACPI and friends, for now)

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Karsten Merker]

On Thu, Dec 21, 2017 at 03:30:18PM +0000, Richard W.M. Jones wrote:

> I'm using Michael's riscv-qemu (07146abe22e5ea5) and the Linux
> riscv-next branch (418457520fbb). I have a small C program called
> ‘poweroff’ which simply calls ‘reboot (LINUX_REBOOT_CMD_POWER_OFF);’
> but this isn't powering off (exiting) qemu.

[...]

> ecall calls from S-mode into M-mode, and I believe from there we end
> up in riscv-pk machine/mtrap.c:
>
> void poweroff(uint16_t code)
> {
> printm("Power off\n");
> finisher_exit(code);
> if (htif) {
> htif_poweroff();
> } else {
> while (1);
> }
> }
>
> This is a problem because this version of qemu isn't emulating HTIF.
>
> Now at this point I've no idea. Is there some other way planned to do
> power off? (I guess without ACPI and friends, for now)

AFAIK Michael Clark has been thinking about emulating the PMU
part from the FE310 in qemu for signalling reboot and poweroff.

From https://github.com/riscv/riscv-qemu/pull/82:
"We still need to solve power and reset, but this is a general
problem for the non-HTIF boards. We'll likely add the SiFive AON
block in a future PR."

Regards,
Karsten
--
Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung
sowie der Weitergabe meiner personenbezogenen Daten für Zwecke der
Werbung sowie der Markt- oder Meinungsforschung.

[Richard W.M. Jones]

Ah thanks, I had actually read that PR but didn't see (or
understand) that comment at the time.

FWIW to get around my immediate problem I added a fixed
address to the qemu emulation which when written to causes
qemu to exit. Not very nice, but gets the job done in the
short term :-)

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org

[Palmer Dabbelt]

If I understand correctly, that's essentially just a very simple AON block
emulation. The FE310's AON block comes setup with a default sleep program that
turns the power off, so all you should have to do is write to the "pmusleep"
register to turn off the chip (well, technycally just deassert the PMU's
control lines, but that's as close as we can get).

I think all you need to do is wrap that up in a Device Tree entry that looks
something like

L13: aon@10000000 {
compatible = "sifive,aon0";
interrupt-parent = <&L0>;
interrupts = <1 2>;
reg = <0x10000000 0x1000>;
reg-names = "control";
};

and then make BBL parse that device tree entry like it does for everything
else, using the offset 0x148 from the "sifive,aon0" block's base as the magic
address for the "pmusleep" register.

See section 13 of https://static.dev.sifive.com/FE310-G000.pdf for more details
on how the AON block works, but if you just trap on writes to any other
register in the AON block's address space then you can just call these missing
features instead of bugs :). At least this way the target software will remain
compatible.

[Michael Clark]

I agree we should use the AON. HTIF should be limited to the Spike boards as it doesn’t exist on any of the other boards. We’d like the virt board to share peripherals with the E300 and U500 boards so AON device emulation makes the most sense.

I’m currently working on syncing with qemu master. I’ll put the AON on the top of the backlog. We might need to teach bbl to power off via the AON, so that the kernel can continue to use the SBI interfaces, otherwise we'll need a driver in Linux.

[Richard W.M. Jones]

On Thu, Dec 21, 2017 at 01:49:56PM -0800, Palmer Dabbelt wrote:
> On Thu, 21 Dec 2017 11:47:42 PST (-0800), rjo...@redhat.com wrote:
> >FWIW to get around my immediate problem I added a fixed
> >address to the qemu emulation which when written to causes
> >qemu to exit. Not very nice, but gets the job done in the
> >short term :-)
>
> If I understand correctly, that's essentially just a very simple AON
> block emulation.

In terms of what I'm using as a work-around it's really dumb to the
point of stupidity, see attached. Emulating a real device would of
course be far better.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

[Michael Clark]

> On 22/12/2017, at 11:29 AM, Richard W.M. Jones <rjo...@redhat.com> wrote:
>
> On Thu, Dec 21, 2017 at 01:49:56PM -0800, Palmer Dabbelt wrote:
>> On Thu, 21 Dec 2017 11:47:42 PST (-0800), rjo...@redhat.com wrote:
>>> FWIW to get around my immediate problem I added a fixed
>>> address to the qemu emulation which when written to causes
>>> qemu to exit. Not very nice, but gets the job done in the
>>> short term :-)
>>
>> If I understand correctly, that's essentially just a very simple AON
>> block emulation.
>
> In terms of what I'm using as a work-around it's really dumb to the
> point of stupidity, see attached. Emulating a real device would of
> course be far better.

Yap. Understand.

As a short term measure, if you comment the following line, you should get power-off and reset working in the virt board:

- https://github.com/riscv/riscv-qemu/blob/07146abe22e5ea5948fda5d511fd1fd228e196fa/hw/riscv/virt.c#L47

Only you’ll also get a console that drops characters, as the HTIF console doesn’t have any queuing support. This would be fine if for example you are using ssh to access the virtual machine. Currently the HTIF console / power-off and reset are intertwined into the same device, so by enabling HTIF, you’ll lose the UART console, and as I mentioned, it is unlikely that we would use HTIF for power-off and reset on any boards other than the Spike boards.

I would like to remove the HTIF code from the virt board once we have AON emulation working for power-off and reset. With the 16550a UART, we are using an unmodified QEMU device and unmodified linux-kernel device driver. The HTIF console on the other hand, will require a lot of effort to get to the same level of capability as the mature device and drivers we are using in its place. It’s likely effort will go into completing SiFive UART emulation before we do anything substantial to the HTIF console driver.

Palmer, I can prioritise AON over synching with upstream? What do you think? Currently i’ve got everything compiling against upstream but am debugging an issue with CPU reset and start up. There have been quite a few changes in upstream over the year between the last sync…

Michael.

[Richard W.M. Jones]

On Fri, Dec 22, 2017 at 11:34:32AM +1300, Michael Clark wrote:
> - https://github.com/riscv/riscv-qemu/blob/07146abe22e5ea5948fda5d511fd1fd228e196fa/hw/riscv/virt.c#L47
>
> Only you’ll also get a console that drops characters, as the HTIF
> console doesn’t have any queuing support. This would be fine if for
> example you are using ssh to access the virtual machine. Currently
> the HTIF console / power-off and reset are intertwined into the same
> device, so by enabling HTIF, you’ll lose the UART console, and as I
> mentioned, it is unlikely that we would use HTIF for power-off and
> reset on any boards other than the Spike boards.
>
> I would like to remove the HTIF code from the virt board once we
> have AON emulation working for power-off and reset. With the 16550a
> UART, we are using an unmodified QEMU device and unmodified
> linux-kernel device driver.

I agree, removing HTIF completely would be excellent. It was very
unreliable when we were doing the previous bootstrap. There's really
no need to work on it when there are real or virtio devices which can
be used instead.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Michael Clark]

> On 22/12/2017, at 11:39 AM, Richard W.M. Jones <rjo...@redhat.com> wrote:
>
> On Fri, Dec 22, 2017 at 11:34:32AM +1300, Michael Clark wrote:
>> - https://github.com/riscv/riscv-qemu/blob/07146abe22e5ea5948fda5d511fd1fd228e196fa/hw/riscv/virt.c#L47
>>
>> Only you’ll also get a console that drops characters, as the HTIF
>> console doesn’t have any queuing support. This would be fine if for
>> example you are using ssh to access the virtual machine. Currently
>> the HTIF console / power-off and reset are intertwined into the same
>> device, so by enabling HTIF, you’ll lose the UART console, and as I
>> mentioned, it is unlikely that we would use HTIF for power-off and
>> reset on any boards other than the Spike boards.
>>
>> I would like to remove the HTIF code from the virt board once we
>> have AON emulation working for power-off and reset. With the 16550a
>> UART, we are using an unmodified QEMU device and unmodified
>> linux-kernel device driver.
>
> I agree, removing HTIF completely would be excellent. It was very
> unreliable when we were doing the previous bootstrap. There's really
> no need to work on it when there are real or virtio devices which can
> be used instead.

Yap. virtio-serial would be an option but there are bugs with chardev handling in the current snapshot of upstream that we are currently ported to, that make dynamic plugging of chardevs difficult or perhaps impossible. I’ve noticed that the chardev system has been overhauled in upstream qemu as I have been porting drivers to it. chardev handling is where i’ve seen the most changes, besides some cpu initialization changes I am currently debugging. There seems to be chardev front-end / back-end hotplugging support in current master.

The issue with virtio-console in the current tree is there is no easy way to configure virtio-serial as the first UART device. I tried. We are just adding “generic” virtio_mmio transports to device tree. When I try to plug virtio-serial I get an error about serial0 already in use. I also note that the arm virt board uses a custom UART for console, not virtio-serial. There were no other boards that plug virtio-serial as the default console.

So we went for the 16550a as it has mature drivers on both sides.

Once we’ve synced with upstream, we can take a look at the chardev stuff again and see if its posible to use virtio-serial. I think upstream allows dynamic plugging of chardevs so using virtio-serial may become easier.

[Richard W.M. Jones]

On Fri, Dec 22, 2017 at 11:58:56AM +1300, Michael Clark wrote:
> The issue with virtio-console in the current tree is there is no easy way to configure virtio-serial as the first UART device. I tried. We are just adding “generic” virtio_mmio transports to device tree. When I try to plug virtio-serial I get an error about serial0 already in use. I also note that the arm virt board uses a custom UART for console, not virtio-serial. There were no other boards that plug virtio-serial as the default console.
>
> So we went for the 16550a as it has mature drivers on both sides.
>
> Once we’ve synced with upstream, we can take a look at the chardev stuff again and see if its posible to use virtio-serial. I think upstream allows dynamic plugging of chardevs so using virtio-serial may become easier.

The main problem with virtio-serial/-console is it requires the whole
virtio-pci/-mmio "stack" to be working before you can reliably get
messages (or using earlycon which can be an exercise in frustration).
Also there was talk at KVM Forum of deprecation.

Simple UARTs are much better for debugging. Using 16550A is a great
choice IMHO.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Michael Clark]

> On 22/12/2017, at 12:13 PM, Richard W.M. Jones <rjo...@redhat.com> wrote:
>
> On Fri, Dec 22, 2017 at 11:58:56AM +1300, Michael Clark wrote:
>> The issue with virtio-console in the current tree is there is no easy way to configure virtio-serial as the first UART device. I tried. We are just adding “generic” virtio_mmio transports to device tree. When I try to plug virtio-serial I get an error about serial0 already in use. I also note that the arm virt board uses a custom UART for console, not virtio-serial. There were no other boards that plug virtio-serial as the default console.
>>
>> So we went for the 16550a as it has mature drivers on both sides.
>>
>> Once we’ve synced with upstream, we can take a look at the chardev stuff again and see if its posible to use virtio-serial. I think upstream allows dynamic plugging of chardevs so using virtio-serial may become easier.
>
> The main problem with virtio-serial/-console is it requires the whole
> virtio-pci/-mmio "stack" to be working before you can reliably get
> messages (or using earlycon which can be an exercise in frustration).
> Also there was talk at KVM Forum of deprecation.
>
> Simple UARTs are much better for debugging. Using 16550A is a great
> choice IMHO.

Okay. We’ll stick to that.

It is likely still possible to add a virtio-serial device using the right qemu command-line incantation, perhaps after we’ve forward ported to current master as there seems to be chardev hot-plugging support. However it’s not necessary that we make any changes to support this as the virtio-mmio transport can be used to plug any of the supported virtio device types. We’ll keep the 16550A as the default UART on the virt board.