Bug#1050030: Potential regression: ovmf package update from Debian bookworm breaks Xen HVM boot

[Paul Leiber]

Package: ovmf
Version: 2022.11-6
Severity: important

Dear Maintainer,

After upgrading from Debian Bullseye to Debian Bookworm, existing HVM
DomUs using "firmware = 'ovmf'" (specifically Windows Server 2022 and
Windows 10) can't boot anymore. Booting these systems leads to the
Windows error 0xc0000225. I wasn't able to fix this error. Booting an
installation .iso leads to the same error. Booting the installation
media with "firmware = 'bios'" leads to a normal boot.

This seems to be the effect of a change in ovmf sources, where a xen
specific platform was created in 2019:
https://lore.kernel.org/all/20190813113119.148...@citrix.com/#t

With some help I found a workaround. I could successfully start the
Windows DomU with ovmf firmware after removing the current ovmf package
and installing a previous ovmf package from Debian repositories,
specifically version 2020.11-2+deb11u1. This strongly indicates that the
cause of this issue lies in the ovmf package.

My HVM config file:

type = "hvm"
memory = 6144
vcpus = 2
name = "kalliope"
firmware = 'ovmf'
firmware = '/usr/local/lib/ovmf/OVMF.fd'
vif = ['bridge=xenbr0,mac=XX:XX:XX:XX:XX:XX,ip=10.0.0.4']
disk = ['phy:/dev/vg0/matrix,hda,w']
device_model_version = 'qemu-xen'
hdtype = 'ahci'
pae = 1
acpi = 1
apic = 1
vga = 'stdvga'
videoram = 16
xen_platform_pci = 1
vendor_device = 'xenserver'
viridian = 1
on_crash = 'restart'
device_model_args_hvm = [
# Debug OVMF
'-chardev', 'file,id=debugcon,path=/var/log/xen/ovmf.log,',
'-device', 'isa-debugcon,iobase=0x402,chardev=debugcon',
]
sdl = 0
serial = 'pty'
vnc = 1
vnclisten = "0.0.0.0"
vncpasswd = ""

As the Windows systems are not usable anymore, Xen is significantly
reduced in functionality after the upgrade.

There is a Debian bug report which could be related, I also described my
situation there: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978595

This (or at least a very similar) issue seems to be fixed in
Redhat/Fedora. A related bug report exists in
https://bugzilla.redhat.com/show_bug.cgi?id=2170930


Additional information:

I also tried building Ovmf following
https://lore.kernel.org/all/20190813113119.148...@citrix.com/#t
, but I wasn't fully able to create a working system:

(1) Using the resulting OVMF.fd from the build process with "firmware
='/path/to/new/OVMF.fd' led consistently to a black screen in VNC or
Spice with the text "Guest has not initialized the display (yet)".

(2) Replacing the OVMF.fd in /var/lib/ovmf with the freshly built
OVMF.fd led to a slight improvement. I could then boot the Windows
Server installation .iso, but booting the Windows 10 installation .iso
lead to a crash where the TianoCore logo was visible, but nothing
happened. The two existing DomUs were still no>

However, I am not sure that I followed the procedure correctly, I might
very well have done something very wrong. Any pointers are welcome.

Thanks,

Paul

-- System Information:
Debian Release: 12.1
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-11-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

-- no debconf information

[Elliott Mitchell]

affects 1050030 src:xen
quit

I'm seeing a similar situation, though instead using FreeBSD/x86 in the
VM.

For FreeBSD the bootloader appears to operate normally, but something
fails quickly after loading the kernel:

Loading kernel...
/boot/kernel/kernel text=0x18aa98 text=0xdfd150 text=0x675154 data=0x140 data=0x1c38e8+0x43b718 0x8+0x18fe70+0x8+0x1ae449/
Loading configured modules...
/boot/entropy size=0x1000
/etc/hostid size=0x25
staging 0xe3e00000 (not copying) tramp 0xe351b000 PT4 0xe3512000
Start @ 0xffffffff8038b000 ...
EFI framebuffer information:
addr, size 0xf0000000, 0x1d5000
dimensions 800 x 600
stride 800
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

I believe all these messages are from FreeBSD's bootloader. The first
message from the kernel should be "---<<BOOT>>---", yet that message
never shows. Xen shows the domain spinning on a single processor which
makes me believe the FreeBSD kernel has loaded, panic()ed and the
debugger is loaded (but there is no VGA console).


From reading the available information I suspect Tianocore/EDK2 may have
tried to move some functionality to a distinct build and neither setup
quite works. Notably there is now a "OvmfPkg/OvmfXen.dsc" build
configuration. The OVMF.fd for Qemu for Xen functionality may have been
moved /here/. There might also be an attempt at functionality similar to
"ArmVirtPkg/ArmVirtXen.dsc" (Debian 978595) for x86.


--
(\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/)
\BS ( | ehem+...@m5p.com PGP 87145445 | ) /
\_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[Elliott Mitchell]

On Fri, Aug 18, 2023 at 02:05:31PM -0700, Elliott Mitchell wrote:
> >From reading the available information I suspect Tianocore/EDK2 may have
> tried to move some functionality to a distinct build and neither setup
> quite works. Notably there is now a "OvmfPkg/OvmfXen.dsc" build
> configuration. The OVMF.fd for Qemu for Xen functionality may have been
> moved /here/. There might also be an attempt at functionality similar to
> "ArmVirtPkg/ArmVirtXen.dsc" (Debian 978595) for x86.

Now confirmed reverting to 2020.11-2+deb11u1 takes care of the issues I'm
running into. I've been able to build OvmfPkg/OvmfXen.dsc, but haven't
gotten it to do anything. I'm suspecting the support for running
headless didn't get into OvmfXen. I'm interacting with someone
knowledgeable, but nothing yet.

I suspect the "ovmf" package needs to be split. I've gotten the
impression the build needed for normal `qemu` isn't going to be the same
as the build needed for xen-qemu.

I think what is really needed is for xen-utils-X.YY to Recommend a
virtual package "xen-domu-bootloader" which is then provided by tools
which can load VMs. The current other in-service tool is grub-xen-host,
but it appears OvmfXen may also be able to provide the service.

I'm attaching two patches which should help organize the source package.
These leave all the "./edksetup.sh" lines identical. Perhaps make use
of this to make the build cleaner?