Bug#983357: Netinst crashes xen domU when loading kernel

[Phillip Susi]

Package: debian-installer
Version: 20201202

Every bullseye netinst image I have tried to boot in a xen domU has
crashed and rebooted the domU after choosing any entry from the boot
menu. I thought there might have been something wrong with my xen
server, which was running Ubuntu 18.04, but I rebuilt it the other day
using bullseye as the dom0, and I still can't install bullseye in a
domU.

[Phillip Susi]

reassign 983357 linux
severity grave
thanks

I rebuilt the iso using the version of isolinux from stable and it still
crashed the domU. When I rebuilt it using the vmlinux and initrd.gz
from the stable iso, it successfully boots, so it appears to be caused
by the kernel.

Interestingly, there appears to be a different kernel build just for use
under xen in install.amd/xen and using that one also works. Maybe we
need a menu option in isolinux to load that kernel instead?

[Phillip Susi]

reopen 983357
thanks

I don't know what happened, but it is back to not working, even with the
install.amd/xen/ kernel.

Phillip Susi writes:

> The netinst image I had contained the -3 kernel. I rebuit the image
> with the current -6 kernel and it worked. I downloaded the latest
> weekly netinst iso and it already contains the -6 kernel, so it appears
> that this has been fixed in -4, -5, or -6.

[Phillip Susi]

affects 983357 + debian-installer
severify 983357 serious
thanks

It appears that the root cause of this bug has been reported upstream
here:

https://bugzilla.kernel.org/show_bug.cgi?id=207695

It seems that there is an error trying to udev trigger the Xen virtual
keyboard, and this causes start-udev to bail out, which causes init to
bail out and the kernel to panic. Removing the set -e from start-udev
appears to be a viable workaround that d-i might want to consider.

[Phillip Susi]

I bisected the problem to this commit:

commit df44b479654f62b478c18ee4d8bc4e9f897a9844
Author: Peter Rajnoha <praj...@redhat.com>
Date: Wed Dec 5 12:27:44 2018 +0100

kobject: return error code if writing /sys/.../uevent fails

Propagate error code back to userspace if writing the /sys/.../uevent
file fails. Before, the write operation always returned with success,
even if we failed to recognize the input string or if we failed to
generate the uevent itself.

With the error codes properly propagated back to userspace, we are
able to react in userspace accordingly by not assuming and awaiting
a uevent that is not delivered.

Signed-off-by: Peter Rajnoha <praj...@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gre...@linuxfoundation.org>

So it appears that the Xen Virtual Keyboard driver was always broken but
the error in triggering the uevent was not previously reported. The
upstream bug report notes another driver failing. There are probably
other drivers too.

I will continue to try to find and fix the Xen keyboard error so it no
longer fails anyway, but it is probably a good idea to patch the
start-udev script in d-i to ignore errors. It is better to continue
with some device not triggering a cold plug event than to instantly panic
the kernel in early boot.

[Phillip Susi]

I dug down to the the -ENOMEM coming from the fact that the modalias is
over 2KB of crap so it won't fit in the environment block when the input
core tries to add it for the uevent. I don't see how it gets this way
though because the MODULE_ALIAS() statement in the code just says it
should be "xen: vkbd".

When I read the modalias in sysfs, it says:

input:b0001v5853pFFFFe0000-e0,1,k71,72,73,74,75,76,77,78,79,7A,7B,7C,7D,\
7E,7F,80,81,82,83,84,85,86,87,88,89,8A,8B,8C,8D,8E,8F,90,91,92,93,94,95,\
96,97,98,99,9A,9B,9C,9D,9E,9F,A0,A1,A2,A3,A4,A5,A6,A7,A8,A9,AA,AB,AC,AD,\
AE,AF,B0,B1,B2,B3,B4,B5,B6,B7,B8,B9,BA,BB,BC,BD,BE,BF,C0,C1,C2,C3,C4,C5,\
C6,C7,C8,C9,CA,CB,CC,CD,CE,CF,D0,D1,D2,D3,D4,D5,D6,D7,D8,D9,DA,DB,DC,DD,\
DE,DF,E0,E1,E2,E3,E4,E5,E6,E7,E8,E9,EA,EB,EC,ED,EE,EF,160,161,162,163,16\
4,165,166,167,168,169,16A,16B,16C,16D,16E,16F,170,171,172,173,174,175,17\
6,177,178,179,17A,17B,17C,17D,17E,17F,180,181,182,183,184,185,186,187,18\
8,189,18A,18B,18C,18D,18E,18F,190,191,192,193,194,195,196,197,198,199,19\
A,19B,19C,19D,19E,19F,1A0,1A1,1A2,1A3,1A4,1A5,1A6,1A7,1A8,1A9,1AA,1AB,1A\
C,1AD,1AE,1AF,1B0,1B1,1B2,1B3,1B4,1B5,1B6,1B7,1B8,1B9,1BA,1BB,1BC,1BD,1B\
E,1BF,1C0,1C1,1C2,1C3,1C4,1C5,1C6,1C7,1C8,1C9,1CA,1CB,1CC,1CD,1CE,1CF,1D\
0,1D1,1D2,1D3,1D4,1D5,1D6,1D7,1D8,1D9,1DA,1DB,1DC,1DD,1DE,1DF,1E0,1E1,1E\
2,1E3,1E4,1E5,1E6,1E7,1E8,1E9,1EA,1EB,1EC,1ED,1EE,1EF,1F0,1F1,1F2,1F3,1F\
4,1F5,1F6,1F7,1F8,1F9,1FA,1FB,1FC,1FD,1FE,1FF,200,201,202,203,204,205,20\
6,207,208,209,20A,20B,20C,20D,20E,20F,210,211,212,213,214,215,216,217,21\
8,219,21A,21B,21C,21D,21E,21F,220,221,222,223,224,225,226,227,228,229,22\
A,22B,22C,22D,22E,22F,230,231,232,233,234,235,236,237,238,239,23A,23B,23\
C,23D,23E,23F,240,241,242,243,244,245,246,247,248,249,24A,24B,24C,24D,24\
E,24F,250,251,252,253,254,255,256,257,258,259,25A,25B,25C,25D,25E,25F,26\
0,261,262,263,264,265,266,267,268,269,26A,26B,26C,26D,26E,26F,270,271,27\
2,273,274,275,276,277,278,279,27A,27B,27C,27D,27E,27F,280,281,282,283,28\
4,285,286,287,288,289,28A,28B,28C,28D,28E,28F,290,291,292,293,294,295,29\
6,297,298,299,29A,29B,29C,29D,29E,29F,2A0,2A1,2A2,2A3,2A4,2A5,2A6,2A7,2A\
8,2A9,2AA,2AB,2AC,2AD,2AE,2AF,2B0,2B1,2B2,2B3,2B4,2B5,2B6,2B7,2B8,2B9,2B\
A,2BB,2BC,2BD,2BE,2BF,2C0,2C1,2C2,2C3,2C4,2C5,2C6,2C7,2C8,2C9,2CA,2CB,2C\
C,2CD,2CE,2CF,2D0,2D1,2D2,2D3,2D4,2D5,2D6,2D7,2D8,2D9,2DA,2DB,2DC,2DD,2D\
E,2DF,2E0,2E1,2E2,2E3,2E4,2E5,2E6,2E7,2E8,2E9,2EA,2EB,2EC,2ED,2EE,2EF,2F\
0,2F1,2F2,2F3,2F4,2F5,2F6,2F7,2F8,2F9,2FA,2FB,2FC,2FD,2FE,ramlsfw

[Phillip Susi]

The discussion upstream does not seem to be converging on a proper fix
in the kernel, so I'm going to clone this bug and suggest that
debian-installer patch the start-udev script to ignore the failure of
the udevadm trigger command.

To summarize: init ends up calling start-udev which calls udevadm
trigger to cold plug all devices. Both scripts are set -e. The Xen
Virtual Keyboard driver and at least one other driver have always failed
to trigger due to having absurdly long modalias, but the error used to
be ignored. The kernel now returns the error to udevadm, so it exits
with an error, so start-udev exits with an error, so init exits with an
error, causing the kernel to panic.

[Cyril Brulebois]

Hi Phillip,

And thanks for debugging this… I must confess I've never touched
anything Xen related and I'd like to keep it that way in the near
future. ;-)

Phillip Susi <ph...@thesusis.net> (2021-05-19):

Well, it's a little more complicated:
- start-udev is actually a script shipped by the udev udeb, i.e. the
responsibility of systemd maintainers;
- based on a quick grep, the installer contains two calls to that
start-udev script, in the rootskel source package:
+ rootskel/src/init (shipped as /init in the udeb)
+ rootskel/src/sbin/init-linux (shipped as /sbin/init in the udeb)

I'd be happy to have a comment from systemd maintainers before thinking
about patching rootskel. :)


Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[Michael Biebl]

Hi Phillip

Am 24.05.2021 um 06:19 schrieb Cyril Brulebois:
>> trigger to cold plug all devices. Both scripts are set -e. The Xen
>> Virtual Keyboard driver and at least one other driver have always failed
>> to trigger due to having absurdly long modalias, but the error used to
>> be ignored. The kernel now returns the error to udevadm

So this is a change in behaviour in the kernel?
What happens if you boot the installed system? Does udevadm trigger fail
there as well?

I feel a bit uneasy changing the udev start script this late in the
release cycle (especially when it appears like covering up an issue
someplace else).

I'll let Marco make the judgement on this though, as he has the most
experience with those udev udeb start scripts as the original author.

Michael

[Phillip Susi]

Michael Biebl writes:

> So this is a change in behaviour in the kernel?

Yes, this commit fixed the kernel to report the error instead of
silently failing:

commit df44b479654f62b478c18ee4d8bc4e9f897a9844
Author: Peter Rajnoha <praj...@redhat.com>
Date: Wed Dec 5 12:27:44 2018 +0100

kobject: return error code if writing /sys/.../uevent fails

Propagate error code back to userspace if writing the /sys/.../uevent
file fails. Before, the write operation always returned with success,
even if we failed to recognize the input string or if we failed to
generate the uevent itself.

With the error codes properly propagated back to userspace, we are
able to react in userspace accordingly by not assuming and awaiting
a uevent that is not delivered.

Signed-off-by: Peter Rajnoha <praj...@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gre...@linuxfoundation.org>

> What happens if you boot the installed system? Does udevadm trigger fail
> there as well?

Yes, it does; that is how I was able to track down the problem.

> I feel a bit uneasy changing the udev start script this late in the
> release cycle (especially when it appears like covering up an issue
> someplace else).
>
> I'll let Marco make the judgement on this though, as he has the most
> experience with those udev udeb start scripts as the original author.

So far I have been removing the -e from the shbang line in the
start-udev script and remastering the iso so I can get it to boot. It
would probably be a better idea to just add a || true to the udevadm
trigger call. I feel fairly certain that no matter what the cause of
the coldplug failure, the user is going to be better off ignoring it and
trying to proceed than a kernel panic.

[Ben Hutchings]

On Tue, 2021-08-24 at 10:56 -0400, Chuck Zmudzinski wrote:

> After reviewing Philip's message at
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983357#43
>
> which seems to point to the root cause of this bug, I can add:
>
> On my Xen HVM DomU I see the absurdly long modalias for the Xen
> Virtual keyboard that seems to be causing this crash in sysfs at
>
> /sys/devices/virtual/input/input2/modalias
>
> But at /sys/devices/vkbd-0/modalias, I see just 'xen:vkbd', which would
> probably not result in an error in the udev script if this was also
> written as the modalias at /sys/devices/virtual/input/input2/modalias
>
> So the Xen virtual keyboard appears more than once in sysfs, and
> modalias is not the same in the different places. This seems
> to be a problem.

They are two different devices, and they should have different
modaliases.

Linux has code for discovering devices on each kind of bus, including
virtual buses, and that code creates "bus devices" such as vkbd-0. At
this point the kernel doesn't know what the device is capable of. The
modalias for a bus device carries some identifying information that can
be used to select a driver module for it.

The driver does know what the device is capable of, and how to use it.
It will normally create one or more "class devices" that support a
particular set of operations; in this case input device operations.
Class devices typically don't have modaliases, since they don't need
another layer of drivers on top. However, for input devices the
modalias carries information about the device's capabilities. These
may trigger loading of the evdev or joydev module.

> I understand the correct way to fix this bug is by modifying the
> Xxen virtual keyboard (and any other devices that might cause
> this crash) and not the start-udev script on the netinst
> installation media, which is so far the only available workaround.
> Hopefully Xen will accept a fix if we can come up with a fix.
[...]

I think a proper fix would be one of:

a. If the Xen virtual keyboard driver is advertising capabilities it
doesn't have, stop it doing that.
b. Change the implementation of modalias attributes to allow longer
values.

It's not clear to me whether the Xen driver is advertising correctly or
not. If it is, then the solution should be b, but that may be too
disruptive a change to the kernel. So a reasonable workaround might
be:

c. Change the input subsystem to limit the length of the
capabilities part of the modalias.


Ben.

--
Ben Hutchings
73.46% of all statistics are made up.

[Phillip Susi]

Ben Hutchings <b...@decadent.org.uk> writes:

> I think a proper fix would be one of:
>
> a. If the Xen virtual keyboard driver is advertising capabilities it
> doesn't have, stop it doing that.
> b. Change the implementation of modalias attributes to allow longer
> values.
>
> It's not clear to me whether the Xen driver is advertising correctly or
> not. If it is, then the solution should be b, but that may be too
> disruptive a change to the kernel. So a reasonable workaround might
> be:
>
> c. Change the input subsystem to limit the length of the
> capabilities part of the modalias.

The problem with a) is that the Xen keyboard is not a physical keyboard
and so it has no way of knowing what keys it actually has. It is a fake
input device designed to pass through whatever input the Xen hypervisor
sends down. As such, any key could come in. If it doesn't advertise
that it has all of these keys, then they would not be accepted by
libinput when the hypervisor sends them down.

This seems to be the heart of the problem: libinput was designed
assuming that all keyboards can and must report what keys are actually
present, and then libinput tries to cram that information into the
modalias rather than some other sysfs attribute as it should ( or not at
all... I still don't see how this information is actually supposed to be
useful to userspace ).

As for b), the problem isn't with the modalias attribute itself, but
when the kernel tries to copy it into the environment block for the udev
callout. The environment block is only a single page, and so limited to
4 KB. And that's for everything else that goes into the environment,
not just the modalias.

[Ben Hutchings]

On Tue, Aug 24, 2021 at 03:27:19PM -0400, Phillip Susi wrote:
>
> Ben Hutchings <b...@decadent.org.uk> writes:
>
> > I think a proper fix would be one of:
> >
> > a. If the Xen virtual keyboard driver is advertising capabilities it
> > doesn't have, stop it doing that.
> > b. Change the implementation of modalias attributes to allow longer
> > values.
> >
> > It's not clear to me whether the Xen driver is advertising correctly or
> > not. If it is, then the solution should be b, but that may be too
> > disruptive a change to the kernel. So a reasonable workaround might
> > be:
> >
> > c. Change the input subsystem to limit the length of the
> > capabilities part of the modalias.
>
> The problem with a) is that the Xen keyboard is not a physical keyboard
> and so it has no way of knowing what keys it actually has. It is a fake
> input device designed to pass through whatever input the Xen hypervisor
> sends down. As such, any key could come in. If it doesn't advertise
> that it has all of these keys, then they would not be accepted by
> libinput when the hypervisor sends them down.

Right, that's what I feared.

xen-kbdfront is setting the bits for keys in the ranges [KEY_ESC,
KEY_UNKNOWN) and [KEY_OK, KEY_MAX), which I think works out to 654
keys and 2362 bytes in the modalias.

> This seems to be the heart of the problem: libinput was designed
> assuming that all keyboards can and must report what keys are actually
> present, and then libinput tries to cram that information into the
> modalias rather than some other sysfs attribute as it should ( or not at
> all... I still don't see how this information is actually supposed to be
> useful to userspace ).

I think modaliases aren't intended to be interpreted by user-space,
other than processing wildcards when matching to modules.

For input devices, the same information is available through other
variables in the uevent, in a more compact form. The information *is*
useful for user-space; e.g. in initramfs-tools we recognise keyboard
devices and add their drivers to the initramfs but ignore other input
devices.

> As for b), the problem isn't with the modalias attribute itself, but
> when the kernel tries to copy it into the environment block for the udev
> callout. The environment block is only a single page, and so limited to
> 4 KB. And that's for everything else that goes into the environment,
> not just the modalias.

Text-based sysfs attributes are limited to a page, but udev receives
uevents through netlink, not sysfs.

The current limit on the environment of a uevent appears to be 2 KB
(UEVENT_BUFFER_SIZE defined in <linux/kobject.h>). That seems like it
*might* be easier to change, so long as user-space doesn't have a
similar limit.

I looked into systemd/udev, and it seems to use an 8 KB buffer for
receiving uevents:

https://sources.debian.org/src/systemd/247.9-1/src/libsystemd/sd-device/device-monitor.c/?hl=390#L390

But as a first step I think increasing the kernel buffer size to 4 KB
would be enough. Perhaps someone could test whether this patch to the
domU kernel makes udev happier:

--- a/include/linux/kobject.h
+++ b/include/linux/kobject.h
@@ -30,7 +30,7 @@

#define UEVENT_HELPER_PATH_LEN 256
#define UEVENT_NUM_ENVP 64 /* number of env pointers */
-#define UEVENT_BUFFER_SIZE 2048 /* buffer for the variables */
+#define UEVENT_BUFFER_SIZE 4096 /* buffer for the variables */

#ifdef CONFIG_UEVENT_HELPER
/* path to the userspace helper executed on an event */
--- END ---

?

Ben.

--
Ben Hutchings
Design a system any fool can use, and only a fool will want to use it.

[Ben Hutchings]

On Tue, 2021-08-24 at 15:19 -0400, Chuck Zmudzinski wrote:

> On 8/24/2021 1:12 PM, Ben Hutchings wrote:
[...]

> > I think a proper fix would be one of:
> >
> > a. If the Xen virtual keyboard driver is advertising capabilities it
> >     doesn't have, stop it doing that.
> > b. Change the implementation of modalias attributes to allow longer
> >     values.
> >
> > It's not clear to me whether the Xen driver is advertising correctly or
> > not. If it is, then the solution should be b, but that may be too
> > disruptive a change to the kernel. So a reasonable workaround might
> > be:
> >
> > c. Change the input subsystem to limit the length of the
> >     capabilities part of the modalias.
> >
> >
> > Ben.
> >
>

> So workaround c would not involve disruptions to the kernel or
> systemd? Workaround c seems too disruptive for stable to me,
> but maybe could go into unstable and eventually into testing.

I don't think it would be very disruptive. It might require a kernel
ABI bump, but we do those regularly during a stable release. And this
bug is severe enough that I think a fix would be suitable for Debian
stable.

> A problem with the approach of fixing this bug in the Xen
> keyboard driver is that the fix must be implemented in the underlying
> Dom0 system, which could be almost anything - another Linux distro
> or Debian stable or oldstable. Any fix upstream would probably get into
> a bullseye Dom0, but not oldstable Dom0, but perhaps it could be
> provided as a backport for anyone who is still on oldstable for their
> Xen Dom0.
[...]

I agree that we need to fix this for domU independently of any protocol
change to allow discovery of which keys the underlying input device
has. So we can't solve this with approach a.

[Ben Hutchings]

On Wed, 2021-08-25 at 12:45 -0400, Chuck Zmudzinski wrote:
[...]
>
> I will try it in my bullseye Xen HVM DomU.
>
> I am not sure how to rebuild the installation media with a patched
> systemd, but I can patch my installed Xen HVM DomU system
> with a patched systemd with the increased buffer size and see if the
> Coldplug failure early in the boot process goes away. If so, then it
> is likely this patch to systemd would also fix the installation media.
[...]

Sorry for not being clear - this is a patch for the kernel.
Instructions for rebuilding the kernel package are at
<https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official>.

I agree that you should check whether this fixes the coldplug error
before we try rebuilding the installer.

[Phillip Susi]

Chuck Zmudzinski <brch...@netscape.net> writes:
> If it doesn't work, I am also willing to try approach a by patching
> the Linux kernel xen-kbdfront driver by removing the for loops that
> advertise those 654 keys. I tend to agree with Philip that this is
> totally unnecessary, but I suppose I could be wrong about that.
> I read the discussion Philip had with the Xen developers and they
> seemed to want to keep the Xen keyboard driver as it is.

That was the first thing I tried and the libinput maintainer pointed out
that if you don't advertise the keys, you can't use the keys. In other
words, somebody presses that key on their keyboard and the domU won't
recognize it.

[Paul Gevers]

Hi Chuck,

On 10-02-2022 01:34, Chuck Zmudzinski wrote:
> The problem as I see it is that the debian
> installer team is already aware of the problem and has been aware of it
> for over six months because #983357 is marked as affecting d-i. So I do
> not understand why #983357 is not included on the d-i bullseye errata page.

Please assume good faith. I would go for the most obvious possibility,
that while being aware of this issue, they just didn't think at the same
time of the installation guide.

> Do I need to file another bug in
> addition to #983357 to get this problem listed on the bullseye (and also
> RC versions) d-i errata pages?

This is the most establish process, so yes, I suggest you just go and
follow that route, even without a reply here. Than it's documented in
the right place [1].

Paul

[1] unless I'm much mistaken, that would be against the
installation-guide package, but it can be reassigned if I'm wrong.

[Håkon Alstadheim]

On Tue, 24 Aug 2021 15:27:19 -0400 Phillip Susi <ph...@thesusis.net> wrote:

> This seems to be the heart of the problem: libinput was designed
> assuming that all keyboards can and must report what keys are actually
> present, and then libinput tries to cram that information into the
> modalias rather than some other sysfs attribute as it should ( or not at
> all... I still don't see how this information is actually supposed to be
> useful to userspace ).
>

Firefox in guest will crash trying to handle the “joystick“ with this enormous number of keys, that does not actually respond as a joystick.

[zithro]

Hi all,

in an effort to solve this bug the quickest way possible, I want to
share some pointers (and some time !).

First, it can affect not only "upstream" Debian but also Debian-based
distros. I personaly had the problem trying to use Kali as a HVM domU.
Although Chuck's fix worked on Kali, I dunno which distros may fail without.
But from memory, looking for solutions to the bug, I only found posts
about Debian.

From what I read, understood, and guessed :
- it appears to be a kernel+udev+Xen bug,
- it has nothing to do with Debian itself,
- all distros using Xen are also using upstream Xen virtual KB

So I'm wondering :
- how can this bug only affect Debian and Debian-based distros but not
others like Suse, fedora, etc ?
- If I'm right, how are other distros handling this bug ?
- How is the "Debian cloud installer" handling this bug ?
- If the problem can't be solved "at its roots", can't we just apply the
(harmless) workaround in the installer ?

We should really speed up the fix before the Full Freeze.
I can help for testing and reporting stuff, but I'm a noob in packaging.


Thanks all, have a nice day,

zithro / Cyril REBERT


PS: off-topic remarks.
I'm a regular Debian user since a dozen years, and Debian+Xen since 5
years, as dom0s and domUs. I recently joined the Debian Xen team.
I want to say this is one of the most critical and obscure bug -I-
encountered on a Debian stable installer.

It's obvious but I'll state it nonetheless : it's really not a good
"publicity" for Debian. Note I'm pointing no finger at anyone as I
really don't care for it, I only care for a resolution.
We don't really know how much users turned their backs on Debian because
of this bug : "Deb does not work, but distro X works, let's just use the
one that works OOTB".
But maybe this bug is only affecting a handful of users ?