Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#1021918: debian-installer: Kernel module blacklisting inconsistent

178 views
Skip to first unread message

Olaf Meeuwissen

unread,
Oct 17, 2022, 7:20:03 AM10/17/22
to

Package: debian-installer
Version: 20220917
Severity: normal
Tags: d-i

Dear Maintainer,

I recently tried this version with hardware that triggers loading of the
mt7921e kernel module. Loading the module fails due to a firmware file
load error but the installer starts okay. However, the installer later
crashes when probing for network hardware (when it tries to rmmod the
kernel module).

Since the hardware has 2 additional wired NICs that work fine (and I
wanted to do an air-gap install anyway!), I decided to blacklist the
module.

The first issue I ran into was that the documented[1] way to blacklist
kernel modules is no longer correct

[1]: https://www.debian.org/releases/testing/amd64/ch05s03.en.html#module-blacklist

Instead of

mt7921e.blacklist=yes

I had to use

modprobe.blacklist=mt7921e

to prevent the kernel from loading the module (between the boot menu and
starting the installer proper). With the latter appended to the `linux`
line of the boot menu entry, the installation appeared to have completed
without problems.

However, upon booting I saw a pile of ATA bus and I/O errors that made
me suspicious. The disk is brand new and a smartmontools extended test
reports no errors.

I found a /etc/modprobe.d/blacklist.local.conf file with

blacklist modprobe

Removing that file made the ATA bus and I/O errors go away.

For completeness' sake, the /etc/default/grub file included

GRUB_CMDLINE_LINUX="modprobe.blacklist=mt7921e"

so the offending module is not loaded upon reboot after installation.

Seeing that the kernel boot argument is added correctly to the GRUB
configuration, there is no need to create a file in /etc/modprobe.d/.
In addition, the installation manual needs to be updated to use the
correct syntax.

Hope this helps,

-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 5.18.0-3-amd64 (SMP w/16 CPU threads; PREEMPT)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: runit (via /run/runit.stopit)
LSM: AppArmor: enabled
--
Olaf Meeuwissen

Pascal Hambourg

unread,
Oct 18, 2022, 8:10:03 PM10/18/22
to
On 17/10/2022 at 13:13, Olaf Meeuwissen wrote:
>
> I recently tried this version with hardware that triggers loading of the
> mt7921e kernel module. Loading the module fails due to a firmware file
> load error but the installer starts okay. However, the installer later
> crashes when probing for network hardware (when it tries to rmmod the
> kernel module).

How does the installer crash exactly ? Kernel panic ? Freeze ? Error ?

> The first issue I ran into was that the documented[1] way to blacklist
> kernel modules is no longer correct
>
> [1]: https://www.debian.org/releases/testing/amd64/ch05s03.en.html#module-blacklist
>
> Instead of
>
> mt7921e.blacklist=yes
>
> I had to use
>
> modprobe.blacklist=mt7921e

/lib/debian-installer-startup.d/S02module-params has the following comment:

# Before udev is started, parse kernel command word for module params of
# the form module.param=value and register them so they will be used when
# modules are loaded. Also check for modules to be blacklisted.

But udev is actually started earlier, so the first method does not work
with modules included in initrd.gz (e.g. storage drivers).

However it should work with network driver modules which are installed
much later.

> However, upon booting I saw a pile of ATA bus and I/O errors that made
> me suspicious. The disk is brand new and a smartmontools extended test
> reports no errors.
>
> I found a /etc/modprobe.d/blacklist.local.conf file with
>
> blacklist modprobe

This is a minor bug in /lib/debian-installer-startup.d/S02module-params
which can be easily fixed. However, it should not have any actual impact
as "modprobe" does not match any kernel module name or alias.

> For completeness' sake, the /etc/default/grub file included
>
> GRUB_CMDLINE_LINUX="modprobe.blacklist=mt7921e"

As expected.

> Seeing that the kernel boot argument is added correctly to the GRUB
> configuration, there is no need to create a file in /etc/modprobe.d/.
> In addition, the installation manual needs to be updated to use the
> correct syntax.

If I understand correctly, this is how things work:

The kernel runs /init.
/init runs /lib/debian-installer/start-udev which starts udevd.
udevd gets hotplug events and calls modprobe to load matching modules
included in initrd.gz.
Then /init exec's /bin/busybox init.

busybox init reads /etc/inittab and runs /sbin/debian-installer-startup.
debian-installer-startup runs
/lib/debian-installer-startup.d/S02module-params.
/lib/debian-installer-startup.d/S02module-params calls
/bin/register-module for each module parameter or blacklist in the
kernel command line.
register-module writes module blacklists in
/etc/modprobe.d/blacklist.local.conf and module parameters in
/etc/modprobe.d/local.conf.

Later, network driver modules are installed and loaded.

Later, /usr/bin/grub-installer calls /bin/user-params which retrieves
kernel parameters after "--" or "---" in the command line and writes
them to grub2/linux_cmdline in debconf and GRUB_CMDLINE_LINUX in
/etc/default/grub.

So indeed kernel module parameters and blacklists may be duplicated in
/etc/modprobe.d and GRUB_CMDLINE_LINUX. To avoid duplication they should
be inserted before "---" in the installer command line.

Pascal Hambourg

unread,
Nov 1, 2022, 5:10:04 AM11/1/22
to
On 22/10/2022 at 03:39, Olaf Meeuwissen wrote:
> Pascal Hambourg <pas...@plouf.fr.eu.org> writes:
>> On 17/10/2022 at 13:13, Olaf Meeuwissen wrote:
>>> I recently tried this version with hardware that triggers loading of the
>>> mt7921e kernel module. Loading the module fails due to a firmware file
>>> load error but the installer starts okay. However, the installer later
>>> crashes when probing for network hardware (when it tries to rmmod the
>>> kernel module).
>>
>> How does the installer crash exactly ? Kernel panic ? Freeze ? Error ?
>
> Freeze. Even after ten minutes the network hardware probe does not
> complete. FWIW, I have seen an error log as well but that may have
> been with Devuan's preview installer for daedalus.

I could not reproduce this after tricking the installer into unloading
and reloading the mt7921e module. The module unloads and reloads
cleanly. But I do not have any hardware matching this module.

> I've attached the installer's syslog. /dev/sdc is the installer ISO.
> The other disks, /dev/sda, /dev/sdb and /dev/nmve0n1 are the machine's
> internal disks. I just ran the installer after the machine was already
> installed with the workaround I mentioned in the original bug report.
>
> The error starts at
>
> Oct 19 23:06:13 check-missing-firmware: removing and loading kernel module mt7921e

> Oct 19 23:06:13 kernel: [ 40.024088] BUG: unable to handle page fault for address: 0000000000006500
> Oct 19 23:06:13 kernel: [ 40.024092] #PF: supervisor write access in kernel mode
> Oct 19 23:06:13 kernel: [ 40.024094] #PF: error_code(0x0002) - not-present page
(...)
> Oct 19 23:06:13 kernel: [ 40.024120] Call Trace:
> Oct 19 23:06:13 kernel: [ 40.024121] <TASK>
> Oct 19 23:06:13 kernel: [ 40.024124] __cancel_work_timer+0x3c/0x190
> Oct 19 23:06:13 kernel: [ 40.024128] ? __kernfs_remove.part.0+0x190/0x2b0
> Oct 19 23:06:13 kernel: [ 40.024131] mt7921_pci_remove+0x2c/0x110 [mt7921e]

It looks like a kernel bug when unloading this module. Can you trigger
the bug in an installed system ? If yes it means that it not specific to
the installer.

>>> The first issue I ran into was that the documented[1] way to blacklist
>>> kernel modules is no longer correct
>>> [1]:
>>> https://www.debian.org/releases/testing/amd64/ch05s03.en.html#module-blacklist
>>> Instead of
>>> mt7921e.blacklist=yes
>>> I had to use
>>> modprobe.blacklist=mt7921e
>>
>> /lib/debian-installer-startup.d/S02module-params has the following comment:
>>
>> # Before udev is started, parse kernel command word for module params of
>> # the form module.param=value and register them so they will be used when
>> # modules are loaded. Also check for modules to be blacklisted.
>>
>> But udev is actually started earlier, so the first method does not
>> work with modules included in initrd.gz (e.g. storage drivers).
>
> In that case, shouldn't that be mentioned in the installation manual?
> Actually, a single method that works for *all* modules, whether in the
> initrd.gz or installed later is much preferred.
>
>> However it should work with network driver modules which are installed
>> much later.
>
> You may want to double check how the kernel command parse results are
> used then.

I did, and <module>.blacklist works as expected with NIC modules
matching my hardware (iwlwifi and e1000e).

> Or maybe the mt7921e module is in the initrd.gz?
> Just checked, it is not.

Indeed, it is in the package nic-wireless-modules-<kernel-version>-di.

>>> However, upon booting I saw a pile of ATA bus and I/O errors that made
>>> me suspicious. The disk is brand new and a smartmontools extended test
>>> reports no errors.
>>> I found a /etc/modprobe.d/blacklist.local.conf file with
>>> blacklist modprobe
>>
>> This is a minor bug in
>> /lib/debian-installer-startup.d/S02module-params which can be easily
>> fixed. However, it should not have any actual impact as "modprobe"
>> does not match any kernel module name or alias.
>
> Strange, because removing it made those ATA bus and I/O errors go away,
> reproducibly at that.

Yes, really strange. I cannot explain nor reproduce it.

>>> Seeing that the kernel boot argument is added correctly to the GRUB
>>> configuration, there is no need to create a file in /etc/modprobe.d/.

It avoids cluttering the kernel command line with module parameters.

> The blacklist.local.conf file is created as documented but using the
> alternative syntax I had to use leads to the oxymoronic
>
> blacklist modprobe
>
> entry, trying to tell modprobe to blacklist itself :-)

As I wrote, "modprobe" is not a module name so this should be a no-op.

> You mentioned above that's a minor bug and easily fixed. If so, then
> please fix it.

I am not a Debian developer. The best I can do is submit a patch.

>> Later, network driver modules are installed and loaded.
>
> Seeing that the module is not in initrd.gz, this is where it would be
> loaded according to your understanding. Does this step happen *before*
> the installer screen appears?

No, it should happen at the "Detect network hardware" (or so) step.

> If no, and network driver modules are installed and loaded
> at the network hardware probe step of the installer then that does *not*
> correspond to what I have seen. That is to say, unless I blacklist the
> module with modprobe.blacklist=mt7921e, I see piles or firmware loading
> error fly by before the installer screen appears, asking me to select a
> language.

Which installation image did you use ?

Pascal Hambourg

unread,
Nov 1, 2022, 5:30:05 AM11/1/22
to
Control: reassign -1 rootskel
Control: tags -1 patch d-i

Here are two proposed patches for rootskel.

1) The first patch adds "modprobe" to the ignore list when parsing
module parameters in the kernel command line to skip modprobe.blacklist
parameters which are intended to modprobe, not to a kernel module.

2) Module parameters passed in the kernel command line after --- or --
are duplicated in both /etc/modprobe.d/local.conf by rootskel and
/etc/default/grub by grub-installer. The second patch skips parsing of
module parameters after --- or -- to avoid the useless duplicate.

I seek advice from d-i developpers about a third issue in rootskel.

The installer parses "<module>.blacklist=yes" kernel parameters and
creates "blacklist <module>" entries in
/etc/modprobe.d/blacklist.local.conf. However it happens after udev is
started so it is uneffective with modules included in initrd.gz.

Note: "modprobe.blacklist=<module>" kernel parameters are parsed by
modprobe so do not have this flaw.

1. Should this flaw be fixed ?
2. If yes, how ?
3. Should the <module>.blacklist=yes method be deprecated in favour of
modprobe.blacklist=<module> which is handled directly by modprobe and
does not require any special processing by rootskel ?

I tried to delay the start of udev until after blacklist.local.conf has
been created, but then /dev/tty* device files are created late and the
selected console is /dev/console instead of /dev/tty0|1. As a result,
the installer does not seem to use the framebuffer and does not offer
languages other than English.
0001-S02module-params-add-modprobe-to-ignore-list.patch
0002-S02module-params-skip-module-parameters-after-or.patch

Olaf Meeuwissen

unread,
Nov 7, 2022, 6:40:03 AM11/7/22
to
Sorry for the belated follow-up.

Pascal Hambourg <pas...@plouf.fr.eu.org> writes:

> On 22/10/2022 at 03:39, Olaf Meeuwissen wrote:
>> Pascal Hambourg <pas...@plouf.fr.eu.org> writes:
>>> On 17/10/2022 at 13:13, Olaf Meeuwissen wrote:
>>>> I recently tried this version with hardware that triggers loading of the
>>>> mt7921e kernel module. Loading the module fails due to a firmware file
>>>> load error but the installer starts okay. However, the installer later
>>>> crashes when probing for network hardware (when it tries to rmmod the
>>>> kernel module).
>>>
>>> How does the installer crash exactly ? Kernel panic ? Freeze ? Error ?
>> Freeze. Even after ten minutes the network hardware probe does not
>> complete. FWIW, I have seen an error log as well but that may have
>> been with Devuan's preview installer for daedalus.
>
> I could not reproduce this after tricking the installer into unloading
> and reloading the mt7921e module. The module unloads and reloads
> cleanly. But I do not have any hardware matching this module.

The freeze probably only occurs when an attempt to loadh the firmware
file is made. Unlikely that will happen if the hardware is not found.
Anyway, this issue is not with this particular kernel module but with
the installer's inconsistent module blacklisting behaviour.

FWIW, I've since installed firmware-misc-nonfree and removed all the
blacklisting bits for the module. WiFi works fine.

>> I've attached the installer's syslog. /dev/sdc is the installer ISO.
>> The other disks, /dev/sda, /dev/sdb and /dev/nmve0n1 are the machine's
>> internal disks. I just ran the installer after the machine was already
>> installed with the workaround I mentioned in the original bug report.
>> The error starts at
>> Oct 19 23:06:13 check-missing-firmware: removing and loading
>> kernel module mt7921e
>
>> Oct 19 23:06:13 kernel: [ 40.024088] BUG: unable to handle page fault for address: 0000000000006500
>> Oct 19 23:06:13 kernel: [ 40.024092] #PF: supervisor write access in kernel mode
>> Oct 19 23:06:13 kernel: [ 40.024094] #PF: error_code(0x0002) - not-present page
> (...)
>> Oct 19 23:06:13 kernel: [ 40.024120] Call Trace:
>> Oct 19 23:06:13 kernel: [ 40.024121] <TASK>
>> Oct 19 23:06:13 kernel: [ 40.024124] __cancel_work_timer+0x3c/0x190
>> Oct 19 23:06:13 kernel: [ 40.024128] ? __kernfs_remove.part.0+0x190/0x2b0
>> Oct 19 23:06:13 kernel: [ 40.024131] mt7921_pci_remove+0x2c/0x110 [mt7921e]
>
> It looks like a kernel bug when unloading this module. Can you trigger
> the bug in an installed system ? If yes it means that it not specific
> to the installer.

Machine's at the office, will see if I can test tomorrow or the day after.
I meant double check by reading the source code, not by trying with some
rather commonly used modules.

>> Or maybe the mt7921e module is in the initrd.gz?
>> Just checked, it is not.
>
> Indeed, it is in the package nic-wireless-modules-<kernel-version>-di.
>
>>>> However, upon booting I saw a pile of ATA bus and I/O errors that made
>>>> me suspicious. The disk is brand new and a smartmontools extended test
>>>> reports no errors.
>>>> I found a /etc/modprobe.d/blacklist.local.conf file with
>>>> blacklist modprobe
>>>
>>> This is a minor bug in
>>> /lib/debian-installer-startup.d/S02module-params which can be easily
>>> fixed. However, it should not have any actual impact as "modprobe"
>>> does not match any kernel module name or alias.
>> Strange, because removing it made those ATA bus and I/O errors go
>> away,
>> reproducibly at that.
>
> Yes, really strange. I cannot explain nor reproduce it.
>
>>>> Seeing that the kernel boot argument is added correctly to the GRUB
>>>> configuration, there is no need to create a file in /etc/modprobe.d/.
>
> It avoids cluttering the kernel command line with module parameters.
>
>> The blacklist.local.conf file is created as documented but using the
>> alternative syntax I had to use leads to the oxymoronic
>> blacklist modprobe
>> entry, trying to tell modprobe to blacklist itself :-)
>
> As I wrote, "modprobe" is not a module name so this should be a no-op.

It may be a no-op for modprobe but it's not a no-op for my brain ;-)
Actually, it's more like a WTF for my brain.

>> You mentioned above that's a minor bug and easily fixed. If so, then
>> please fix it.
>
> I am not a Debian developer. The best I can do is submit a patch.

By all means, please do. Thanks in advance.

>>> Later, network driver modules are installed and loaded.
>> Seeing that the module is not in initrd.gz, this is where it would
>> be
>> loaded according to your understanding. Does this step happen *before*
>> the installer screen appears?
>
> No, it should happen at the "Detect network hardware" (or so) step.
>
>> If no, and network driver modules are installed and loaded
>> at the network hardware probe step of the installer then that does *not*
>> correspond to what I have seen. That is to say, unless I blacklist the
>> module with modprobe.blacklist=mt7921e, I see piles or firmware loading
>> error fly by before the installer screen appears, asking me to select a
>> language.
>
> Which installation image did you use ?

The netinst one, according to the syslog I attached earlier. Looking at
that log, I don't see any proof of the firmware loading error fly by
before the installer starts. Weird. I'll see if I can reproduce it.
Maybe it is a default vs. advanced install difference. I do remember
using both while trying to get the blacklisting to work but I think I
captured that syslog using the default install.

Hope this helps,
--
Olaf Meeuwissen

Olaf Meeuwissen

unread,
Nov 8, 2022, 7:00:03 AM11/8/22
to
Hi again,
Checked this morning. With the firmware file loaded, rmmod succeeds
without problems. With the firmware file not loaded, trying to rmmod it
echoes "Killed" on the command-line and created a longish call trace in
/var/log/kern.log. Not sure if it was the same, though.
I cannot reproduce the firmware file load failures before the installer
starts. Tried graphical and text installs as well as the expert text
install. No idea why I believed they did. Some for the confusion.

Pascal Hambourg

unread,
Nov 8, 2022, 6:20:04 PM11/8/22
to
On 08/11/2022 at 12:35, Olaf Meeuwissen wrote:
>>
>> Pascal Hambourg <pas...@plouf.fr.eu.org> writes:
>>>
>>> I could not reproduce this after tricking the installer into unloading
>>> and reloading the mt7921e module. The module unloads and reloads
>>> cleanly. But I do not have any hardware matching this module.
>>
>> The freeze probably only occurs when an attempt to loadh the firmware
>> file is made.

I don't think so. I think it happens when unloading the module.

>> Anyway, this issue is not with this particular kernel module but with
>> the installer's inconsistent module blacklisting behaviour.

The freeze is related with this kernel module, not the installer.

>>> It looks like a kernel bug when unloading this module. Can you trigger
>>> the bug in an installed system ? If yes it means that it not specific
>>> to the installer.
>
> Checked this morning. With the firmware file loaded, rmmod succeeds
> without problems. With the firmware file not loaded, trying to rmmod it
> echoes "Killed" on the command-line and created a longish call trace in
> /var/log/kern.log. Not sure if it was the same, though.

As expected.

>>>> You may want to double check how the kernel command parse results
>>>> are used then.
>>>
>>> I did, and <module>.blacklist works as expected with NIC modules
>>> matching my hardware (iwlwifi and e1000e).
>>
>> I meant double check by reading the source code

I did.

>> not by trying with some rather commonly used modules.

The blacklist handling is not related to a specific module, so any
module matching present hardware will do.

>>>> You mentioned above that's a minor bug and easily fixed. If so, then
>>>> please fix it.
>>>
>>> I am not a Debian developer. The best I can do is submit a patch.
>>
>> By all means, please do. Thanks in advance.

Already done in another mail with attached patches sent to the bug.

>> The netinst one, according to the syslog I attached earlier. Looking at
>> that log, I don't see any proof of the firmware loading error fly by
>> before the installer starts.

Indeed, the log clearly shows that the module is loaded at the
"detecting networking hardware" step as expected.

> I cannot reproduce the firmware file load failures before the installer
> starts.
Consistent with the previous log.

Ian Jackson

unread,
Jan 7, 2023, 2:10:04 PM1/7/23
to
Control: retitle -1 Kernel module blacklisting doesn't work
Control: tags -1 confirmed

Hi. I just did an install of bullseye on an amd64 machine with a
troublesome wifi interface - suspected hardware fault[1]. We did
achieve a successful install, blacklisting the module for the bad
hardware, but we did encounter a version of this bug.

We edited the vmlinuz line in the installer image boot menu (we were
using the non-graphical installer) to add
modprobe.blacklist=rtw88_8723de

The installer was able to find the USB wifi dongle we'd added, didn't
touch the cursed hardware, and ran to completion. But, as described
in this bug report, it had created a file
/etc/modprobe.d/blacklist.local.conf
containing just
blacklist modprobe

I deleted this using the installer shell before rebooting into the
installed system. The installer had *not* included anything in
GRUB_CMDLINE_LINUX.

So the troublesome hardware *wasn't* blacklisted on our reboot.
Happily nothing tried to use it. I manually created a file
/etc/modprobe.d/rtw88.conf
containing
blacklist rtw88_8723de
blacklist rtw88_8723d
and that has resulted in a working setup.

Thanks,
Ian.

[1] The fault results in periodic wifi breakage and strange symtoms
with an older Debian release. With the bullseye installer, it causes
the installer environment to reliably hang (crashed with no response
even to capslock led).

--
Ian Jackson <ijac...@chiark.greenend.org.uk> These opinions are my own.

Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Pascal Hambourg

unread,
Jan 8, 2023, 5:00:04 AM1/8/23
to
Hello,

On 07/01/2023, Ian Jackson wrote :
>
> We edited the vmlinuz line in the installer image boot menu (we were
> using the non-graphical installer) to add
> modprobe.blacklist=rtw88_8723de
(...)
> The installer had *not* included anything in GRUB_CMDLINE_LINUX.

Did you add the parameter before or after "---" ? As explained
previously, only parameters located after "---" are added to the target
system /etc/default/grub GRUB_CMDLINE_LINUX by grub-installer.
Parameters located before "---" are intended for the installer only, not
the target system.

Note: modprobe.blacklist=<module> or "blacklist <module>" only prevent
modprobe from loading the module by alias (mostly when called by udev).
It does not prevent modprobe from loading the module by name or
dependency of another module nor loading the module by pathname with
insmod. Since v4.8, the kernel supports the parameter
module_blacklist=<module>,<module>... which prevents from loading a list
of modules in any way.
0 new messages