Bug#1040401: dkms: forgets to rebuild all kernel modules when autoinstall_all_kernels=1

[Vincent Lefevre]

Package: dkms
Version: 3.0.11-3
Severity: normal

I have autoinstall_all_kernels="1" in /etc/dkms/framework.conf
so that all kernel modules are rebuilt when some dependency
changes.

But when I upgraded linux-kbuild-6.3 from 6.3.7-1 to 6.3.11-1 and
installed linux-image-6.3.0-2-amd64 at the same time, a rebuild
of the NVIDIA kernel module for linux-image-6.3.0-1-amd64 (still
installed together with linux-headers-6.3.0-1-*) was not attempted.

A rebuild may be important since an upgrade of the kbuild package may
fix bugs in the build (it may also introduce bugs, but one expects
fixes more often).

[...]
Log started: 2023-07-04 13:23:14
(Reading database [...] 661586 files and directories currently installed.)
Preparing to unpack .../00-linux-compiler-gcc-12-x86_6.3.11-1_amd64.deb ...
Unpacking linux-compiler-gcc-12-x86 (6.3.11-1) over (6.3.7-1) ...
Preparing to unpack .../01-linux-doc_6.3.11-1_all.deb ...
Unpacking linux-doc (6.3.11-1) over (6.3.7-1) ...
Preparing to unpack .../02-linux-doc-6.3_6.3.11-1_all.deb ...
Unpacking linux-doc-6.3 (6.3.11-1) over (6.3.7-1) ...
Selecting previously unselected package linux-headers-6.3.0-2-common.
Preparing to unpack .../03-linux-headers-6.3.0-2-common_6.3.11-1_all.deb ...
Unpacking linux-headers-6.3.0-2-common (6.3.11-1) ...
Preparing to unpack .../04-linux-kbuild-6.3_6.3.11-1_amd64.deb ...
Unpacking linux-kbuild-6.3 (6.3.11-1) over (6.3.7-1) ...
Selecting previously unselected package linux-headers-6.3.0-2-amd64.
Preparing to unpack .../05-linux-headers-6.3.0-2-amd64_6.3.11-1_amd64.deb ...
Unpacking linux-headers-6.3.0-2-amd64 (6.3.11-1) ...
Preparing to unpack .../06-linux-headers-amd64_6.3.11-1_amd64.deb ...
Unpacking linux-headers-amd64 (6.3.11-1) over (6.3.7-1) ...
Selecting previously unselected package linux-image-6.3.0-2-amd64.
Preparing to unpack .../07-linux-image-6.3.0-2-amd64_6.3.11-1_amd64.deb ...
Unpacking linux-image-6.3.0-2-amd64 (6.3.11-1) ...
Preparing to unpack .../08-linux-image-amd64_6.3.11-1_amd64.deb ...
Unpacking linux-image-amd64 (6.3.11-1) over (6.3.7-1) ...
Preparing to unpack .../09-linux-libc-dev_6.3.11-1_i386.deb ...
De-configuring linux-libc-dev:amd64 (6.3.7-1), to allow configuration of linux-libc-dev:i386 (6.3.11-1) ...
Unpacking linux-libc-dev:i386 (6.3.11-1) over (6.3.7-1) ...
Preparing to unpack .../10-linux-libc-dev_6.3.11-1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (6.3.11-1) over (6.3.7-1) ...
Setting up linux-image-6.3.0-2-amd64 (6.3.11-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-6.3.0-1-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-6.3.0-1-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-6.3.0-2-amd64
I: /initrd.img is now a symlink to boot/initrd.img-6.3.0-2-amd64
/etc/kernel/postinst.d/dkms:
dkms: running auto installation service for kernel 6.3.0-2-amd64.
[...]
dkms autoinstall on 6.3.0-2-amd64/x86_64 succeeded for nvidia-legacy-390xx
dkms: autoinstall for kernel: 6.3.0-2-amd64.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.3.0-2-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-6.3.0-2-amd64
Found initrd image: /boot/initrd.img-6.3.0-2-amd64
Found linux image: /boot/vmlinuz-6.3.0-1-amd64
Found initrd image: /boot/initrd.img-6.3.0-1-amd64
Found linux image: /boot/vmlinuz-6.1.0-9-amd64
Found initrd image: /boot/initrd.img-6.1.0-9-amd64
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
Setting up linux-headers-6.3.0-2-common (6.3.11-1) ...
Setting up linux-libc-dev:amd64 (6.3.11-1) ...
Setting up linux-libc-dev:i386 (6.3.11-1) ...
Setting up linux-image-amd64 (6.3.11-1) ...
Setting up linux-compiler-gcc-12-x86 (6.3.11-1) ...
Setting up linux-doc-6.3 (6.3.11-1) ...
Setting up linux-kbuild-6.3 (6.3.11-1) ...
Setting up linux-headers-6.3.0-2-amd64 (6.3.11-1) ...
/etc/kernel/header_postinst.d/dkms:
dkms: running auto installation service for kernel 6.3.0-2-amd64.
dkms: autoinstall for kernel: 6.3.0-2-amd64.
Setting up linux-doc (6.3.11-1) ...
Setting up linux-headers-amd64 (6.3.11-1) ...
Log ended: 2023-07-04 13:25:07

cventin:~> dpkg -l | fgrep ' 6.3'
ii linux-compiler-gcc-12-x86 6.3.11-1 amd64 Compiler for Linux on x86 (meta-package)
ii linux-doc 6.3.11-1 all Linux kernel specific documentation (meta-package)
ii linux-doc-6.3 6.3.11-1 all Linux kernel specific documentation for version 6.3
ii linux-headers-6.3.0-1-amd64 6.3.7-1 amd64 Header files for Linux 6.3.0-1-amd64
ii linux-headers-6.3.0-1-common 6.3.7-1 all Common header files for Linux 6.3.0-1
ii linux-headers-6.3.0-2-amd64 6.3.11-1 amd64 Header files for Linux 6.3.0-2-amd64
ii linux-headers-6.3.0-2-common 6.3.11-1 all Common header files for Linux 6.3.0-2
ii linux-headers-amd64 6.3.11-1 amd64 Header files for Linux amd64 configuration (meta-package)
ii linux-image-6.3.0-1-amd64 6.3.7-1 amd64 Linux 6.3 for 64-bit PCs (signed)
ii linux-image-6.3.0-2-amd64 6.3.11-1 amd64 Linux 6.3 for 64-bit PCs (signed)
ii linux-image-amd64 6.3.11-1 amd64 Linux for 64-bit PCs (meta-package)
ii linux-kbuild-6.3 6.3.11-1 amd64 Kbuild infrastructure for Linux 6.3
ii linux-libc-dev:amd64 6.3.11-1 amd64 Linux support headers for userspace development
ii linux-libc-dev:i386 6.3.11-1 i386 Linux support headers for userspace development

-- System Information:
Debian Release: trixie/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'stable-security'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
merged-usr: no
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.3.0-2-amd64 (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=POSIX, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages dkms depends on:
ii build-essential 12.10
ii clang-10 [c-compiler] 1:10.0.1-8+b1
ii clang-11 [c-compiler] 1:11.1.0-6+b2
ii clang-12 [c-compiler] 1:12.0.1-21
ii clang-13 [c-compiler] 1:13.0.1-11+b2
ii clang-14 [c-compiler] 1:14.0.6-12
ii clang-15 [c-compiler] 1:15.0.7-6
ii clang-3.6 [c-compiler] 1:3.6.2-3
ii clang-3.7 [c-compiler] 1:3.7.1-3+b2
ii clang-6.0 [c-compiler] 1:6.0.1-14.1
ii clang-7 [c-compiler] 1:7.0.1-12
ii clang-8 [c-compiler] 1:8.0.1-10+b1
ii clang-9 [c-compiler] 1:9.0.1-20+b1
ii dpkg-dev 1.21.22
ii gcc [c-compiler] 4:12.3.0-1
ii gcc-10 [c-compiler] 10.4.0-9
ii gcc-11 [c-compiler] 11.4.0-1
ii gcc-12 [c-compiler] 12.3.0-5
ii gcc-13 [c-compiler] 13.1.0-7
ii gcc-4.8 [c-compiler] 4.8.5-4
ii gcc-4.9 [c-compiler] 4.9.4-2
ii gcc-5 [c-compiler] 5.5.0-12
ii gcc-6 [c-compiler] 6.5.0-2
ii gcc-8 [c-compiler] 8.4.0-7
ii gcc-9 [c-compiler] 9.5.0-3+b1
ii kmod 30+20230519-1
ii lsb-release 12.0-1
ii make 4.3-4.1
ii patch 2.7.6-7
ii tcc [c-compiler] 0.9.27+git20200814.62c30a4a-1

Versions of packages dkms recommends:
ii fakeroot 1.31-1.2
ii linux-headers-amd64 [linux-headers-generic] 6.3.11-1
ii sudo 1.9.13p3-3

Versions of packages dkms suggests:
ii e2fsprogs 1.47.0-2
ii menu 2.1.49

-- Configuration Files:
/etc/dkms/framework.conf changed:
verbose="1"
autoinstall_all_kernels="1"


-- no debconf information

--
Vincent Lefèvre <vin...@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[Andreas Beckmann]

On 05/07/2023 16.17, Vincent Lefevre wrote:
> I have autoinstall_all_kernels="1" in /etc/dkms/framework.conf
> so that all kernel modules are rebuilt when some dependency
> changes.

Does that at least cause a rebuild if the linux-headers-* package gets
updated? IIRC as long as KVERS does not change, dkms does not rebuild
modules on header change.

More general: if linux-headers-* gets updated but the ABI does not
change (and the ABI is not 0), does dkms have to rebuild the modules? Why?

If the ABI changes, the package gets renamed and the new installation
implies a rebuild.

If the ABI is 0, this should be treated a non-stable ABI and an update
of the header package should definitively trigger a module rebuild.

from dkms(8):
> .B $autoinstall_all_kernels
> Used by the common postinst for DKMS modules. It controls if the build
> should be done for all installed kernels or only for the current and
> latest installed kernel. It has no command
> line equivalent.

So that flag is not doing what you expected. And I think on Debian it is
ignored and we always build for all kernels. But only on new header
installation or on dkms module update. Not on header update.


Andreas

[Vincent Lefevre]

On 2023-07-05 17:21:04 +0200, Andreas Beckmann wrote:
> On 05/07/2023 16.17, Vincent Lefevre wrote:
> > I have autoinstall_all_kernels="1" in /etc/dkms/framework.conf
> > so that all kernel modules are rebuilt when some dependency
> > changes.
>
> Does that at least cause a rebuild if the linux-headers-* package gets
> updated? IIRC as long as KVERS does not change, dkms does not rebuild
> modules on header change.
>
> More general: if linux-headers-* gets updated but the ABI does not change
> (and the ABI is not 0), does dkms have to rebuild the modules? Why?

I'm quoting what you said in bug 1040178:

| This needs to be fixed before linux 6.3.0-2-* can migrate to testing,
| otherwise it will break dkms module building for everyone still having
| linux-headers-6.3.0-1-* installed (which is probably for the currently
| running kernel).

Though I track unstable instead of testing, the issue is basically
the same (I'm concerned now instead of later, if nothing is changed).

What you said is that the upgrade to 6.3.0-2-* is expected to break
dkms module building for everyone still having linux-headers-6.3.0-1-*
installed (which is exactly my case), and 6.3.0-1 was indeed the
currently running kernel in my case.

This means that you expect the upgrade to 6.3.0-2-* to rebuild the
kernel modules for 6.3.0-1 (otherwise, i.e. if the 6.3.0-1 modules
are not rebuilt, there are no bugs at all). But this is not what
happened.

And it doesn't seem that autoinstall_all_kernels actually matters.

> from dkms(8):
> > .B $autoinstall_all_kernels
> > Used by the common postinst for DKMS modules. It controls if the build
> > should be done for all installed kernels or only for the current and
> > latest installed kernel. It has no command
> > line equivalent.
>
> So that flag is not doing what you expected. And I think on Debian it is
> ignored and we always build for all kernels. But only on new header
> installation or on dkms module update. Not on header update.

Yes, it is probably a different matter. I now remember that
autoinstall_all_kernels is not sufficient as the ABI may change
without being announced. See

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=856355#54

This is crap (I had written a workaround in /etc/kernel/postinst.d,
and I've just noticed that it got broken after an upgrade 3 years
ago), but this is unrelated to the above issue.

[Andreas Beckmann]

On 05/07/2023 21.52, Vincent Lefevre wrote:
> What you said is that the upgrade to 6.3.0-2-* is expected to break
> dkms module building for everyone still having linux-headers-6.3.0-1-*
> installed (which is exactly my case), and 6.3.0-1 was indeed the
> currently running kernel in my case.
>
> This means that you expect the upgrade to 6.3.0-2-* to rebuild the
> kernel modules for 6.3.0-1 (otherwise, i.e. if the 6.3.0-1 modules
> are not rebuilt, there are no bugs at all). But this is not what
> happened.

Right. The installation of 6.3.0-2-* won't trigger a rebuild of
6.3.0-1-*. But the upgrade (or new installation) of any *-dkms package will.

In sid there was a short window (2 days) between the updates of
src:linux and src:linux-signed-amd64 etc. where linux-headers-amd64 was
pulling in linux-headers-6.3.0-1-* 6.3.7-1 and linux-kbuild-6.3 6.3.11-1
which made this bug very noticable.

Andreas

[Vincent Lefevre]

On 2023-07-06 10:59:35 +0200, Andreas Beckmann wrote:
> On 05/07/2023 21.52, Vincent Lefevre wrote:
> > What you said is that the upgrade to 6.3.0-2-* is expected to break
> > dkms module building for everyone still having linux-headers-6.3.0-1-*
> > installed (which is exactly my case), and 6.3.0-1 was indeed the
> > currently running kernel in my case.
> >
> > This means that you expect the upgrade to 6.3.0-2-* to rebuild the
> > kernel modules for 6.3.0-1 (otherwise, i.e. if the 6.3.0-1 modules
> > are not rebuilt, there are no bugs at all). But this is not what
> > happened.
>
> Right. The installation of 6.3.0-2-* won't trigger a rebuild of 6.3.0-1-*.
> But the upgrade (or new installation) of any *-dkms package will.

In this case, the user can safely ignore the error, as it doesn't
make the upgrade fail. The old kernel should still work: it will
just lack new features provided by the new kernel modules. This
should be no worse than removing the *working* modules for the old
kernel, as you suggested in bug 1040178: "[...] that the headers
for the older kernel are going to be removed and as a consequence
all dkms modules will be removed for this old kernel, which may make
the system unbootable into this old kernel."