Bug#1006753: dkms modules not rebuilt on kernel upgrades with unattended upgrades

Felix Dörre

unread,

Mar 4, 2022, 5:50:04 AM3/4/22

to

Package: dkms
Version: 2.8.7-2
Severity: important

Dear Maintainer,

When upgrading the kernel (with unattended upgrades) on a system with a dkms-built module, dkms fails to build the new module. This is the excerpt from "/var/log/apt/term.log":

Selecting previously unselected package linux-image-5.10.0-0.bpo.9-amd64.
[...]
Preparing to unpack .../linux-image-5.10.0-0.bpo.9-amd64_5.10.70-1~bpo10+1_amd64.deb ...(Reading database ... 149449 files and directories currently installed.)
Unpacking linux-image-5.10.0-0.bpo.9-amd64 (5.10.70-1~bpo10+1) ...~bpo10+1_amd64.deb ...
Preparing to unpack .../linux-image-amd64_5.10.70-1~bpo10+1_amd64.deb ...
Unpacking linux-image-amd64 (5.10.70-1~bpo10+1) over (5.10.46-4~bpo10+1) ...
Setting up linux-image-5.10.0-0.bpo.9-amd64 (5.10.70-1~bpo10+1) ...10+1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.10.0-0.bpo.8-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.10.0-0.bpo.8-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.10.0-0.bpo.9-amd64po.8-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.10.0-0.bpo.9-amd64
/etc/kernel/postinst.d/dkms:ink to boot/initrd.img-5.10.0-0.bpo.9-amd64
Error! Your kernel headers for kernel 5.10.0-0.bpo.9-amd64 cannot be found.
Please install the linux-headers-5.10.0-0.bpo.9-amd64 package,not be found.
or use the --kernelsourcedir option to tell DKMS where it's located
/etc/kernel/postinst.d/initramfs-tools:tell DKMS where it's located
update-initramfs: Generating /boot/initrd.img-5.10.0-0.bpo.9-amd64
/etc/kernel/postinst.d/zz-update-grub:trd.img-5.10.0-0.bpo.9-amd64
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.10.0-0.bpo.9-amd64
Found initrd image: /boot/initrd.img-5.10.0-0.bpo.9-amd64
Found linux image: /boot/vmlinuz-5.10.0-0.bpo.8-amd64md64
Found initrd image: /boot/initrd.img-5.10.0-0.bpo.8-amd64
Found linux image: /boot/vmlinuz-5.10.0-0.bpo.7-amd64md64
Found initrd image: /boot/initrd.img-5.10.0-0.bpo.7-amd64
Found linux image: /boot/vmlinuz-5.10.0-0.bpo.5-amd64md64
Found initrd image: /boot/initrd.img-5.10.0-0.bpo.5-amd64
Adding boot menu entry for EFI firmware configurationmd64
doneng boot menu entry for EFI firmware configuration
Setting up linux-image-amd64 (5.10.70-1~bpo10+1) ...
Log ended: 2021-11-01 06:26:47.10.70-1~bpo10+1) ...
Log ended: 2021-11-01 06:26:47

Log started: 2021-11-01 06:26:50
Selecting previously unselected package linux-headers-5.10.0-0.bpo.9-common.
[...]
Preparing to unpack .../linux-headers-5.10.0-0.bpo.9-common_5.10.70-1~bpo10+1_all.deb ...
Unpacking linux-headers-5.10.0-0.bpo.9-common (5.10.70-1~bpo10+1) ...
Selecting previously unselected package linux-headers-5.10.0-0.bpo.9-amd64.
Preparing to unpack .../linux-headers-5.10.0-0.bpo.9-amd64_5.10.70-1~bpo10+1_amd64.deb ...
Unpacking linux-headers-5.10.0-0.bpo.9-amd64 (5.10.70-1~bpo10+1) ...
Preparing to unpack .../linux-headers-amd64_5.10.70-1~bpo10+1_amd64.deb ...
Unpacking linux-headers-amd64 (5.10.70-1~bpo10+1) over (5.10.46-4~bpo10+1) ...
Setting up linux-headers-5.10.0-0.bpo.9-common (5.10.70-1~bpo10+1) ...0+1) ...
Setting up linux-headers-5.10.0-0.bpo.9-amd64 (5.10.70-1~bpo10+1) ....
Setting up linux-headers-amd64 (5.10.70-1~bpo10+1) ...-1~bpo10+1) ...
Log ended: 2021-11-01 06:27:32(5.10.70-1~bpo10+1) ...
Log ended: 2021-11-01 06:27:32

This appareantly happens because unatteneded upgrade unnecessarily splits the upgrades of the kernel image and linux-headers
into two separate apt calls, as there is no dependency between them (in either way). With a bit of bad luck the installation
of the headers happens afterwards and the trigger from dkms fails to rebuild the module.

I observed this with several versions of dkms.

I've set the severity to "important", because the failed dkms upgrade does not seem to fail the upgrade which together
with unattended upgrades sneakily "removes" a module from the most recent kernel which will lead to the next boot failing
if correct booting requires that module.

Trent W. Buck

unread,

May 4, 2022, 9:00:05 PM5/4/22

to

I can reproduce this issue.
It has bitten me 3 or 4 times.
I think happens every time the ABI bumps (5.10-n → 5.10-n+1).

For me, the timeline is this:

1. unattended-upgrades installs new kernel
2. kernel postinst builds new initrd
3. unattended-upgrades installs new headers
4. kernel postinst new zfs.ko

Because #2 happens before #4, I get an initrd that says

Failed to load ZFS modules.
Manually load the modules and exit.

⋮

(initramfs)

Attached are the detailed logs.

Note that unattended-upgrades runs once, but dpkg appears to run twice.
This might explain why triggers are run in the wrong order?

Please note that

• linux-headers-amd64 is installed.
This is not simply "unattended-upgrades doesn't upgrade the header package"

• No backport kernels are involved.
This is not simply "unattended-upgrades is a bit weird for bpo kernels"

dpkg-query-W.txt

unattended-upgrades.log

unattended-upgrades-dpkg.log

Trent W. Buck

unread,

May 4, 2022, 9:30:03 PM5/4/22

to

Trent W. Buck wrote:
> I can reproduce this issue.
> It has bitten me 3 or 4 times.
> I think happens every time the ABI bumps (5.10-n → 5.10-n+1).
>
> For me, the timeline is this:
>
> 1. unattended-upgrades installs new kernel
> 2. kernel postinst builds new initrd
> 3. unattended-upgrades installs new headers
> 4. kernel postinst new zfs.ko

PS: from unattended-upgrades-dpkg.log, I'm pretty sure the problem is "above" dpkg, but
I'm not clear if it's in unattended-upgrades or in apt.

If it's happening in the apt layer,
I guess that is because linux-image-N-amd64 does not Depends on linux-headers-N-amd64.
So apt thinks it can do 2 separate dpkg runs, and (sometimes?) in the wrong order?

If that's what happens, it's unreasonable to add stronger dependencies, because
then EVERYONE will be forced to install headers.

Is it reasonable for unattended-upgrades to have a special-case safety net for this?
Something like

If unattended-upgradse is upgrading a kernel AND a headers,
then ensure headers installs first.

Felix Dörre

unread,

May 7, 2022, 9:40:03 AM5/7/22

to

I think your issue could be related but the symptoms seem to be slightly
different:

When the problem occurred to me, the zfs module was not even built, so
that's even one step before the initramdisk should be built.
Consequently not a simple "update-initkamfs" was enough.

Regarding which package is at fault, my best guess still lies with dkms,
as I understand it, a dkms build requires both, linux image and linux
headers installed and you can install both of them individually, so on
upgrade starting with -image, I see:

- linux image installed
- dkms build fails
- linux headers installed
- dkms gets notified through /etc/kernel/header_postinst.d/dkms
- the dkms autoinstall script *somehow* (still don't get how though)
misses the failed install and does nothing in my case

additionally, that an update-initramfs should be triggered is another
problem. So I guess the desired resolution would be to understand the
problem with autoinstall and maybe somehow trigger the update-initramfs
from there.

changing the behavior of unattended upgrades seem an easier, but hacky
way to resolve this and additionally the problem with update-initramfs
in one go.

--
Kind regards,
Felix Dörre

Jan Stibila

unread,

Jul 6, 2022, 12:10:03 PM7/6/22

to

Hi,

I have same problem, that headers are installed only after the kernel. I
believe, that problem lies with unattended-upgrades and not with apt, as
with apt, 100% times it will install it in correct order, yet with
unattended-upgrades it will not.

I think that apt is solving installation order internally here:
https://github.com/Debian/apt/blob/766b24b7f7484751950c76bc66d3d6cdeaf949a5/apt-pkg/orderlist.cc

If unattended-upgrades is upgrading packages one at the time (it seems
like it) then lower layers can't properly order packages. In that case
unattended-upgrades should be responsible for proper ordering.