Bug#996461: nvidia-kernel-dkms: DKMS tree already contains: nvidia-current-470.74

[Drew Parsons]

On 2021-10-15 19:23, Andreas Beckmann wrote:
> On 14/10/2021 12.15, Drew Parsons wrote:
>> Package: nvidia-kernel-dkms
>> Version: 470.74-1
>> Severity: serious
>> Justification: fails to configure
>>
>> Upgrade to the new nvidia-driver 470.74-1 (nvidia-kernel-dkms) fails
>> at the configuration step with this error message:
>>
>> $ sudo aptitude
>> Performing actions...
>> Setting up nvidia-kernel-dkms (470.74-1) ...
>> Removing old nvidia-current-470.74 DKMS files...
>> Loading new nvidia-current-470.74 DKMS files...
>> Error! DKMS tree already contains: nvidia-current-470.74
>> You cannot add the same module/version combo more than once.
>
> What happened earlier?
> Did you somehow try to manually install the 470.74 driver from the .run
> file?

No, I didn't try any manual installations. Only upgrading nvidia-driver
and nvidia-kernel-dkms etc to the version in experimental at times.


> What happened at the first configuration attempt of the package?
> You should find the full transcript of upgrading to 470 somewhere in
> /var/log/apt/term.log*


/var/log/apt/term.log tells me it was upgrading from 470.63.01-1

Nothing unusual in the unpacking step (I guess we can ignore the message
"Deleting from: /lib/modules/5.14.0-1-amd64/
rmdir: failed to remove '': No such file or directory"),

...
Unpacking nvidia-vdpau-driver:i386 (470.74-1) over (470.63.01-1) ...
Preparing to unpack .../056-nvidia-kernel-dkms_470.74-1_amd64.deb ...
Module nvidia-current-470.63.01 for kernel 5.14.0-1-amd64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

nvidia-current.ko:
- Uninstallation
- Deleting from: /lib/modules/5.14.0-1-amd64/
rmdir: failed to remove '': No such file or directory
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module
version.


nvidia-current-modeset.ko:
- Uninstallation
- Deleting from: /lib/modules/5.14.0-1-amd64/
rmdir: failed to remove '': No such file or directory
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module
version.
...
Deleting module nvidia-current-470.63.01 completely from the DKMS tree.
Unpacking nvidia-kernel-dkms (470.74-1) over (470.63.01-1) ...
Preparing to unpack .../057-nvidia-kernel-support_470.74-1_amd64.deb ...
Unpacking nvidia-kernel-support (470.74-1) over (470.63.01-1) ...
...


Then setup step shows no errors, until
...
Setting up nvidia-vdpau-driver:amd64 (470.74-1) ...
Setting up nvidia-vdpau-driver:i386 (470.74-1) ...
Setting up libgl1-nvidia-glvnd-glx:amd64 (470.74-1) ...
Setting up nvidia-kernel-dkms (470.74-1) ...
Loading new nvidia-current-470.74 DKMS files...
Building for 5.14.0-2-amd64
Building initial module for 5.14.0-2-amd64
+ [ clean = modules ]
At main.c:160:
- SSL error:02001002:system library:fopen:No such file or directory:
../crypto/bio/bss_file.c:69
- SSL error:2006D080:BIO routines:BIO_new_file:no such file:
../crypto/bio/bss_file.c:76
sign-file: /root/dkms.key: No such file or directory
Error! Bad exit status 1 for sign tool.
Consult
/var/lib/dkms/nvidia-current/470.74/5.14.0-2-amd64/x86_64/log/make.log
for more information.
dpkg: error processing package nvidia-kernel-dkms (--configure):
installed nvidia-kernel-dkms package post-installation script
subprocess returned error exit status 11
Setting up libgles-nvidia1:amd64 (470.74-1) ...
Setting up libegl-nvidia0:amd64 (470.74-1) ...
Setting up nvidia-smi (470.74-1) ...
dpkg: dependency problems prevent configuration of nvidia-driver:
nvidia-driver depends on nvidia-kernel-dkms (= 470.74-1) |
nvidia-kernel-470.74; however:
Package nvidia-kernel-dkms is not configured yet.
Package nvidia-kernel-470.74 is not installed.
Package nvidia-kernel-dkms which provides nvidia-kernel-470.74 is not
configured yet.

dpkg: error processing package nvidia-driver (--configure):
dependency problems - leaving unconfigured
Setting up nvidia-driver-bin (470.74-1) ...


I guess that reference to bss_file.c must be a clue, if not the
reference to the missing /root/dkms.key

The 470.74/5.14.0-2 make.log in the error message shows no errors
itself, but the last line is
"Signing /var/lib/dkms/nvidia-current/470.74/build/nvidia.ko"

Drew

[Andreas Beckmann]

Control: severity -1 normal

On 15/10/2021 19.42, Drew Parsons wrote:
> I guess that reference to bss_file.c must be a clue, if not the
> reference to the missing /root/dkms.key >
> The 470.74/5.14.0-2 make.log in the error message shows no errors
> itself, but the last line is
> "Signing /var/lib/dkms/nvidia-current/470.74/build/nvidia.ko"

I think there are two bugs at work here and both are out of the scope of
the nvidia driver.

1. Signing the kernel module fails. That's probably your local
configuration that activates it, or do we have packages adding support
for that now? I've never looked into signing the kernel module.

Can you send me your signing bits (and instructions how to generate a
key) s.t. I can try to rebuild that setup in a chroot in order to
reproduce the bug?

2. dkms getting some hickup after the package configuration failed (due
to the signing failure) and not properly reinitializing on the next
attempt. (The relevant maintainer script bits should all come from dh_dkms.)

Andreas

[Drew Parsons]

On 2021-10-17 11:57, Andreas Beckmann wrote:
> Control: severity -1 normal
>
> On 15/10/2021 19.42, Drew Parsons wrote:
>> I guess that reference to bss_file.c must be a clue, if not the
>> reference to the missing /root/dkms.key >
>> The 470.74/5.14.0-2 make.log in the error message shows no errors
>> itself, but the last line is
>> "Signing /var/lib/dkms/nvidia-current/470.74/build/nvidia.ko"
>
> I think there are two bugs at work here and both are out of the scope
> of the nvidia driver.
>
> 1. Signing the kernel module fails. That's probably your local
> configuration that activates it, or do we have packages adding support
> for that now? I've never looked into signing the kernel module.

I've set my computer with MOK signing so that the modules can load after
booting with Secure Boot, following the instructions at
https://wiki.debian.org/SecureBoot#MOK_-_Machine_Owner_Key

I'm not certain if that's the same signing that's causing the problem
here. I use nvidia in bumblebee mode (primusrun). That was a little
flakey, so in practice I signed the modules manually after upgrades to
be sure they were signed.

> Can you send me your signing bits (and instructions how to generate a
> key) s.t. I can try to rebuild that setup in a chroot in order to
> reproduce the bug?

There are more signing instructions at
https://wiki.debian.org/SecureBoot#Using_your_key_to_sign_modules

I created /root/mok.priv and /root/mok.der

The script I've been using to sign the modules manually is
------------------------------------
cd /lib/modules
for mdir in */updates/dkms; do
cd $mdir
# get kernel version major.minor
kernel_version=$( echo $mdir | sed "s/\(.*\..*\)\..*/\1/" )
echo "signing modules for kernel ${kernel_version} in $PWD"
for k in *.ko; do
/usr/lib/linux-kbuild-${kernel_version}/scripts/sign-file sha256
/root/mok.priv /root/mok.der $k
echo " $k"
done
cd /lib/modules
done
-----------------------------------

That's still working, signing my last nvidia build. The modules on my
system that get signed are
bbswitch.ko
nvidia-current-drm.ko
nvidia-current-modeset.ko
nvidia-current-peermem.ko
nvidia-current-uvm.ko
nvidia-current.ko

> 2. dkms getting some hickup after the package configuration failed
> (due to the signing failure) and not properly reinitializing on the
> next attempt. (The relevant maintainer script bits should all come
> from dh_dkms.)

Agreed, it sounds like that's what's happened. There was a problem at
some point (perhaps in the experimental package), and it hasn't been
able to recover.

The problem is still current, by the way, with 470.86-1
Should I try a complete nvidia purge and reinstall, or would it be
useful to keep my computer in a broken state for further debugging?

Drew

[Drew Parsons]

On 2021-11-20 09:48, Drew Parsons wrote:
...

>>> I guess that reference to bss_file.c must be a clue, if not the
>>> reference to the missing /root/dkms.key >
>>> The 470.74/5.14.0-2 make.log in the error message shows no errors
>>> itself, but the last line is
>>> "Signing /var/lib/dkms/nvidia-current/470.74/build/nvidia.ko"

...

> I created /root/mok.priv and /root/mok.der
>
> The script I've been using to sign the modules manually is

...

> /usr/lib/linux-kbuild-${kernel_version}/scripts/sign-file sha256
> /root/mok.priv /root/mok.der $k

Thinking about it some more, looks like this is what's going on.

Following the instructions at https://wiki.debian.org/SecureBoot,
I've created mok.priv and mok.der, but there is no /root/dkms.key on my
system.

sign-file itself is working, and working with my MOK keys.

When dkms runs, it evidently triggers sign-file, invoking /root/dkms.key

I don't think I configured it to do that. I would have used mok.der (or
.priv) not dkms.key

So I see 2 questions here:
1) what is making the dkms scripts invoke sign-file ?
2) what is making them invoke sign-file with /root/dkms.key ?

i.e. where does dkms expect /root/dkms.key to come from?

[Drew Parsons]

On 2021-11-20 10:03, Drew Parsons wrote:
...

> sign-file itself is working, and working with my MOK keys.
>
> When dkms runs, it evidently triggers sign-file, invoking
> /root/dkms.key

...

> So I see 2 questions here:
> 1) what is making the dkms scripts invoke sign-file ?
> 2) what is making them invoke sign-file with /root/dkms.key ?

2) is easily answered, from the dkms package,

$ cat /etc/dkms/sign_helper.sh
#!/bin/sh
/lib/modules/"$1"/build/scripts/sign-file sha512 /root/dkms.key
/root/dkms.der "$2"

1) is also easy to answer after probing dkms.
/usr/share/doc/dkms/README.md.gz provides Secure Boot signing
instructions, including uncommenting
sign_tool="/etc/dkms/sign_helper.sh"
in /etc/dkms/framework.conf
It is uncommented on my system.

From one perspective looks like this is a documentation bug with
discrepancy between dkms documentation and
https://wiki.debian.org/SecureBoot. Or perhaps dkms expects to have its
own dkms.key, distinct from any MOK.der you might also create.

I see 3 workarounds

1) create a symlink from dkms.key to mok.priv and dkms.der to mok.der.

2) Create a separate dkms.key. Not so convenient to have 2 keys though
(though I could just stop using mok.der)

3) Comment out sign_tool in /etc/dkms/framework.conf, and sign manually.
A little inelegant for dkms, but otherwise easy.

1) or 3) would seem simplest to do, but strangely neither of them fixes
the final problem. Still gives the same error, "DKMS tree already
contains: nvidia-current-470.86. You cannot add the same module/version
combo more than once."
Not sure if it would work after nvidia purge/reinstall. Or if it needs a
reboot.

I'll try 2). dkms is supposed to work on its own after all.

[Drew Parsons]

On 2021-11-20 10:31, Drew Parsons wrote:
>
> I see 3 workarounds
>
> 1) create a symlink from dkms.key to mok.priv and dkms.der to mok.der.
>
> 2) Create a separate dkms.key. Not so convenient to have 2 keys
> though (though I could just stop using mok.der)
>
> 3) Comment out sign_tool in /etc/dkms/framework.conf, and sign
> manually. A little inelegant for dkms, but otherwise easy.
>
> 1) or 3) would seem simplest to do, but strangely neither of them
> fixes the final problem. Still gives the same error, "DKMS tree
> already contains: nvidia-current-470.86. You cannot add the same
> module/version combo more than once."
> Not sure if it would work after nvidia purge/reinstall. Or if it needs
> a reboot.

This error is pervasive.

I've reconfigured to follow the dkms instructions at
/usr/share/doc/dkms/README.md.gz, just using /root/dkms.key and
dkms.der. Generated the keys afresh and registered them. Manually
deleted any nvidia-current*ko files under /lib/modules/*. Rebooted.
Purged nvidia-kernel-dkms. Rebooted again.

But the error comes back. Even after deleting, purging, rebooting, when
I then install nvidia-kernel-dkms anew, I still get the same error
message:

Setting up nvidia-kernel-dkms (470.86-1) ...
Removing old nvidia-current-470.86 DKMS files...
Loading new nvidia-current-470.86 DKMS files...
Error! DKMS tree already contains: nvidia-current-470.86

You cannot add the same module/version combo more than once.

dpkg: error processing package nvidia-kernel-dkms (--configure):
installed nvidia-kernel-dkms package post-installation script

subprocess returned error exit status 3

dpkg: dependency problems prevent configuration of nvidia-driver:

nvidia-driver depends on nvidia-kernel-dkms (= 470.86-1) |
nvidia-kernel-470.86; however:

Package nvidia-kernel-dkms is not configured yet.

Package nvidia-kernel-470.86 is not installed.
Package nvidia-kernel-dkms which provides nvidia-kernel-470.86 is not

configured yet.

dpkg: error processing package nvidia-driver (--configure):
dependency problems - leaving unconfigured

Errors were encountered while processing:
nvidia-kernel-dkms
nvidia-driver
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Drew Parsons]

Package: nvidia-kernel-dkms
Followup-For: Bug #996461
Control: found -1 470.86-1
Control: fixed -1 495.44-1

This bug seems to be fixed now in 495.44-1.
It was still broken with 470.86-1.

I guess we can close this bug now. Would any more diagnostic data be
useful before closing?

Drew


-- Package-specific info:
uname -a:
Linux sandy 5.15.0-2-amd64 #1 SMP Debian 5.15.5-2 (2021-12-18) x86_64 GNU/Linux

/proc/version:
Linux version 5.15.0-2-amd64 (debian...@lists.debian.org) (gcc-11 (Debian 11.2.0-13) 11.2.0, GNU ld (GNU Binutils for Debian) 2.37) #1 SMP Debian 5.15.5-2 (2021-12-18)

/proc/driver/nvidia/version:
NVRM version: NVIDIA UNIX x86_64 Kernel Module 495.44 Fri Oct 22 06:13:12 UTC 2021
GCC version: gcc version 11.2.0 (Debian 11.2.0-13)

lspci 'display controller [030?]':
00:02.0 VGA compatible controller [0300]: Intel Corporation CometLake-U GT2 [UHD Graphics] [8086:9b41] (rev 02) (prog-if 00 [VGA controller])
Subsystem: Lenovo CometLake-U GT2 [UHD Graphics] [17aa:22b0]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 158
IOMMU group: 1
Region 0: Memory at ed000000 (64-bit, non-prefetchable) [size=16M]
Region 2: Memory at d0000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at 4000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: i915
Kernel modules: i915

2d:00.0 3D controller [0302]: NVIDIA Corporation GP108M [GeForce MX330] [10de:1d16] (rev a1)
Subsystem: Lenovo GP108M [GeForce MX330] [17aa:22b0]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 179
IOMMU group: 18
Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at 3000 [size=128]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidia



-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.15.0-2-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages nvidia-kernel-dkms depends on:
ii dkms 2.8.7-2
pn nvidia-firmware-495.44 <none>
ii nvidia-installer-cleanup 20151021+13
ii nvidia-kernel-support [nvidia-kernel-support--v1] 495.44-1

Versions of packages nvidia-kernel-dkms recommends:
ii libcuda1 495.44-1
ii nvidia-driver 495.44-1

nvidia-kernel-dkms suggests no packages.

Versions of packages nvidia-driver depends on:
ii nvidia-alternative 495.44-1
ii nvidia-driver-bin 495.44-1
ii nvidia-driver-libs 495.44-1
ii nvidia-installer-cleanup 20151021+13
ii nvidia-legacy-check 495.44-1
ii nvidia-support 20151021+13
ii nvidia-vdpau-driver 495.44-1
ii xserver-xorg-video-nvidia 495.44-1

Versions of packages nvidia-driver recommends:
ii libnvidia-cfg1 495.44-1
ii nvidia-persistenced 470.82.00-1
pn nvidia-settings <none>

nvidia-driver suggests no packages.

Versions of packages nvidia-driver-libs:amd64 depends on:
ii libgl1-nvidia-glvnd-glx 495.44-1
ii nvidia-egl-icd 495.44-1

Versions of packages nvidia-driver-libs:amd64 recommends:
ii libgles-nvidia1 495.44-1
ii libgles-nvidia2 495.44-1
ii libglx-nvidia0 495.44-1
ii libnvidia-cfg1 495.44-1
ii libnvidia-encode1 495.44-1
ii libopengl0 1.3.4-2+b1
ii nvidia-driver-libs 495.44-1
ii nvidia-vulkan-icd 495.44-1

Versions of packages nvidia-driver-libs:i386 depends on:
ii libgl1-nvidia-glvnd-glx 495.44-1
ii nvidia-egl-icd 495.44-1

Versions of packages nvidia-driver-libs:i386 recommends:
ii libgles-nvidia1 495.44-1
ii libgles-nvidia2 495.44-1
ii libglx-nvidia0 495.44-1
ii libnvidia-encode1 495.44-1
ii libopengl0 1.3.4-2+b1
ii nvidia-vulkan-icd 495.44-1

Versions of packages xserver-xorg-video-nvidia depends on:
ii libc6 2.33-1
ii libnvidia-glcore 495.44-1
ii nvidia-alternative 495.44-1
ii nvidia-installer-cleanup 20151021+13
ii nvidia-legacy-check 495.44-1
ii nvidia-support 20151021+13
ii xserver-xorg-core [xorg-video-abi-24] 2:1.20.13-3

Versions of packages xserver-xorg-video-nvidia recommends:
ii nvidia-driver 495.44-1
pn nvidia-settings <none>
ii nvidia-vdpau-driver 495.44-1

xserver-xorg-video-nvidia suggests no packages.

Versions of packages nvidia-alternative depends on:
ii dpkg 1.21.1
ii glx-alternative-nvidia 1.2.1
ii nvidia-legacy-check 495.44-1

Versions of packages glx-alternative-nvidia depends on:
ii dpkg 1.21.1
ii glx-alternative-mesa 1.2.1
ii glx-diversions 1.2.1
ii update-glx 1.2.1

Versions of packages glx-alternative-nvidia suggests:
ii nvidia-driver [nvidia-driver-any] 495.44-1

Versions of packages xserver-xorg-video-intel depends on:
ii libc6 2.33-1
ii libdrm-intel1 2.4.109-2
ii libdrm2 2.4.109-2
ii libpciaccess0 0.16-1
ii libpixman-1-0 0.40.0-1
ii libudev1 249.7-1
ii libx11-6 2:1.7.2-2+b1
ii libx11-xcb1 2:1.7.2-2+b1
ii libxcb-dri2-0 1.14-3
ii libxcb-util1 0.4.0-1+b1
ii libxcb1 1.14-3
ii libxcursor1 1:1.2.0-2
ii libxdamage1 1:1.1.5-2
ii libxext6 2:1.3.4-1
ii libxfixes3 1:5.0.3-2
ii libxinerama1 2:1.1.4-2
ii libxrandr2 2:1.5.2-1
ii libxrender1 1:0.9.10-1
ii libxss1 1:1.2.3-1
ii libxtst6 2:1.2.3-1
ii libxvmc1 2:1.0.12-2
ii xserver-xorg-core [xorg-video-abi-24] 2:1.20.13-3

Versions of packages nvidia-kernel-dkms is related to:
ii bumblebee 3.2.1-27
pn bumblebee-nvidia <none>
pn ccache <none>
pn libcuda.so.1 <none>
ii libcuda1 [libcuda1-any] 495.44-1
pn libdrm-nouveau1 <none>
pn libdrm-nouveau1a <none>
ii libdrm-nouveau2 2.4.109-2
ii libegl1 1.3.4-2+b1
ii libgl1 1.3.4-2+b1
ii libgl1-nvidia-glvnd-glx [libgl1-nvidia-glx-any] 495.44-1
ii libgles1 1.3.4-2+b1
ii libgles2 1.3.4-2+b1
ii libglvnd0 1.3.4-2+b1
ii libglx0 1.3.4-2+b1
ii libnvidia-cfg1 [libnvidia-cfg1-any] 495.44-1
ii libnvidia-ml1 [libnvidia-ml.so.1] 495.44-1
pn libopencl0 <none>
ii libprimus-vk1 1.6.1-1
ii libvulkan1 1.2.189.0-2
pn linux-headers <none>
ii make 4.3-4.1
ii mesa-vulkan-drivers [vulkan-icd] 21.2.6-1
ii nvidia-driver [nvidia-glx-any] 495.44-1
pn nvidia-driver-any <none>
ii nvidia-driver-libs [nvidia-driver-libs-any] 495.44-1
pn nvidia-glx <none>
ii nvidia-kernel-common 20151021+13
ii nvidia-kernel-dkms 495.44-1
pn nvidia-kernel-source <none>
ii nvidia-kernel-support [nvidia-kernel-support-any] 495.44-1
ii nvidia-modprobe 495.44-1
ii nvidia-opencl-icd [opencl-icd] 495.44-1
ii nvidia-primus-vk-wrapper 1.6.1-1
pn nvidia-settings <none>
ii nvidia-support 20151021+13
ii nvidia-vulkan-icd [vulkan-icd] 495.44-1
pn nvidia-vulkan-icd-any <none>
pn nvidia-xconfig <none>
ii ocl-icd-libopencl1 [libopencl1] 2.2.14-3
ii primus 0~20150328-13
ii primus-libs 0~20150328-13
ii primus-nvidia 0~20150328-13
ii primus-vk 1.6.1-1
ii primus-vk-nvidia 1.6.1-1
ii xserver-xorg 1:7.7+23
ii xserver-xorg-core 2:1.20.13-3
ii xserver-xorg-legacy 2:1.20.13-3
ii xserver-xorg-video-nouveau 1:1.0.17-1
ii xserver-xorg-video-nvidia [xserver-xorg-video-nvidia-any] 495.44-1

-- no debconf information