Bug#1006500: Missing bnx2x firmware 7.13.21.0 renders NIC unusable with Linux 5.16

[Salvatore Bonaccorso]

Hi,

On Sat, Feb 26, 2022 at 12:37:55PM +0000, Etienne Dechamps wrote:
> Package: firmware-bnx2x
> Version: 20210818-1
> Severity: grave
>
> On Linux 5.16, the bnx2x module requests firmware 7.13.21.0:
>
> # modinfo bnx2x
> filename:
> /lib/modules/5.16.0-2-amd64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
> firmware: bnx2x/bnx2x-e2-7.13.21.0.fw
> firmware: bnx2x/bnx2x-e1h-7.13.21.0.fw
> firmware: bnx2x/bnx2x-e1-7.13.21.0.fw
>
> This firmware is not present in the firmware-bnx2x package.
>
> Now, my understanding is that the bnx2x module can fall back to an
> earlier version (7.13.15.0) as needed, but in practice that might not
> actually help, because update-initramfs only looks for the firmware
> version from the module information and doesn't include any other
> version:
>
> # update-initramfs -u
> W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e2-7.13.21.0.fw
> for module bnx2x
> W: Possible missing firmware
> /lib/firmware/bnx2x/bnx2x-e1h-7.13.21.0.fw for module bnx2x
> W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e1-7.13.21.0.fw
> for module bnx2x
>
> Since no firmware is included in the initramfs, the bnx2x module
> unsurprisingly fails to initialize on boot:
>
> kernel: bnx2x 0000:02:00.1: firmware: failed to load
> bnx2x/bnx2x-e2-7.13.21.0.fw (-2)
> kernel: bnx2x 0000:02:00.1: Direct firmware load for
> bnx2x/bnx2x-e2-7.13.21.0.fw failed with error -2
> kernel: bnx2x 0000:02:00.1: firmware: failed to load
> bnx2x/bnx2x-e2-7.13.15.0.fw (-2)
> kernel: bnx2x 0000:02:00.1: Direct firmware load for
> bnx2x/bnx2x-e2-7.13.15.0.fw failed with error -2
>
> Adding insult to injury, it doesn't look like it's possible to recover
> from this state - as far as I could tell, once bnx2x fails to
> initialize, it's game over until the next reboot, even if the module
> is unloaded and reloaded, and even if the PCI device is removed and
> rescanned.
>
> This basically means that, when booting current Debian Unstable with
> Linux 5.16, bnx2x NICs become *permanently unusable*, potentially
> locking users and admins out of the system.
>
> My suggested short-term fix would be to update the firmware-bnx2x
> package to include the 7.13.21.0 firmware version. There's also a
> discussion to be had with regard to update-initramfs, which should
> perhaps try harder to find potentially compatible firmware versions -
> indeed, if update-initramfs had included 7.13.15.0, I believe this
> issue would have been avoided.
>
> For those affected, here's the workaround I used:
>
> 1. Manually download the 7.13.21.0 firmware files from
> https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/bnx2x
> 2. Put the files in /lib/firmware/bnx2x
> 3. Run update-initramfs -u
> 4. Reboot

Unless mistaken, there is the kernel side of this issue upstream
https://bugzilla.kernel.org/show_bug.cgi?id=215627 which was fixed in
5.17-rc6[1], and got backported as well to 5.16.12 (and so in the next
unstable upload).

[1]: https://git.kernel.org/linus/e13ad1443684f7afaff24cf207e85e97885256bd

Regards,
Salvatore

[Christoph Anton Mitterer]

Hey.

I just ran into the same problem in bullseye.
Linux lcg-lrz-admin 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 GNU/Linux

Salvatore, AFAICS, that kernel should already contain the commit you've
mentioned, however, it still gives an error about missing firmware.

Downloading these 3 files from:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/bnx2x

and manually adding it to /lib/firmware fixes the problem for me.


Could someone please add them to the package and also for bullseye?


Thanks,
Chris.

[Frank Burkhardt]

Hi,

the problem affects Debian stable as well. Use case is installing
an old HP server using PXE/Netboot(5.10.0-16)/Preseed. I managed
to smuggle the 3 missing files (bnx2x-e1-7.13.21.0.fw,
bnx2x-e1h-7.13.21.0.fw, bnx2x-e2-7.13.21.0.fw) into
/lib/firmware of the installer but without them being included
in the firmware-bnx2x package, I can't install them properly into
the target partition.

Best regards,

Frank

[Enrique]

Package: firmware-bnx2x
Version: 20210315-3
Followup-For: Bug #1006500
X-Debbugs-Cc: cqu...@arcor.de

I have this problem when installing a server using the netinst ISO image. I
then tried with the firmware image from here
https://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-
firmware/11.5.0+nonfree/amd64/iso-cd/ but it does not work because it ships
with version 20210315-3, which does not contain bnx2x/bnx2x-e2-7.13.21.0.fw,
which the installer insists on installing.

I workarounded the issue repackaging the ISO image adding version 20221109-2
from bookworm and then it finally worked.

It would be good if that version of the firmware is included in bullseye and
added to the (unofficial) firmware CD.

-- System Information:
Debian Release: 11.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500,
'stable'), (100, 'bullseye-fasttrack'), (100, 'bullseye-backports-staging')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-16-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

firmware-bnx2x depends on no packages.

firmware-bnx2x recommends no packages.

Versions of packages firmware-bnx2x suggests:

[Shane St. Savage]

As someone who has bricked multiple servers and weekends trying to work around this in an iPXE/preseed installer environment, I would also be very appreciative of a backport release of 20220913-1~deb11u1 (assuming enough settlement time has passed). Thanks for all your work on Debian!