Bug#1026906: /lib/firmware/iwlwifi-cc-a0-72.ucode: "Microcode SW error detected" after idle for a while and breaks wifi

[Yuxuan Wang]

Package: firmware-iwlwifi
Version: 20221109-4
Severity: important
File: /lib/firmware/iwlwifi-cc-a0-72.ucode
X-Debbugs-Cc: fish...@gmail.com

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

* What led up to the situation?
* What exactly did you do (or not do) that was effective (or
ineffective)?
* What was the outcome of this action?
* What outcome did you expect instead?

*** End of the template - remove these template lines ***

With firmware iwlwifi-cc-a0-72.ucode, the wifi chip would throw "Microcode SW
error detected" after the system is idle for a while. After wakig up the system,
the wifi would appears to be connected (in NetworkManager), but pinging the
router will give "Destination Host Unreachable", and I have to turn wifi off and
one again in NetworkManager to fix it.

This is the kernel log when the error happens:

Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Microcode SW error detected. Restarting 0x0.
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Transport status: 0x0000004A, valid: 6
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Loaded firmware version: 72.daa05125.0 cc-a0-72.ucode
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000A210 | trm_hw_status0
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | trm_hw_status1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x004FBE16 | branchlink2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x004F23FE | interruptlink1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x004F23FE | interruptlink2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000B5C2 | data1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00001000 | data2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | data3
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x2540D56E | beacon time
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xB0C55A96 | tsf low
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000043B | tsf hi
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | time gp1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x59D3B493 | time gp2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000001 | uCode revision type
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000048 | uCode version major
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xDAA05125 | uCode version minor
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000340 | hw version
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00489000 | board version
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x8086FC03 | hcmd
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x24020000 | isr0
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00400000 | isr1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x08F00002 | isr2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00C3380C | isr3
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | isr4
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x05E4001C | last cmd Id
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000B5C2 | wait_event
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000094 | l2p_control
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00010034 | l2p_duration
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000000F | l2p_mhvalid
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | l2p_addr_match
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000009 | lmpm_pmg_sel
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | timestamp
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00005078 | flow_handler
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Transport status: 0x0000004A, valid: 7
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x20003463 | ADVANCED_SYSASSERT
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | umac branchlink1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x80455E3C | umac branchlink2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xC0081200 | umac interruptlink1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | umac interruptlink2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xB0C55A88 | umac data1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x59D3B483 | umac data2
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x57C61FD2 | umac data3
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000048 | umac major
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xDAA05125 | umac minor
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x59D3B48D | frame pointer
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0xC0885E00 | stack pointer
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0046010C | last host cmd
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | isr status reg
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: IML/ROM dump:
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000003 | IML/ROM error/state
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00005DE5 | IML/ROM data1
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: Fseq Registers:
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x60000041 | FSEQ_ERROR_CODE
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00290021 | FSEQ_TOP_INIT_VERSION
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00050008 | FSEQ_CNVIO_INIT_VERSION
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000A503 | FSEQ_OTP_VERSION
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00100530 | FSEQ_CNVI_ID
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000532 | FSEQ_CNVR_ID
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
Dec 20 16:00:14 perch kernel: iwlwifi 0000:3b:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms).
Dec 20 16:00:14 perch kernel: ieee80211 phy0: Hardware restart was requested

The hardware info according to `lspci -nn -d ::280` is:

3b:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)

I tried to downgrade it, via:

mv /lib/firmware/iwlwifi-cc-a0-72.ucode /lib/firmware/iwlwifi-cc-a0-72.ucode.backup

And then reload the kernel models, which loaded 71 instead, but that still has
a similar issue: instead of "Destination Host Unreachable", pinging the router
gives me super high latency (in seconds instead of milliseconds) and the network
is almost unusable, and I have to turn wifi off and on again in NetworkManager
to fix it. The kernel log for 71 is:

Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Microcode SW error detected. Restarting 0x0.
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Transport status: 0x0000004A, valid: 6
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Loaded firmware version: 71.058653f6.0 cc-a0-71.ucode
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00A0A200 | trm_hw_status0
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | trm_hw_status1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x004FB4C6 | branchlink2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x004FE6D4 | interruptlink1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x004FE6D4 | interruptlink2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00014DC6 | data1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00001000 | data2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | data3
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x4A403FFF | beacon time
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0xD36FB005 | tsf low
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x000003B9 | tsf hi
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | time gp1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x22E16CD6 | time gp2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000001 | uCode revision type
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000047 | uCode version major
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x058653F6 | uCode version minor
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000340 | hw version
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x18489000 | board version
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x804DFC03 | hcmd
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x20028000 | isr0
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00400000 | isr1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x08F8000A | isr2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00C3780C | isr3
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | isr4
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x057D001C | last cmd Id
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00014DC6 | wait_event
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00006288 | l2p_control
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00018034 | l2p_duration
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | l2p_mhvalid
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x000000E7 | l2p_addr_match
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000009 | lmpm_pmg_sel
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | timestamp
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x0000D0D0 | flow_handler
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Transport status: 0x0000004A, valid: 7
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x20003463 | ADVANCED_SYSASSERT
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | umac branchlink1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x80455B2C | umac branchlink2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0xC0081200 | umac interruptlink1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | umac interruptlink2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0xD36FAFF8 | umac data1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x22E16CC7 | umac data2
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x739CEFD2 | umac data3
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000047 | umac major
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x058653F6 | umac minor
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x22E16CD1 | frame pointer
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0xC0885E00 | stack pointer
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00BB010C | last host cmd
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000000 | isr status reg
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: IML/ROM dump:
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000003 | IML/ROM error/state
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00005FB8 | IML/ROM data1
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: Fseq Registers:
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x60000000 | FSEQ_ERROR_CODE
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x80290021 | FSEQ_TOP_INIT_VERSION
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00050008 | FSEQ_CNVIO_INIT_VERSION
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x0000A503 | FSEQ_OTP_VERSION
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00100530 | FSEQ_CNVI_ID
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000532 | FSEQ_CNVR_ID
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
Dec 22 00:21:35 perch kernel: iwlwifi 0000:3b:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms).
Dec 22 00:21:35 perch kernel: ieee80211 phy1: Hardware restart was requested

I have to downgrade again (from 71 to 63, there's no version in between) to fix
the issue.

-- System Information:
Debian Release: bookworm/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-6-amd64 (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

firmware-iwlwifi depends on no packages.

firmware-iwlwifi recommends no packages.

Versions of packages firmware-iwlwifi suggests:
ii initramfs-tools 0.142

-- no debconf information

[Diederik de Haas]

On Friday, 23 December 2022 18:40:32 CET Yuxuan Wang wrote:
> Package: firmware-iwlwifi
> Version: 20221109-4

>
> With firmware iwlwifi-cc-a0-72.ucode, the wifi chip would throw "Microcode
> SW error detected" after the system is idle for a while. After wakig up the
> system, the wifi would appears to be connected (in NetworkManager), but
> pinging the router will give "Destination Host Unreachable", and I have to
> turn wifi off and one again in NetworkManager to fix it.
>
> This is the kernel log when the error happens:

> ...

> (delay=0ms). Dec 20 16:00:14 perch kernel: ieee80211 phy0: Hardware restart
> was requested
>
> The hardware info according to `lspci -nn -d ::280` is:
>
> 3b:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200
> [8086:2723] (rev 1a)
>
> I tried to downgrade it, via:
>
> mv /lib/firmware/iwlwifi-cc-a0-72.ucode
> /lib/firmware/iwlwifi-cc-a0-72.ucode.backup
>
> And then reload the kernel models, which loaded 71 instead, but that still
> has a similar issue: instead of "Destination Host Unreachable", pinging the
> router gives me super high latency (in seconds instead of milliseconds) and
> the network is almost unusable, and I have to turn wifi off and on again in
> NetworkManager to fix it. The kernel log for 71 is:
>

> I have to downgrade again (from 71 to 63, there's no version in between) to
> fix the issue.

So to summarize: It works with 63, but it fails with 71 or 72? And in order to
make it work, you have to (actively) remove the 71 and 72 versions?

[Yuxuan Wang]

Correct.