[v6.1] BUG: workqueue lockup

syzbot

unread,

Mar 7, 2023, 12:51:41 PM3/7/23

to syzkaller...@googlegroups.com

Hello,

syzbot found the following issue on:

HEAD commit: 42616e0f09fb Linux 6.1.15
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12b162ccc80000
kernel config: https://syzkaller.appspot.com/x/.config?x=650737f7e9682672
dashboard link: https://syzkaller.appspot.com/bug?extid=98a7372a1d818f0d98a6
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/f10713d1fd0f/disk-42616e0f.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5a1307bb774e/vmlinux-42616e0f.xz
kernel image: https://storage.googleapis.com/syzbot-assets/388238a30fe4/Image-42616e0f.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+98a737...@syzkaller.appspotmail.com

BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 52s!
Showing busy workqueues and worker pools:
workqueue md_misc: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=-65535/1172321806 refcnt=1 MAYDAY
inactive: 0x2030312034203233
Unable to handle kernel paging request at virtual address 0006064406466408
Mem abort info:
ESR = 0x0000000096000004
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x04: level 0 translation fault
Data abort info:
ISV = 0, ISS = 0x00000004
CM = 0, WnR = 0
[0006064406466408] address between user and kernel address ranges
Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.1.15-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
pstate: 004000c5 (nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : pr_cont_work kernel/workqueue.c:4714 [inline]
pc : show_pwq+0x634/0x97c kernel/workqueue.c:4791
lr : show_pwq+0x704/0x97c kernel/workqueue.c:4790
sp : ffff8000080079a0
x29: ffff8000080079b0 x28: dfff800000000000 x27: ffff800008225c30
x26: 2030312034203233 x25: 0000000000000001 x24: 2034322032332041
x23: ffff8000123a20c0 x22: ffff8000123a1fa0 x21: 2034322032332031
x20: ffff8000123a1f20 x19: ffff0001b459be68 x18: ffff800008007700
x17: 0000000000000000 x16: ffff8000122508ac x15: 0000000000000100
x14: 0000000000000000 x13: 3333323330323433 x12: 0000000000000001
x11: ff8080000821de64 x10: 0000000000000000 x9 : ffff80000821de64
x8 : 0406864406466408 x7 : 3330323433303231 x6 : 3330333032783020
x5 : ffff80001978feb8 x4 : ffff80001978feba x3 : ffff8000085890fc
x2 : 0000000000000001 x1 : 0000000000000000 x0 : 0000000000000000
Call trace:
pr_cont_work kernel/workqueue.c:4714 [inline]
show_pwq+0x634/0x97c kernel/workqueue.c:4791
show_one_workqueue+0x1b4/0x25c kernel/workqueue.c:4828
show_all_workqueues+0x11c/0x840 kernel/workqueue.c:4904
wq_watchdog_timer_fn+0x624/0x918 kernel/workqueue.c:5880
call_timer_fn+0x250/0xcb4 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1519 [inline]
__run_timers+0x554/0x718 kernel/time/timer.c:1790
run_timer_softirq+0xcc/0x114 kernel/time/timer.c:1805
__do_softirq+0x36c/0xfdc kernel/softirq.c:571
____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:79
call_on_irq_stack+0x2c/0x54 arch/arm64/kernel/entry.S:889
do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:84
invoke_softirq kernel/softirq.c:452 [inline]
__irq_exit_rcu+0x28c/0x534 kernel/softirq.c:650
irq_exit_rcu+0x14/0x84 kernel/softirq.c:662
__el1_irq arch/arm64/kernel/entry-common.c:472 [inline]
el1_interrupt+0x38/0x68 arch/arm64/kernel/entry-common.c:486
el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491
el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:577
arch_local_irq_enable+0xc/0x18 arch/arm64/include/asm/irqflags.h:35
default_idle_call+0x68/0xdc kernel/sched/idle.c:109
cpuidle_idle_call kernel/sched/idle.c:191 [inline]
do_idle+0x1e0/0x514 kernel/sched/idle.c:303
cpu_startup_entry+0x24/0x28 kernel/sched/idle.c:400
rest_init+0x278/0x290 init/main.c:729
start_kernel+0x0/0x608 init/main.c:890
start_kernel+0x448/0x608 init/main.c:1145
__primary_switched+0xb8/0xc0 arch/arm64/kernel/head.S:468
Code: b0050c37 910302f7 910042b8 d343ff08 (387c6908)
---[ end trace 0000000000000000 ]---
----------------
Code disassembly (best guess):
0: b0050c37 adrp x23, 0xa185000
4: 910302f7 add x23, x23, #0xc0
8: 910042b8 add x24, x21, #0x10
c: d343ff08 lsr x8, x24, #3
* 10: 387c6908 ldrb w8, [x8, x28] <-- trapping instruction

---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,

Mar 11, 2023, 3:51:41 AM3/11/23

to syzkaller...@googlegroups.com

Hello,

syzbot found the following issue on:

HEAD commit: abddfcf701a5 Linux 5.15.99
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=11349732c80000
kernel config: https://syzkaller.appspot.com/x/.config?x=53e47369740caba3
dashboard link: https://syzkaller.appspot.com/bug?extid=fc5688cf31fbb0d8587f

compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15dfb314c80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=175b9f6cc80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/78c522505d54/disk-abddfcf7.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/caca388168a7/vmlinux-abddfcf7.xz
kernel image: https://storage.googleapis.com/syzbot-assets/e6db198604be/Image-abddfcf7.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by: syzbot+fc5688...@syzkaller.appspotmail.com

BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 31s!

Showing busy workqueues and worker pools:

workqueue events: flags=0x0
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: psi_avgs_work
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4
in-flight: 1529:free_work
pending: vmstat_shepherd, ovs_dp_masks_rebalance
workqueue events_long: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: defense_work_handler
workqueue events_unbound: flags=0x2
pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/512 refcnt=3
in-flight: 9:flush_memcg_stats_dwork
workqueue events_power_efficient: flags=0x82
pwq 4: cpus=0-1 flags=0x4 nice=0 active=5/256 refcnt=7
in-flight: 364:gc_worker
pending: fb_flashcursor, neigh_periodic_work, neigh_periodic_work, do_cache_clean
workqueue mm_percpu_wq: flags=0x8
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: vmstat_update
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: vmstat_update
workqueue writeback: flags=0x4a
pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3
pending: wb_workfn
workqueue kblockd: flags=0x18
pwq 3: cpus=1 node=0 flags=0x0 nice=-20 active=1/256 refcnt=2
pending: blk_mq_timeout_work
workqueue dm_bufio_cache: flags=0x8
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: work_fn
workqueue krxrpcd: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2
pending: rxrpc_peer_keepalive_worker
pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=3 idle: 3770 13
pool 4: cpus=0-1 flags=0x4 nice=0 hung=0s workers=5 idle: 4147 136 516

---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

syzbot

unread,

Mar 19, 2023, 4:39:46 PM3/19/23

to syzkaller...@googlegroups.com

syzbot has found a reproducer for the following issue on:

HEAD commit: 7eaef76fbc46 Linux 6.1.20
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=114ed681c80000
kernel config: https://syzkaller.appspot.com/x/.config?x=29ad3fe3c7b61175

dashboard link: https://syzkaller.appspot.com/bug?extid=98a7372a1d818f0d98a6
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=140ca616c80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/34f95428f5fb/disk-7eaef76f.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/1bdd9b2c390d/vmlinux-7eaef76f.xz
kernel image: https://storage.googleapis.com/syzbot-assets/419140981cfa/Image-7eaef76f.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+98a737...@syzkaller.appspotmail.com

BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 41s!

Showing busy workqueues and worker pools:

workqueue events: flags=0x0
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2

in-flight: 4530:nsim_fib_event_work
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=14/256 refcnt=15
in-flight: 4422:nsim_dev_hwstats_traffic_work
pending: nsim_dev_hwstats_traffic_work, nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, kfree_rcu_monitor, psi_avgs_work, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, switchdev_deferred_process_work, rht_deferred_worker, vmstat_shepherd
workqueue events_long: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=8/256 refcnt=9
pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler, br_multicast_gc_work, br_multicast_gc_work, br_fdb_cleanup, br_fdb_cleanup
workqueue events_power_efficient: flags=0x82
pwq 4: cpus=0-1 flags=0x4 nice=0 active=2/256 refcnt=4
pending: neigh_managed_work, neigh_managed_work
workqueue netns: flags=0xe000a
pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3
in-flight: 9:cleanup_net
workqueue mm_percpu_wq: flags=0x8

pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: vmstat_update

workqueue mld: flags=0x40008
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=3
pending: mld_dad_work
inactive: mld_dad_work
workqueue ipv6_addrconf: flags=0x40008
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=18
pending: addrconf_dad_work
inactive: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work
workqueue krxrpcd: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=5
pending: rxrpc_peer_keepalive_worker
inactive: rxrpc_peer_keepalive_worker, rxrpc_peer_keepalive_worker, rxrpc_peer_keepalive_worker
workqueue bat_events: flags=0xe000a
pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=6
in-flight: 11:batadv_nc_worker
inactive: batadv_nc_worker, batadv_nc_worker, batadv_nc_worker
workqueue wg-crypt-wg0: flags=0x28