INFO: rcu detected stall in prog_array_map_clear_deferred
梅开彦
Reported-by: Kaiyan Mei <M2024...@hust.edu.cn>
Reported-by: Yinhao Hu <ddd...@hust.edu.cn>
Reviewed-by: Dongliang Mu <dz...@hust.edu.cn>
# Crash Report
```
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 0-...!: (2 ticks this GP) idle=a2bc/1/0x4000000000000000 softirq=187184/187184 fqs=0
rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P56520/1:b..l
rcu: (detected by 1, t=12103 jiffies, g=87725, q=1060 ncpus=2)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 9930 Comm: sshd Not tainted 6.17.0-g39e9d5f63075 #1 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:arch_atomic_fetch_add arch/x86/include/asm/atomic.h:93 [inline]
RIP: 0010:raw_atomic_fetch_sub_release include/linux/atomic/atomic-arch-fallback.h:949 [inline]
RIP: 0010:atomic_fetch_sub_release include/linux/atomic/atomic-instrumented.h:401 [inline]
RIP: 0010:__refcount_sub_and_test include/linux/refcount.h:389 [inline]
RIP: 0010:__refcount_dec_and_test include/linux/refcount.h:432 [inline]
RIP: 0010:refcount_dec_and_test include/linux/refcount.h:450 [inline]
RIP: 0010:neigh_parms_put net/core/neighbour.c:910 [inline]
RIP: 0010:neigh_destroy+0x3a1/0x640 net/core/neighbour.c:942
Code: fa 48 c1 ea 03 80 3c 02 00 0f 85 8f 02 00 00 4c 8b 65 28 be 04 00 00 00 bb ff ff ff ff 4d 8d 6c 24 44 4c 89 ef e8 ff 0f 08 f9 <f0> 41 0f c1 5c 24 44 31 ff 89 de e8 0f 82 9f f8 85 db 0f 8e 3e 01
RSP: 0018:ffa0000000007808 EFLAGS: 00010246
RAX: 0000000000000001 RBX: 00000000ffffffff RCX: ffffffff891b3271
RDX: ffe21c000ee16ee9 RSI: 0000000000000004 RDI: ff110000770b7744
RBP: ff1100013c790800 R08: 0000000000000001 R09: ffe21c000ee16ee8
R10: ff110000770b7747 R11: 0000000000000001 R12: ff110000770b7700
R13: ff110000770b7744 R14: ff11000195a43810 R15: ff1100013c790888
FS: 00007f70537ee480(0000) GS:ff1100010d2a7000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000d4c000 CR3: 00000001080e2000 CR4: 0000000000753ef0
DR0: 00000000000003ff DR1: 00000000000003ff DR2: 00000000000003ff
DR3: 00000000000003ff DR6: 00000000ffff0ff0 DR7: 0000000000000600
PKRU: 55555554
Call Trace:
<IRQ>
neigh_release include/net/neighbour.h:450 [inline]
neigh_cleanup_and_release net/core/neighbour.c:122 [inline]
neigh_remove_one+0x26d/0x300 net/core/neighbour.c:250
neigh_forced_gc net/core/neighbour.c:281 [inline]
neigh_alloc net/core/neighbour.c:513 [inline]
___neigh_create+0x13a4/0x2860 net/core/neighbour.c:656
ip6_finish_output2+0xa94/0x1ac0 net/ipv6/ip6_output.c:128
__ip6_finish_output net/ipv6/ip6_output.c:209 [inline]
ip6_finish_output+0x701/0x11d0 net/ipv6/ip6_output.c:220
NF_HOOK_COND include/linux/netfilter.h:307 [inline]
ip6_output+0x25f/0x710 net/ipv6/ip6_output.c:247
dst_output include/net/dst.h:464 [inline]
NF_HOOK include/linux/netfilter.h:318 [inline]
ndisc_send_skb+0xa69/0x1ee0 net/ipv6/ndisc.c:512
ndisc_send_rs+0x128/0x690 net/ipv6/ndisc.c:722
addrconf_rs_timer+0x3ee/0x840 net/ipv6/addrconf.c:4037
call_timer_fn+0x1a5/0x630 kernel/time/timer.c:1747
expire_timers kernel/time/timer.c:1798 [inline]
__run_timers+0x666/0x920 kernel/time/timer.c:2372
__run_timer_base kernel/time/timer.c:2384 [inline]
__run_timer_base kernel/time/timer.c:2376 [inline]
run_timer_base+0xc5/0x120 kernel/time/timer.c:2393
run_timer_softirq+0x1a/0x40 kernel/time/timer.c:2403
handle_softirqs+0x1d4/0x870 kernel/softirq.c:622
do_softirq kernel/softirq.c:523 [inline]
do_softirq+0xac/0xe0 kernel/softirq.c:510
</IRQ>
<TASK>
__local_bh_enable_ip+0x100/0x120 kernel/softirq.c:450
local_bh_enable include/linux/bottom_half.h:33 [inline]
rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline]
__dev_queue_xmit+0x1bf9/0x4160 net/core/dev.c:4790
dev_queue_xmit include/linux/netdevice.h:3365 [inline]
neigh_hh_output include/net/neighbour.h:531 [inline]
neigh_output include/net/neighbour.h:545 [inline]
ip_finish_output2+0xc2b/0x1f50 net/ipv4/ip_output.c:237
__ip_finish_output net/ipv4/ip_output.c:315 [inline]
__ip_finish_output+0x447/0x950 net/ipv4/ip_output.c:297
ip_finish_output+0x35/0x380 net/ipv4/ip_output.c:325
NF_HOOK_COND include/linux/netfilter.h:307 [inline]
ip_output+0x1dc/0x520 net/ipv4/ip_output.c:438
dst_output include/net/dst.h:464 [inline]
ip_local_out net/ipv4/ip_output.c:131 [inline]
__ip_queue_xmit+0x18c4/0x1ef0 net/ipv4/ip_output.c:534
__tcp_transmit_skb+0x2cd9/0x4480 net/ipv4/tcp_output.c:1628
tcp_transmit_skb net/ipv4/tcp_output.c:1646 [inline]
tcp_write_xmit+0x188e/0x7ab0 net/ipv4/tcp_output.c:2988
__tcp_push_pending_frames+0xae/0x390 net/ipv4/tcp_output.c:3171
tcp_push+0x229/0x700 net/ipv4/tcp.c:777
tcp_sendmsg_locked+0x2e1c/0x3f90 net/ipv4/tcp.c:1376
tcp_sendmsg+0x2f/0x50 net/ipv4/tcp.c:1413
inet_sendmsg+0xb9/0x150 net/ipv4/af_inet.c:853
sock_sendmsg_nosec net/socket.c:727 [inline]
__sock_sendmsg net/socket.c:742 [inline]
sock_write_iter+0x518/0x600 net/socket.c:1195
new_sync_write fs/read_write.c:593 [inline]
vfs_write+0xc0d/0x1170 fs/read_write.c:686
ksys_write+0x1ef/0x240 fs/read_write.c:738
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xcb/0xfa0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f7053b17300
Code: 40 00 48 8b 15 01 9b 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d e1 22 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89
RSP: 002b:00007ffe78cbed08 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000334 RCX: 00007f7053b17300
RDX: 0000000000000334 RSI: 00007f70536f8010 RDI: 0000000000000004
RBP: 00005596c69f4420 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
R13: 00007ffe78cbee10 R14: 0000000000000000 R15: 00007ffe78cbed90
</TASK>
task:syz.5.11903 state:R running task stack:27064 pid:56520 tgid:56517 ppid:38689 task_flags:0x400140 flags:0x00080003
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5325 [inline]
__schedule+0x1044/0x5bb0 kernel/sched/core.c:6929
preempt_schedule_irq+0x51/0x90 kernel/sched/core.c:7256
irqentry_exit+0x36/0x90 kernel/entry/common.c:211
asm_sysvec_call_function_single+0x1a/0x20 arch/x86/include/asm/idtentry.h:704
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x80 kernel/kcov.c:210
Code: 5d 41 5c 41 5d e9 50 5b 79 09 48 c7 c0 f4 ff ff ff eb 92 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <f3> 0f 1e fa 48 8b 34 24 65 48 8b 15 98 d6 9c 12 65 8b 05 a9 d6 9c
RSP: 0018:ffa00000048af960 EFLAGS: 00000246
RAX: 0000000000080000 RBX: 0000000000000000 RCX: ffa0000014af9000
RDX: 0000000000080000 RSI: ffffffff81db5b43 RDI: 0000000000000007
RBP: ff11000050a44000 R08: 0000000000000001 R09: 0000000000000001
R10: 0000000000000200 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000200 R14: 0000000000000000 R15: ff110000931fe008
htab_map_update_elem_in_place+0x4c9/0xa40 kernel/bpf/hashtab.c:1308
bpf_percpu_hash_update+0xc1/0x240 kernel/bpf/hashtab.c:2404
bpf_map_update_value+0xad8/0x1050 kernel/bpf/syscall.c:270
generic_map_update_batch+0x441/0x620 kernel/bpf/syscall.c:2029
bpf_map_do_batch+0x4ac/0x610 kernel/bpf/syscall.c:5617
__sys_bpf+0xf00/0x5390 kernel/bpf/syscall.c:6198
__do_sys_bpf kernel/bpf/syscall.c:6244 [inline]
__se_sys_bpf kernel/bpf/syscall.c:6242 [inline]
__x64_sys_bpf+0x78/0xc0 kernel/bpf/syscall.c:6242
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xcb/0xfa0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fab9cbadead
RSP: 002b:00007fab9da73f98 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007fab9cde5fa0 RCX: 00007fab9cbadead
RDX: 0000000000000038 RSI: 0000200000000340 RDI: 000000000000001a
RBP: 00007fab9cc47d9f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fab9cde5fa0 R15: 00007fab9da54000
</TASK>
rcu: rcu_preempt kthread starved for 12112 jiffies! g87725 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:R running task stack:28200 pid:16 tgid:16 ppid:2 task_flags:0x208040 flags:0x00080000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5325 [inline]
__schedule+0x1044/0x5bb0 kernel/sched/core.c:6929
__schedule_loop kernel/sched/core.c:7011 [inline]
schedule+0xe7/0x3a0 kernel/sched/core.c:7026
schedule_timeout+0x113/0x280 kernel/time/sleep_timeout.c:99
rcu_gp_fqs_loop+0x18c/0xa00 kernel/rcu/tree.c:2083
rcu_gp_kthread+0x26f/0x370 kernel/rcu/tree.c:2285
kthread+0x3d0/0x780 kernel/kthread.c:463
ret_from_fork+0x676/0x7d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 799 Comm: kworker/0:2 Not tainted 6.17.0-g39e9d5f63075 #1 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: events prog_array_map_clear_deferred
RIP: 0010:fib6_check_expired include/net/ip6_fib.h:271 [inline]
RIP: 0010:__find_rr_leaf+0x32c/0xdc0 net/ipv6/route.c:842
Code: d0 f7 4c 8d 63 64 4c 89 e0 48 c1 e8 03 0f b6 14 28 4c 89 e0 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 58 08 00 00 44 8b 73 64 <31> ff 45 89 f7 41 81 e7 00 00 40 00 44 89 fe e8 f0 4b d0 f7 45 85
RSP: 0018:ffa0000000007558 EFLAGS: 00000246
RAX: 0000000000000007 RBX: ff1100010de72400 RCX: ffffffff89ea6856
RDX: 0000000000000000 RSI: ffffffff89ea6864 RDI: 0000000000000004
RBP: dffffc0000000000 R08: 0000000000000001 R09: ffa0000000007728
R10: 0000000000000100 R11: 0000000000000000 R12: ff1100010de72464
R13: ffa0000000007840 R14: 0000000000000001 R15: ff11000107d7f4c8
FS: 0000000000000000(0000) GS:ff1100010d2a7000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000d6e000 CR3: 000000019245e000 CR4: 0000000000753ef0
DR0: 00000000000003ff DR1: 00000000000003ff DR2: 00000000000003ff
DR3: 00000000000003ff DR6: 00000000ffff0ff0 DR7: 0000000000000600
PKRU: 55555554
Call Trace:
<IRQ>
find_rr_leaf net/ipv6/route.c:889 [inline]
rt6_select net/ipv6/route.c:933 [inline]
fib6_table_lookup+0x591/0x9f0 net/ipv6/route.c:2233
ip6_pol_route+0x1d2/0x1230 net/ipv6/route.c:2269
pol_lookup_func include/net/ip6_fib.h:617 [inline]
fib6_rule_lookup+0x536/0x720 net/ipv6/fib6_rules.c:120
ip6_route_input_lookup net/ipv6/route.c:2338 [inline]
ip6_route_input+0x6b5/0xc70 net/ipv6/route.c:2641
ip6_rcv_finish_core.constprop.0+0x1a4/0x5d0 net/ipv6/ip6_input.c:66
ip6_rcv_finish net/ipv6/ip6_input.c:77 [inline]
NF_HOOK include/linux/netfilter.h:318 [inline]
NF_HOOK include/linux/netfilter.h:312 [inline]
ipv6_rcv+0x1eb/0x650 net/ipv6/ip6_input.c:311
__netif_receive_skb_one_core+0x12d/0x1e0 net/core/dev.c:6079
__netif_receive_skb+0x1d/0x170 net/core/dev.c:6192
process_backlog+0x310/0x1450 net/core/dev.c:6544
__napi_poll.constprop.0+0xb9/0x540 net/core/dev.c:7594
napi_poll net/core/dev.c:7657 [inline]
net_rx_action+0x92d/0xe00 net/core/dev.c:7784
handle_softirqs+0x1d4/0x870 kernel/softirq.c:622
__do_softirq kernel/softirq.c:656 [inline]
invoke_softirq kernel/softirq.c:496 [inline]
__irq_exit_rcu+0x109/0x170 kernel/softirq.c:723
irq_exit_rcu+0x9/0x30 kernel/softirq.c:739
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1052 [inline]
sysvec_apic_timer_interrupt+0xa8/0xc0 arch/x86/kernel/apic/apic.c:1052
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:arch_atomic64_try_cmpxchg arch/x86/include/asm/atomic64_64.h:101 [inline]
RIP: 0010:raw_atomic64_try_cmpxchg_acquire include/linux/atomic/atomic-arch-fallback.h:4296 [inline]
RIP: 0010:raw_atomic_long_try_cmpxchg_acquire include/linux/atomic/atomic-long.h:1482 [inline]
RIP: 0010:atomic_long_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:4458 [inline]
RIP: 0010:__mutex_trylock_common+0x107/0x260 kernel/locking/mutex.c:113
Code: be 08 00 00 00 4c 09 eb e8 b6 fd 8c 00 be 08 00 00 00 48 8d 7c 24 30 e8 a7 fd 8c 00 48 8b 44 24 30 48 89 04 24 f0 49 0f b1 1f <48> c7 c2 c0 00 67 9b 0f 85 e1 00 00 00 4d 39 f5 0f 84 e2 00 00 00
RSP: 0018:ffa00000052d7960 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ff11000024708000 RCX: ffffffff819644c9
RDX: fff3fc0000a5af33 RSI: 0000000000000008 RDI: ffa00000052d7990
RBP: 1ff4000000a5af2e R08: 0000000000000001 R09: fff3fc0000a5af32
R10: ffa00000052d7997 R11: 0000000000000000 R12: fffffbfff36ce018
R13: ff11000024708000 R14: ff11000024708000 R15: ff1100002362a018
__mutex_trylock kernel/locking/mutex.c:136 [inline]
__mutex_lock_common kernel/locking/mutex.c:601 [inline]
__mutex_lock+0x17f/0x1000 kernel/locking/mutex.c:760
__fd_array_map_delete_elem+0x129/0x310 kernel/bpf/arraymap.c:926
bpf_fd_array_map_clear kernel/bpf/arraymap.c:1003 [inline]
prog_array_map_clear_deferred+0x113/0x1c0 kernel/bpf/arraymap.c:1144
process_one_work+0x992/0x1b60 kernel/workqueue.c:3263
process_scheduled_works kernel/workqueue.c:3346 [inline]
worker_thread+0x67e/0xe90 kernel/workqueue.c:3427
kthread+0x3d0/0x780 kernel/kthread.c:463
ret_from_fork+0x676/0x7d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
clocksource: Long readout interval, skipping watchdog check: cs_nsec: 116501184056 wd_nsec: 116501184104
```
## Kernel Configuration Requirements for Reproduction
The vulnerability can be triggered with the kernel config in the attachment. Additionally, we provide the execution logs in Syzkaller format to facilitate further verification.
Alexei Starovoitov
>
> Our fuzzer discovered a task hung vulnerability in the BPF subsystem. The crash can be trigger on bpf-next(39e9d5f63075f4d54e3b59b8238478c32af92755).
> CPU: 0 UID: 0 PID: 9930 Comm: sshd Not tainted 6.17.0-g39e9d5f63075 #1 PREEMPT(full)