INFO: rcu detected stall in ieee80211_tasklet_handler

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 148842c9 Merge tag 'x86-apic-2020-12-14' of git://git.kern..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=10f86c93500000
kernel config: https://syzkaller.appspot.com/x/.config?x=3e26d08f66b19a15
dashboard link: https://syzkaller.appspot.com/bug?extid=7bb955045fc0840decd3
compiler: gcc (GCC) 10.1.0-syz 20200507
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=110c0613500000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+7bb955...@syzkaller.appspotmail.com

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 1-....: (10498 ticks this GP) idle=efa/1/0x4000000000000000 softirq=19185/19191 fqs=5250
(t=10502 jiffies g=23877 q=34)
NMI backtrace for cpu 1
CPU: 1 PID: 10164 Comm: syz-executor.0 Not tainted 5.10.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x107/0x163 lib/dump_stack.c:120
nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x1e3/0x21e kernel/rcu/tree_stall.h:337
print_cpu_stall kernel/rcu/tree_stall.h:569 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:643 [inline]
rcu_pending kernel/rcu/tree.c:3751 [inline]
rcu_sched_clock_irq.cold+0x472/0xec3 kernel/rcu/tree.c:2580
update_process_times+0x16d/0x200 kernel/time/timer.c:1782
tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1376
__run_hrtimer kernel/time/hrtimer.c:1519 [inline]
__hrtimer_run_queues+0x1ce/0xea0 kernel/time/hrtimer.c:1583
hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1102
run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:91 [inline]
sysvec_apic_timer_interrupt+0x48/0x100 arch/x86/kernel/apic/apic.c:1096
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:628
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/irqflags.h:85 [inline]
RIP: 0010:lock_acquire kernel/locking/lockdep.c:5440 [inline]
RIP: 0010:lock_acquire+0x2c7/0x740 kernel/locking/lockdep.c:5402
Code: 48 c7 c7 20 68 4b 89 48 83 c4 20 e8 43 fc 8f 07 b8 ff ff ff ff 65 0f c1 05 26 6f ab 7e 83 f8 01 0f 85 36 03 00 00 ff 34 24 9d <e9> 3a fe ff ff 65 ff 05 8d 5d ab 7e 48 8b 05 86 51 82 0b e8 61 f1
RSP: 0018:ffffc90000d90810 EFLAGS: 00000246
RAX: 0000000000000001 RBX: 1ffff920001b2104 RCX: 0000000000000001
RDX: 1ffff1100242cb38 RSI: 0000000000000302 RDI: 0000000000000000
RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff8ebb083f
R10: fffffbfff1d76107 R11: 0000000000000001 R12: 0000000000000000
R13: ffff8880275114b8 R14: 0000000000000000 R15: 0000000000000000
__raw_spin_lock_bh include/linux/spinlock_api_smp.h:135 [inline]
_raw_spin_lock_bh+0x2f/0x40 kernel/locking/spinlock.c:175
spin_lock_bh include/linux/spinlock.h:359 [inline]
ieee80211_rx_handlers+0xd7/0xae50 net/mac80211/rx.c:3769
ieee80211_invoke_rx_handlers net/mac80211/rx.c:3834 [inline]
ieee80211_prepare_and_rx_handle+0x2482/0x6360 net/mac80211/rx.c:4518
__ieee80211_rx_handle_packet net/mac80211/rx.c:4579 [inline]
ieee80211_rx_list+0xdf3/0x23d0 net/mac80211/rx.c:4759
ieee80211_rx_napi+0xf7/0x3d0 net/mac80211/rx.c:4780
ieee80211_rx include/net/mac80211.h:4502 [inline]
ieee80211_tasklet_handler+0xd3/0x130 net/mac80211/main.c:235
tasklet_action_common.constprop.0+0x22f/0x2d0 kernel/softirq.c:560
__do_softirq+0x2a0/0x9f6 kernel/softirq.c:298
asm_call_irq_on_stack+0xf/0x20
</IRQ>
__run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
do_softirq_own_stack+0xaa/0xd0 arch/x86/kernel/irq_64.c:77
invoke_softirq kernel/softirq.c:393 [inline]
__irq_exit_rcu kernel/softirq.c:423 [inline]
irq_exit_rcu+0x132/0x200 kernel/softirq.c:435
sysvec_apic_timer_interrupt+0x4d/0x100 arch/x86/kernel/apic/apic.c:1096
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:628
RIP: 0010:__tlb_remove_page_size+0x38/0x420 mm/mmu_gather.c:70
Code: 89 fb e8 1b 5b cb ff 48 8d 7b 18 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 5b 03 00 00 48 8b 6b 18 <31> ff 48 89 ee e8 5e 53 cb ff 48 85 ed 0f 84 8b 01 00 00 e8 e0 5a
RSP: 0018:ffffc9000a15f760 EFLAGS: 00000246
RAX: dffffc0000000000 RBX: ffffc9000a15fa60 RCX: ffffffff81a1cc28
RDX: 1ffff9200142bf4f RSI: ffffffff81a52265 RDI: ffffc9000a15fa78
RBP: ffffffffffffffff R08: 0000000000000000 R09: ffffea0000cc0047
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8880139a30e8 R14: ffffea0000cc0740 R15: 000000000101e000
__tlb_remove_page include/asm-generic/tlb.h:438 [inline]
zap_pte_range mm/memory.c:1256 [inline]
zap_pmd_range mm/memory.c:1357 [inline]
zap_pud_range mm/memory.c:1386 [inline]
zap_p4d_range mm/memory.c:1407 [inline]
unmap_page_range+0xea7/0x2640 mm/memory.c:1428
unmap_single_vma+0x198/0x300 mm/memory.c:1473
unmap_vmas+0x168/0x2e0 mm/memory.c:1505
exit_mmap+0x2b1/0x530 mm/mmap.c:3220
__mmput+0x122/0x470 kernel/fork.c:1081
mmput+0x53/0x60 kernel/fork.c:1102
exit_mm kernel/exit.c:500 [inline]
do_exit+0xaa9/0x29f0 kernel/exit.c:810
do_group_exit+0x125/0x310 kernel/exit.c:920
get_signal+0x42a/0x1fb0 kernel/signal.c:2792
arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
handle_signal_work kernel/entry/common.c:147 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
exit_to_user_mode_prepare+0x124/0x200 kernel/entry/common.c:201
__syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:302
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x460fba
Code: Unable to access opcode bytes at RIP 0x460f90.
RSP: 002b:00007f178679cc48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e4
RAX: 0000000000000000 RBX: 00000000004cd6bc RCX: 0000000000460fba
RDX: 0000000000503b20 RSI: 00007f178679cc50 RDI: 0000000000000001
RBP: 000000000119bf80 R08: 000000000000000e R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 000000000119bf8c
R13: 00007fff5143dfdf R14: 00007f178679d9c0 R15: 000000000119bf8c


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

[Hillf Danton]

Sat, 19 Dec 2020 07:14:11 -0800

Because it is unlikely to build the giant pyramid in a week, add a
budget for the softint handler and schedule tasklet if we fail to
finish work within the budget.

--- a/net/mac80211/main.c
+++ b/net/mac80211/main.c
@@ -223,10 +223,18 @@ u32 ieee80211_reset_erp_info(struct ieee
static void ieee80211_tasklet_handler(unsigned long data)
{
struct ieee80211_local *local = (struct ieee80211_local *) data;
- struct sk_buff *skb;
+ int i;
+
+ for (i = 0; i < 64; i++) {
+ struct sk_buff *skb;
+
+ skb = skb_dequeue(&local->skb_queue);
+ if (!skb) {
+ skb = skb_dequeue(&local->skb_queue_unreliable);
+ if (!skb)
+ return;
+ }

- while ((skb = skb_dequeue(&local->skb_queue)) ||
- (skb = skb_dequeue(&local->skb_queue_unreliable))) {
switch (skb->pkt_type) {
case IEEE80211_RX_MSG:
/* Clear skb->pkt_type in order to not confuse kernel
@@ -245,6 +253,8 @@ static void ieee80211_tasklet_handler(un
break;
}
}
+
+ tasklet_schedule(&local->tasklet);
}

static void ieee80211_restart_work(struct work_struct *work)

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 3516bd72 Merge tag 's390-5.11-3' of git://git.kernel.org/p..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=13e9d077500000
kernel config: https://syzkaller.appspot.com/x/.config?x=2455d075a1c4afa8

dashboard link: https://syzkaller.appspot.com/bug?extid=7bb955045fc0840decd3
compiler: gcc (GCC) 10.1.0-syz 20200507

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=114c140b500000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=146750eb500000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+7bb955...@syzkaller.appspotmail.com

rcu: INFO: rcu_preempt self-detected stall on CPU

rcu: 1-....: (8387 ticks this GP) idle=6c6/1/0x4000000000000000 softirq=9854/9855 fqs=5094
(t=10501 jiffies g=9217 q=198)

NMI backtrace for cpu 1

CPU: 1 PID: 8914 Comm: syz-executor318 Not tainted 5.11.0-rc1-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x107/0x163 lib/dump_stack.c:120
nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x1e3/0x21e kernel/rcu/tree_stall.h:337
print_cpu_stall kernel/rcu/tree_stall.h:569 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:643 [inline]
rcu_pending kernel/rcu/tree.c:3751 [inline]
rcu_sched_clock_irq.cold+0x472/0xec3 kernel/rcu/tree.c:2580
update_process_times+0x16d/0x200 kernel/time/timer.c:1782
tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226

tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369

__run_hrtimer kernel/time/hrtimer.c:1519 [inline]
__hrtimer_run_queues+0x1ce/0xea0 kernel/time/hrtimer.c:1583
hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1102
run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:91 [inline]
sysvec_apic_timer_interrupt+0x48/0x100 arch/x86/kernel/apic/apic.c:1096
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:628

RIP: 0010:queue_work_on+0x83/0xd0 kernel/workqueue.c:1530
Code: 31 ff 89 ee e8 ae dd 28 00 40 84 ed 74 46 e8 24 d7 28 00 31 ff 48 89 de e8 0a df 28 00 48 85 db 75 26 e8 10 d7 28 00 41 56 9d <48> 83 c4 08 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 f6 d6 28
RSP: 0018:ffffc90000d908a0 EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000100
RDX: ffff888023705340 RSI: ffffffff81499e80 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000001 R09: ffffffff8ebda887
R10: fffffbfff1d7b510 R11: 0000000000000000 R12: ffff88801bc955b0
R13: ffff88801ca14000 R14: 0000000000000246 R15: 0000000000000001
queue_work include/linux/workqueue.h:507 [inline]
ieee80211_queue_work net/mac80211/util.c:932 [inline]
ieee80211_queue_work+0x129/0x160 net/mac80211/util.c:925
ieee80211_rx_h_mgmt net/mac80211/rx.c:3627 [inline]
ieee80211_rx_handlers+0x3b86/0xae50 net/mac80211/rx.c:3793
ieee80211_invoke_rx_handlers net/mac80211/rx.c:3823 [inline]
ieee80211_prepare_and_rx_handle+0x2414/0x61c0 net/mac80211/rx.c:4503
__ieee80211_rx_handle_packet net/mac80211/rx.c:4564 [inline]
ieee80211_rx_list+0x1077/0x2430 net/mac80211/rx.c:4744
ieee80211_rx_napi+0xf7/0x3d0 net/mac80211/rx.c:4767
ieee80211_rx include/net/mac80211.h:4507 [inline]
ieee80211_tasklet_handler+0xd4/0x130 net/mac80211/main.c:235
tasklet_action_common.constprop.0+0x1d7/0x2d0 kernel/softirq.c:555
__do_softirq+0x2a5/0x9f7 kernel/softirq.c:343

asm_call_irq_on_stack+0xf/0x20
</IRQ>
__run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
do_softirq_own_stack+0xaa/0xd0 arch/x86/kernel/irq_64.c:77

invoke_softirq kernel/softirq.c:226 [inline]
__irq_exit_rcu kernel/softirq.c:420 [inline]
irq_exit_rcu+0x134/0x200 kernel/softirq.c:432

sysvec_apic_timer_interrupt+0x4d/0x100 arch/x86/kernel/apic/apic.c:1096
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:628

RIP: 0010:check_kcov_mode kernel/kcov.c:163 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x7/0x60 kernel/kcov.c:197
Code: 0f bd c8 49 8b 14 24 48 63 c9 e9 66 ff ff ff 4c 01 ca 49 89 13 e9 00 fd ff ff 66 0f 1f 84 00 00 00 00 00 65 8b 05 29 7a 8f 7e <89> c1 48 8b 34 24 81 e1 00 01 00 00 65 48 8b 14 25 00 f0 01 00 a9
RSP: 0018:ffffc90001e97768 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff888023705340
RDX: 0000000000000000 RSI: ffff888023705340 RDI: 0000000000000003
RBP: ffff88802f464000 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff81a441c3 R11: 0000000000000000 R12: 000000000000008d
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000424000
__preempt_count_add arch/x86/include/asm/preempt.h:79 [inline]
rcu_read_lock_sched_notrace include/linux/rcupdate.h:758 [inline]
trace_rss_stat+0x180/0x290 include/trace/events/kmem.h:338
mm_trace_rss_stat mm/memory.c:161 [inline]
add_mm_counter include/linux/mm.h:1895 [inline]
add_mm_rss_vec mm/memory.c:491 [inline]
zap_pte_range mm/memory.c:1314 [inline]
zap_pmd_range mm/memory.c:1368 [inline]
zap_pud_range mm/memory.c:1397 [inline]
zap_p4d_range mm/memory.c:1418 [inline]
unmap_page_range+0x100e/0x2640 mm/memory.c:1439
unmap_single_vma+0x198/0x300 mm/memory.c:1484
unmap_vmas+0x168/0x2e0 mm/memory.c:1516
exit_mmap+0x2b1/0x5a0 mm/mmap.c:3220
__mmput+0x122/0x470 kernel/fork.c:1083
mmput+0x53/0x60 kernel/fork.c:1104
exit_mm kernel/exit.c:501 [inline]
do_exit+0xb6a/0x2ae0 kernel/exit.c:812
do_group_exit+0x125/0x310 kernel/exit.c:922
get_signal+0x3ec/0x2010 kernel/signal.c:2770

arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
handle_signal_work kernel/entry/common.c:147 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]

exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:201

__syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:302
entry_SYSCALL_64_after_hwframe+0x44/0xa9

RIP: 0033:0x44c2b9
Code: Unable to access opcode bytes at RIP 0x44c28f.
RSP: 002b:00007f9025121ce8 EFLAGS: 00000246 ORIG_RAX: 0000000000000048
RAX: 0000000000000000 RBX: 00000000006e8a08 RCX: 000000000044c2b9
RDX: 0000000000042000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: 00000000006e8a00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006e8a0c
R13: 00007ffc7d12f4ff R14: 00007f90251229c0 R15: 20c49ba5e353f7cf

[syzbot]

syzbot suspects this issue was fixed by commit:

commit 313bbd1990b6ddfdaa7da098d0c56b098a833572
Author: Johannes Berg <johann...@intel.com>
Date: Wed Sep 15 09:29:37 2021 +0000

mac80211-hwsim: fix late beacon hrtimer handling

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=151766bab00000
start commit: 835d31d319d9 Merge tag 'media/v5.15-1' of git://git.kernel..
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=9c32e23fada3a0e4
dashboard link: https://syzkaller.appspot.com/bug?extid=7bb955045fc0840decd3
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15e08125300000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14b17dde300000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: mac80211-hwsim: fix late beacon hrtimer handling

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Dmitry Vyukov]

Looks plausible: