[v5.15] INFO: rcu detected stall in gc_worker

5 views
Skip to first unread message

syzbot

unread,
Aug 6, 2023, 5:00:58 PM8/6/23
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: 38d4ca22a528 Linux 5.15.124
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=11756635a80000
kernel config: https://syzkaller.appspot.com/x/.config?x=45eb463798d34fdb
dashboard link: https://syzkaller.appspot.com/bug?extid=c6ab59ecefe93f5cab03
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=146e0e79a80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17cdeb49a80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/0470c022da0e/disk-38d4ca22.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/56cf48a83529/vmlinux-38d4ca22.xz
kernel image: https://storage.googleapis.com/syzbot-assets/ac8ac64c33ec/bzImage-38d4ca22.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c6ab59...@syzkaller.appspotmail.com

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 0-...!: (2 ticks this GP) idle=c09/1/0x4000000000000000 softirq=4644/4644 fqs=105
(detected by 1, t=19429 jiffies, g=4117, q=140)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 3539 Comm: kworker/0:0 Not tainted 5.15.124-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/12/2023
Workqueue: events_power_efficient gc_worker
RIP: 0010:rcu_preempt_read_exit kernel/rcu/tree_plugin.h:380 [inline]
RIP: 0010:__rcu_read_unlock+0x33/0x100 kernel/rcu/tree_plugin.h:419
Code: 53 49 bc 00 00 00 00 00 fc ff df 65 4c 8b 35 94 e0 96 7e 4d 8d ae 3c 04 00 00 4c 89 eb 48 c1 eb 03 42 0f b6 04 23 84 c0 75 61 <41> 8b 6d 00 ff cd 42 0f b6 04 23 84 c0 75 69 41 89 6d 00 85 ed 75
RSP: 0018:ffffc90000007c88 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 1ffff1100f08543f RCX: ffff888078429dc0
RDX: 0000000000010000 RSI: ffffffff8ad86c20 RDI: ffffffff8ad86be0
RBP: ffff888018accae8 R08: ffffffff886b2270 R09: ffffed1003159932
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff88807842a1fc R14: ffff888078429dc0 R15: 00000067538a4b00
FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000600 CR3: 000000000c68e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
</NMI>
<IRQ>
rcu_read_unlock include/linux/rcupdate.h:728 [inline]
advance_sched+0x7cf/0x940 net/sched/sch_taprio.c:771
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x598/0xcf0 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:seqcount_lockdep_reader_access+0x1dc/0x220 include/linux/seqlock.h:106
Code: f9 4d 85 ed 75 16 e8 b3 85 05 f9 eb 15 e8 ac 85 05 f9 e8 37 9c a1 01 4d 85 ed 74 ea e8 9d 85 05 f9 fb 48 c7 04 24 0e 36 e0 45 <4b> c7 04 3c 00 00 00 00 66 43 c7 44 3c 09 00 00 43 c6 44 3c 0b 00
RSP: 0018:ffffc90002c4fa40 EFLAGS: 00000293
RAX: ffffffff887a7c63 RBX: 0000000000000000 RCX: ffff888078429dc0
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90002c4faf0 R08: ffffffff887a7c39 R09: fffffbfff1f7ac33
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: 0000000000000200 R14: 0000000000000046 R15: 1ffff92000589f48
nf_conntrack_get_ht include/net/netfilter/nf_conntrack.h:330 [inline]
gc_worker+0x30f/0x14b0 net/netfilter/nf_conntrack_core.c:1472
process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
kthread+0x3f6/0x4f0 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
</TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.529 msecs
rcu: rcu_preempt kthread timer wakeup didn't happen for 19204 jiffies! g4117 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: Possible timer handling issue on cpu=1 timer-softirq=24252
rcu: rcu_preempt kthread starved for 19205 jiffies! g4117 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:I stack:27000 pid: 15 ppid: 2 flags:0x00004000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5026 [inline]
__schedule+0x12c4/0x4590 kernel/sched/core.c:6372
schedule+0x11b/0x1f0 kernel/sched/core.c:6455
schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1884
rcu_gp_fqs_loop+0x2af/0xf70 kernel/rcu/tree.c:1959
rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2132
kthread+0x3f6/0x4f0 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
</TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.15.124-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/12/2023
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
rcu_check_gp_kthread_starvation+0x1d2/0x240 kernel/rcu/tree_stall.h:481
print_other_cpu_stall+0x137a/0x14d0 kernel/rcu/tree_stall.h:586
check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
rcu_pending kernel/rcu/tree.c:3911 [inline]
rcu_sched_clock_irq+0x94f/0x1770 kernel/rcu/tree.c:2606
update_process_times+0x196/0x200 kernel/time/timer.c:1788
tick_sched_handle kernel/time/tick-sched.c:237 [inline]
tick_sched_timer+0x22d/0x3c0 kernel/time/tick-sched.c:1441
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x55b/0xcf0 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:22 [inline]
RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline]
RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:132 [inline]
RIP: 0010:acpi_safe_halt drivers/acpi/processor_idle.c:110 [inline]
RIP: 0010:acpi_idle_do_entry+0x10f/0x340 drivers/acpi/processor_idle.c:570
Code: 75 5a f7 48 83 e3 08 0f 85 0a 01 00 00 4c 8d 74 24 20 e8 04 e8 60 f7 0f 1f 44 00 00 e8 8a 71 5a f7 0f 00 2d 63 da bc 00 fb f4 <4c> 89 f3 48 c1 eb 03 42 80 3c 3b 00 74 08 4c 89 f7 e8 bb ea a3 f7
RSP: 0018:ffffc90000d67b00 EFLAGS: 000002d3
RAX: ffffffff8a259076 RBX: 0000000000000000 RCX: ffff88813fe70000
RDX: 0000000000000000 RSI: ffffffff8a8b01e0 RDI: ffffffff8ad86c40
RBP: ffffc90000d67b90 R08: ffffffff818678a0 R09: ffffed1027fce001
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff920001acf60
R13: ffff8881468cd004 R14: ffffc90000d67b20 R15: dffffc0000000000
acpi_idle_enter+0x352/0x4f0 drivers/acpi/processor_idle.c:705
cpuidle_enter_state+0x521/0xef0 drivers/cpuidle/cpuidle.c:237
cpuidle_enter+0x59/0x90 drivers/cpuidle/cpuidle.c:351
call_cpuidle kernel/sched/idle.c:158 [inline]
cpuidle_idle_call kernel/sched/idle.c:239 [inline]
do_idle+0x3e4/0x670 kernel/sched/idle.c:306
cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:403
start_secondary+0x371/0x500 arch/x86/kernel/smpboot.c:281
secondary_startup_64_no_verify+0xb1/0xbb
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to change bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup
Reply all
Reply to author
Forward
0 new messages