[v5.15] INFO: rcu detected stall in sys_timerfd_settime

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 7349e40704a0 Linux 5.15.116
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12a1ce8d280000
kernel config: https://syzkaller.appspot.com/x/.config?x=831c3122ac9c9145
dashboard link: https://syzkaller.appspot.com/bug?extid=42cab3ff8100a09b7d3e
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/8c03c3ad4501/disk-7349e407.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/350c3d79bc87/vmlinux-7349e407.xz
kernel image: https://storage.googleapis.com/syzbot-assets/73a4ed3d5438/bzImage-7349e407.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+42cab3...@syzkaller.appspotmail.com

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 0-...!: (1 GPs behind) idle=947/1/0x4000000000000000 softirq=133666/133670 fqs=418
(detected by 1, t=10502 jiffies, g=214085, q=180)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 31798 Comm: syz-executor.2 Not tainted 5.15.116-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
RIP: 0010:validate_chain+0x1/0x58b0 kernel/locking/lockdep.c:3743
Code: e3 8d 80 e1 07 80 c1 03 38 c1 7c 92 48 c7 c7 08 f7 e3 8d e8 b1 2e 66 00 eb 84 e8 ca c2 b7 08 66 2e 0f 1f 84 00 00 00 00 00 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 e4 e0 48 81 ec 80 02 00
RSP: 0018:ffffc90000007a50 EFLAGS: 00000086
RAX: 1ffffffff1f31c90 RBX: ffffffff8f98e480 RCX: eaf279c302216aba
RDX: 0000000000000001 RSI: ffff88807b1ee430 RDI: ffff88807b1ed940
RBP: eaf279c302216aba R08: dffffc0000000000 R09: fffffbfff1f79665
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: ffff88807b1ee428 R14: ffff88807b1ed940 R15: ffff88807b1ee450
FS: 00007f02b9ec1700(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000280 CR3: 000000002ed99000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
</NMI>
<IRQ>
__lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5011
lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5622
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:363 [inline]
advance_sched+0x47/0x940 net/sched/sch_taprio.c:716
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x598/0xcf0 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:202
Code: 31 9f f6 ff 90 53 48 89 fb 48 83 c7 18 48 8b 74 24 08 e8 2e 4a 3e f7 48 89 df e8 c6 9c 3f f7 e8 c1 20 62 f7 fb bf 01 00 00 00 <e8> 46 da 32 f7 65 8b 05 47 26 de 75 85 c0 74 02 5b c3 e8 54 3d dc
RSP: 0018:ffffc90003d7fd20 EFLAGS: 00000282
RAX: 8a0f705152339b00 RBX: ffff88801db38c88 RCX: ffffffff913c3003
RDX: dffffc0000000000 RSI: ffffffff8a8afc60 RDI: 0000000000000001
RBP: 00000000ffffff83 R08: ffffffff81866a60 R09: ffffed1003b67192
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff88801db38c00
R13: 7fffffffffffffff R14: ffff88801db38c80 R15: 0000000000000003
spin_unlock_irq include/linux/spinlock.h:413 [inline]
do_timerfd_settime+0xd1a/0x1000 fs/timerfd.c:521
__do_sys_timerfd_settime fs/timerfd.c:567 [inline]
__se_sys_timerfd_settime fs/timerfd.c:558 [inline]
__x64_sys_timerfd_settime+0x16b/0x220 fs/timerfd.c:558
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x61/0xcb
RIP: 0033:0x7f02bb94f199
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f02b9ec1168 EFLAGS: 00000246 ORIG_RAX: 000000000000011e
RAX: ffffffffffffffda RBX: 00007f02bba6ef80 RCX: 00007f02bb94f199
RDX: 0000000020000100 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 00007f02bb9aaca1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffd14c4c32f R14: 00007f02b9ec1300 R15: 0000000000022000
</TASK>
rcu: rcu_preempt kthread starved for 9666 jiffies! g214085 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:R running task stack:26616 pid: 15 ppid: 2 flags:0x00004000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5026 [inline]
__schedule+0x12c4/0x4590 kernel/sched/core.c:6372
schedule+0x11b/0x1f0 kernel/sched/core.c:6455
schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1884
rcu_gp_fqs_loop+0x2af/0xf70 kernel/rcu/tree.c:1959
rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2132
kthread+0x3f6/0x4f0 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
</TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 3740 Comm: kworker/1:10 Not tainted 5.15.116-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
Workqueue: events bpf_map_free_deferred
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
rcu_check_gp_kthread_starvation+0x1d2/0x240 kernel/rcu/tree_stall.h:481
print_other_cpu_stall+0x137a/0x14d0 kernel/rcu/tree_stall.h:586
check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
rcu_pending kernel/rcu/tree.c:3888 [inline]
rcu_sched_clock_irq+0x94f/0x1770 kernel/rcu/tree.c:2606
update_process_times+0x196/0x200 kernel/time/timer.c:1788
tick_sched_handle kernel/time/tick-sched.c:226 [inline]
tick_sched_timer+0x22d/0x3c0 kernel/time/tick-sched.c:1430
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x55b/0xcf0 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
__sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:193
Code: 1f 84 00 00 00 00 00 0f 1f 00 53 48 89 fb e8 17 00 00 00 48 8b 3d 58 cc 63 0c 48 89 de 5b e9 07 84 48 00 cc cc cc cc cc cc cc <48> 8b 04 24 65 48 8b 0d 64 76 82 7e 65 8b 15 65 76 82 7e f7 c2 00
RSP: 0018:ffffc9000486f9f8 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 1ffff9200090df49 RCX: ffff88807925d940
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc9000486faf0 R08: ffffffff81743460 R09: ffffed10173474e9
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: 0000000000000001 R14: ffffc9000486fa48 R15: 0000000000000000
csd_lock_wait kernel/smp.c:440 [inline]
smp_call_function_single+0x2ae/0x530 kernel/smp.c:758
rcu_barrier+0x262/0x4e0 kernel/rcu/tree.c:4034
htab_map_free+0x25/0x5e0 kernel/bpf/hashtab.c:1466
process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2307
worker_thread+0xaca/0x1280 kernel/workqueue.c:2454
kthread+0x3f6/0x4f0 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to change bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: e84a4e368abe Linux 6.1.35
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=1306f1eb280000
kernel config: https://syzkaller.appspot.com/x/.config?x=a69b5c9de715622a
dashboard link: https://syzkaller.appspot.com/bug?extid=f03b083161fc94d341e6

compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:

disk image: https://storage.googleapis.com/syzbot-assets/3c7fedd1a86d/disk-e84a4e36.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/8b34c6296ed7/vmlinux-e84a4e36.xz
kernel image: https://storage.googleapis.com/syzbot-assets/a88164798cc2/bzImage-e84a4e36.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by: syzbot+f03b08...@syzkaller.appspotmail.com

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

rcu: 1-...!: (1 GPs behind) idle=3f5c/1/0x4000000000000000 softirq=58922/58923 fqs=0
(detected by 0, t=10505 jiffies, g=82253, q=26 ncpus=2)
Sending NMI from CPU 0 to CPUs 1:

NMI backtrace for cpu 1

CPU: 1 PID: 17916 Comm: syz-executor.2 Not tainted 6.1.35-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023
RIP: 0010:__lock_release kernel/locking/lockdep.c:5327 [inline]
RIP: 0010:lock_release+0x1db/0xa20 kernel/locking/lockdep.c:5689
Code: 42 0f b6 04 3b 84 c0 0f 85 6e 05 00 00 c7 84 24 80 00 00 00 01 00 00 00 48 c7 c0 24 5f 53 8e 48 c1 e8 03 42 0f b6 04 38 84 c0 <4c> 8b 74 24 10 0f 85 6e 05 00 00 83 3d d7 64 e9 0c 00 0f 84 c4 03
RSP: 0018:ffffc900001e0b60 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 1ffff9200003c17c RCX: ffffc900001e0b03
RDX: 0000000000000000 RSI: ffffffff8aebde80 RDI: ffffffff8b3ccb60
RBP: ffffc900001e0c90 R08: dffffc0000000000 R09: fffffbfff1ca654e
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff9200003c178
R13: 0000000000000046 R14: ffffc900001e0c10 R15: dffffc0000000000
FS: 00007fd7a8926700(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 0000000020000280 CR3: 0000000021f69000 CR4: 00000000003506e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
</NMI>
<IRQ>

__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:149 [inline]
_raw_spin_unlock_irqrestore+0x75/0x130 kernel/locking/spinlock.c:194
debug_hrtimer_deactivate kernel/time/hrtimer.c:425 [inline]
debug_deactivate+0x1d/0x280 kernel/time/hrtimer.c:481
__run_hrtimer kernel/time/hrtimer.c:1653 [inline]
__hrtimer_run_queues+0x334/0xe50 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
__sysvec_apic_timer_interrupt+0x156/0x580 arch/x86/kernel/apic/apic.c:1112
sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1106
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]

RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:202

Code: 41 35 f6 ff 90 53 48 89 fb 48 83 c7 18 48 8b 74 24 08 e8 ae a7 dc f6 48 89 df e8 06 e4 dd f6 e8 d1 37 03 f7 fb bf 01 00 00 00 <e8> a6 a2 d0 f6 65 8b 05 77 1e 75 75 85 c0 74 02 5b c3 e8 74 3c 73
RSP: 0018:ffffc9000cb8fd20 EFLAGS: 00000282
RAX: b01ab34b3276e500 RBX: ffff888085b4d488 RCX: ffffffff91a83103
RDX: dffffc0000000000 RSI: ffffffff8aebd1a0 RDI: 0000000000000001
RBP: 00000000ffffff83 R08: dffffc0000000000 R09: ffffed1010b69a92
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff888085b4d400
R13: 7fffffffffffffff R14: ffff888085b4d480 R15: 0000000000000003
spin_unlock_irq include/linux/spinlock.h:400 [inline]

do_timerfd_settime+0xd1a/0x1000 fs/timerfd.c:521
__do_sys_timerfd_settime fs/timerfd.c:567 [inline]
__se_sys_timerfd_settime fs/timerfd.c:558 [inline]
__x64_sys_timerfd_settime+0x16b/0x220 fs/timerfd.c:558
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80

entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fd7a7c8c389

Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48

RSP: 002b:00007fd7a8926168 EFLAGS: 00000246 ORIG_RAX: 000000000000011e
RAX: ffffffffffffffda RBX: 00007fd7a7dabf80 RCX: 00007fd7a7c8c389

RDX: 0000000020000100 RSI: 0000000000000003 RDI: 0000000000000003

RBP: 00007fd7a7cd7493 R08: 0000000000000000 R09: 0000000000000000

R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000

R13: 00007ffc5ff0bc1f R14: 00007fd7a8926300 R15: 0000000000022000
</TASK>
rcu: rcu_preempt kthread starved for 10505 jiffies! g82253 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0

rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:

task:rcu_preempt state:R running task stack:26712 pid:16 ppid:2 flags:0x00004000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5241 [inline]
__schedule+0x132c/0x4330 kernel/sched/core.c:6554
schedule+0xbf/0x180 kernel/sched/core.c:6630
schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1935
rcu_gp_fqs_loop+0x2c2/0x1010 kernel/rcu/tree.c:1661
rcu_gp_kthread+0xa3/0x3a0 kernel/rcu/tree.c:1860
kthread+0x26e/0x300 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306

</TASK>
rcu: Stack dump where RCU GP kthread last ran:

CPU: 0 PID: 57 Comm: kworker/u4:4 Not tainted 6.1.35-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:413 [inline]
RIP: 0010:smp_call_function_many_cond+0x1f62/0x33d0 kernel/smp.c:987
Code: 2f 44 89 ee 83 e6 01 31 ff e8 6a 0c 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 0a e8 f5 08 0b 00 e9 1b ff ff ff f3 90 <42> 0f b6 04 2b 84 c0 75 14 41 f7 07 01 00 00 00 0f 84 fe fe ff ff
RSP: 0018:ffffc900015875a0 EFLAGS: 00000293
RAX: ffffffff817ecdbd RBX: 1ffff11017328021 RCX: ffff8880186dbb80

RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000

RBP: ffffc90001587980 R08: ffffffff817ecd86 R09: fffffbfff2051645
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000800000000
R13: dffffc0000000000 R14: 0000000000000001 R15: ffff8880b9940108
FS: 0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007fd12cf33718 CR3: 000000000cc8e000 CR4: 00000000003506f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

<IRQ>
</IRQ>
<TASK>
on_each_cpu_cond_mask+0x3b/0x80 kernel/smp.c:1155
on_each_cpu include/linux/smp.h:71 [inline]
text_poke_sync arch/x86/kernel/alternative.c:1316 [inline]
text_poke_bp_batch+0x2bb/0x940 arch/x86/kernel/alternative.c:1516
text_poke_flush arch/x86/kernel/alternative.c:1707 [inline]
text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1714
arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x12e/0x250 kernel/jump_label.c:177
static_key_enable+0x16/0x20 kernel/jump_label.c:190
toggle_allocation_gate+0xbf/0x480 mm/kfence/core.c:804
process_one_work+0x8aa/0x11f0 kernel/workqueue.c:2289
worker_thread+0xa5f/0x1210 kernel/workqueue.c:2436
kthread+0x26e/0x300 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.

[syzbot]