[v5.15] INFO: task hung in perf_pending_task_sync

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: fa93fa65db6e Linux 5.15.165
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=1156084d980000
kernel config: https://syzkaller.appspot.com/x/.config?x=d027d7ba4e9e9586
dashboard link: https://syzkaller.appspot.com/bug?extid=89a27bae950fcb3755f8
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/d8cb886910d9/disk-fa93fa65.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/451672ac83e2/vmlinux-fa93fa65.xz
kernel image: https://storage.googleapis.com/syzbot-assets/a52e4e839226/bzImage-fa93fa65.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+89a27b...@syzkaller.appspotmail.com

INFO: task syz.1.201:4224 blocked for more than 143 seconds.
Not tainted 5.15.165-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.1.201 state:D stack:26208 pid: 4224 ppid: 3569 flags:0x00004002
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5027 [inline]
__schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
schedule+0x11b/0x1f0 kernel/sched/core.c:6456
perf_pending_task_sync+0x13c/0x1c0 kernel/events/core.c:5122
_free_event+0x34/0xe60 kernel/events/core.c:5128
put_event kernel/events/core.c:5253 [inline]
perf_event_release_kernel+0x873/0x900 kernel/events/core.c:5378
perf_release+0x37/0x40 kernel/events/core.c:5388
__fput+0x3fe/0x8e0 fs/file_table.c:280
task_work_run+0x129/0x1a0 kernel/task_work.c:188
exit_task_work include/linux/task_work.h:33 [inline]
do_exit+0x6a3/0x2480 kernel/exit.c:874
do_group_exit+0x144/0x310 kernel/exit.c:996
__do_sys_exit_group kernel/exit.c:1007 [inline]
__se_sys_exit_group kernel/exit.c:1005 [inline]
__x64_sys_exit_group+0x3b/0x40 kernel/exit.c:1005
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f478d6dfef9
RSP: 002b:00007f478bb3af48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007f478d73e900 RCX: 00007f478d6dfef9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: 000000000000000e R08: 00007f478bb38ce6 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000000 R14: 00007f478d87c058 R15: 00007ffe4601e878
</TASK>

Showing all locks held in the system:
1 lock held by khungtaskd/27:
#0: ffffffff8c91fbe0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30
2 locks held by getty/3330:
#0: ffff88814b791098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x21/0x70 drivers/tty/tty_ldisc.c:252
#1: ffffc9000229b2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6af/0x1db0 drivers/tty/n_tty.c:2158
2 locks held by kworker/1:4/3615:
#0: ffff888011c72138 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2283
#1: ffffc90003077d20 ((work_completion)(&rew.rew_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2285
3 locks held by kworker/u4:7/4400:
2 locks held by syz.4.1878/8697:
#0: ffffffff8c96d428 (event_mutex){+.+.}-{3:3}, at: perf_trace_destroy+0x2a/0x140 kernel/trace/trace_event_perf.c:239
#1: ffffffff8c9241a8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:290 [inline]
#1: ffffffff8c9241a8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x280/0x740 kernel/rcu/tree_exp.h:845
2 locks held by syz.1.1882/8716:
#0: ffffffff914b9b18 (&pmus_srcu){....}-{0:0}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312
#1: ffffffff8c96d428 (event_mutex){+.+.}-{3:3}, at: perf_trace_init+0x4f/0x2d0 kernel/trace/trace_event_perf.c:221
2 locks held by syz.0.1884/8717:
#0: ffffffff914b9b18 (&pmus_srcu){....}-{0:0}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312
#1: ffffffff8c96d428 (event_mutex){+.+.}-{3:3}, at: perf_trace_init+0x4f/0x2d0 kernel/trace/trace_event_perf.c:221

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 27 Comm: khungtaskd Not tainted 5.15.165-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
check_hung_uninterruptible_tasks kernel/hung_task.c:210 [inline]
watchdog+0xe72/0xeb0 kernel/hung_task.c:295
kthread+0x3f6/0x4f0 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
</TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 3010 Comm: syslogd Not tainted 5.15.165-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
RIP: 0010:arch_atomic_read arch/x86/include/asm/atomic.h:29 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:331 [inline]
RIP: 0010:rcu_is_watching+0x5a/0xa0 kernel/rcu/tree.c:1123
Code: 03 42 80 3c 38 00 74 08 4c 89 f7 e8 60 11 5f 00 48 c7 c3 c8 b2 03 00 49 03 1e 48 89 d8 48 c1 e8 03 42 0f b6 04 38 84 c0 75 1a <8b> 03 65 ff 0d 95 56 97 7e 74 08 24 01 5b 41 5e 41 5f c3 e8 76 73
RSP: 0018:ffffc900022e7d20 EFLAGS: 00000046
RAX: 0000000000000000 RBX: ffff8880b9a3b2c8 RCX: ffffffff8162d3b8
RDX: 0000000000000000 RSI: ffffffff8ad8f7a0 RDI: ffffffff8ad8f760
RBP: ffffc900022e7e68 R08: dffffc0000000000 R09: fffffbfff1bd2a56
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff9200045cfb4
R13: ffffffff8a3360ec R14: ffffffff8c3b3800 R15: dffffc0000000000
FS: 00007f1b297fb380(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f46b0caa338 CR3: 000000007e8d4000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
<NMI>
</NMI>
<TASK>
trace_lock_release include/trace/events/lock.h:58 [inline]
lock_release+0xb9/0x9a0 kernel/locking/lockdep.c:5634
do_write_seqcount_end include/linux/seqlock.h:565 [inline]
vtime_user_exit+0x2d1/0x400 kernel/sched/cputime.c:732
__context_tracking_exit+0x4c/0x80 kernel/context_tracking.c:160
user_exit_irqoff include/linux/context_tracking.h:47 [inline]
__enter_from_user_mode kernel/entry/common.c:22 [inline]
syscall_enter_from_user_mode+0x1bf/0x240 kernel/entry/common.c:110
do_syscall_64+0x1e/0xb0 arch/x86/entry/common.c:76
entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f1b2994fbf2
Code: 89 c7 48 89 44 24 08 e8 7b 34 fa ff 48 8b 44 24 08 48 83 c4 28 c3 c3 64 8b 04 25 18 00 00 00 85 c0 75 20 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 76 6f 48 8b 15 07 a2 0d 00 f7 d8 64 89 02 48 83
RSP: 002b:00007fff44d03a68 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f1b2994fbf2
RDX: 0000000000000065 RSI: 00005596c2a92c50 RDI: 0000000000000003
RBP: 00005596c2a92c50 R08: 0000000000000001 R09: 0000000000000000
R10: 00007f1b29aee3a3 R11: 0000000000000246 R12: 0000000000000065
R13: 00007f1b297fb300 R14: 0000000000000006 R15: 00005596c2a92a60
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 3a5928702e71 Linux 5.15.167
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=1523b49f980000
kernel config: https://syzkaller.appspot.com/x/.config?x=4d3fd5cca89ae935

dashboard link: https://syzkaller.appspot.com/bug?extid=89a27bae950fcb3755f8
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1526e7c7980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17ae4407980000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/1ca12b270c00/disk-3a592870.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/4c4c618f8bc4/vmlinux-3a592870.xz
kernel image: https://storage.googleapis.com/syzbot-assets/e49c2df1f067/bzImage-3a592870.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+89a27b...@syzkaller.appspotmail.com

INFO: task syz-executor163:4273 blocked for more than 143 seconds.
Not tainted 5.15.167-syzkaller #0

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

task:syz-executor163 state:D stack:23872 pid: 4273 ppid: 3580 flags:0x00004006

Call Trace:
<TASK>
context_switch kernel/sched/core.c:5027 [inline]
__schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
schedule+0x11b/0x1f0 kernel/sched/core.c:6456

perf_pending_task_sync+0x13c/0x1c0 kernel/events/core.c:5123
_free_event+0x34/0xe60 kernel/events/core.c:5129
put_event kernel/events/core.c:5254 [inline]
perf_event_release_kernel+0x873/0x900 kernel/events/core.c:5379
perf_release+0x37/0x40 kernel/events/core.c:5389

__fput+0x3fe/0x8e0 fs/file_table.c:280
task_work_run+0x129/0x1a0 kernel/task_work.c:188

tracehook_notify_resume include/linux/tracehook.h:189 [inline]
exit_to_user_mode_loop+0x106/0x130 kernel/entry/common.c:181
exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:214
__syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
syscall_exit_to_user_mode+0x5d/0x240 kernel/entry/common.c:307
do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f84b5050af9
RSP: 002b:00007fffdfebc0b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f84b5050af9
RDX: 00007f84b5050af9 RSI: 00000000ffffffff RDI: 0000000000000003
RBP: 00000000000279bd R08: 0000000000000006 R09: 0000000000000006
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fffdfebc0cc
R13: 431bde82d7b634db R14: 0000000000000001 R15: 0000000000000001

</TASK>

Showing all locks held in the system:
1 lock held by khungtaskd/27:

#0: ffffffff8c91fc60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30
2 locks held by getty/3326:
#0: ffff888029385098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x21/0x70 drivers/tty/tty_ldisc.c:252
#1: ffffc9000209b2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6af/0x1db0 drivers/tty/n_tty.c:2158

=============================================

NMI backtrace for cpu 1

CPU: 1 PID: 27 Comm: khungtaskd Not tainted 5.15.167-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
check_hung_uninterruptible_tasks kernel/hung_task.c:210 [inline]
watchdog+0xe72/0xeb0 kernel/hung_task.c:295
kthread+0x3f6/0x4f0 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
</TASK>
Sending NMI from CPU 1 to CPUs 0:

NMI backtrace for cpu 0 skipped: idling at native_safe_halt arch/x86/include/asm/irqflags.h:51 [inline]
NMI backtrace for cpu 0 skipped: idling at arch_safe_halt arch/x86/include/asm/irqflags.h:89 [inline]
NMI backtrace for cpu 0 skipped: idling at acpi_safe_halt drivers/acpi/processor_idle.c:108 [inline]
NMI backtrace for cpu 0 skipped: idling at acpi_idle_do_entry+0x10f/0x340 drivers/acpi/processor_idle.c:562


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.