possible deadlock in perf_event_release_kernel

12 views
Skip to first unread message

syzbot

unread,
Feb 1, 2020, 4:27:10 AM2/1/20
to syzkaller...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 9fa690a2 Linux 4.14.169
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=125998d9e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=eb55b601e76e3476
dashboard link: https://syzkaller.appspot.com/bug?extid=f1c8028dcc896ca525b5
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+f1c802...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
4.14.169-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.3/14307 is trying to acquire lock:
(event_mutex){+.+.}, at: [<ffffffff8162f908>] perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234

but task is already holding lock:
(&event->child_mutex){+.+.}, at: [<ffffffff816d7ba7>] perf_event_release_kernel+0x207/0x880 kernel/events/core.c:4397

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (&event->child_mutex){+.+.}:
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_event_for_each_child+0x8a/0x150 kernel/events/core.c:4682
_perf_ioctl kernel/events/core.c:4869 [inline]
perf_ioctl+0x1d9/0xe50 kernel/events/core.c:4881
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x7ae/0x1060 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #4 (&cpuctx_mutex){+.+.}:
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_event_init_cpu+0xc2/0x170 kernel/events/core.c:11226
perf_event_init+0x2d8/0x31a kernel/events/core.c:11273
start_kernel+0x3b6/0x67d init/main.c:620
x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:399
x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:380
secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #3 (pmus_lock){+.+.}:
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_event_init_cpu+0x2f/0x170 kernel/events/core.c:11220
cpuhp_invoke_callback+0x1ea/0x1ab0 kernel/cpu.c:184
cpuhp_up_callbacks kernel/cpu.c:573 [inline]
_cpu_up+0x228/0x530 kernel/cpu.c:1135
do_cpu_up kernel/cpu.c:1170 [inline]
do_cpu_up+0xa6/0x160 kernel/cpu.c:1142
cpu_up+0x1b/0x20 kernel/cpu.c:1178
smp_init+0x157/0x170 kernel/smp.c:578
kernel_init_freeable+0x30b/0x532 init/main.c:1066
kernel_init+0x12/0x162 init/main.c:998
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #2 (cpu_hotplug_lock.rw_sem){++++}:
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
cpus_read_lock+0x3d/0xc0 kernel/cpu.c:295
static_key_slow_inc+0x13/0x30 kernel/jump_label.c:123
tracepoint_add_func kernel/tracepoint.c:223 [inline]
tracepoint_probe_register_prio+0x4d6/0x6d0 kernel/tracepoint.c:283
tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8109
perf_try_init_event+0x164/0x200 kernel/events/core.c:9345
perf_init_event kernel/events/core.c:9383 [inline]
perf_event_alloc.part.0+0xd92/0x25c0 kernel/events/core.c:9643
perf_event_alloc kernel/events/core.c:9996 [inline]
SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10100
SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9986
do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (tracepoints_mutex){+.+.}:
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
tracepoint_probe_register_prio+0x36/0x6d0 kernel/tracepoint.c:279
tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8109
perf_try_init_event+0x164/0x200 kernel/events/core.c:9345
perf_init_event kernel/events/core.c:9383 [inline]
perf_event_alloc.part.0+0xd92/0x25c0 kernel/events/core.c:9643
perf_event_alloc kernel/events/core.c:9996 [inline]
SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10100
SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9986
do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (event_mutex){+.+.}:
check_prev_add kernel/locking/lockdep.c:1901 [inline]
check_prevs_add kernel/locking/lockdep.c:2018 [inline]
validate_chain kernel/locking/lockdep.c:2460 [inline]
__lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8093
_free_event+0x330/0xe70 kernel/events/core.c:4238
free_event+0x38/0x50 kernel/events/core.c:4265
perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
perf_release+0x37/0x50 kernel/events/core.c:4435
__fput+0x275/0x7a0 fs/file_table.c:210
____fput+0x16/0x20 fs/file_table.c:244
task_work_run+0x114/0x190 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:191 [inline]
exit_to_usermode_loop+0x1da/0x220 arch/x86/entry/common.c:164
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
event_mutex --> &cpuctx_mutex --> &event->child_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&event->child_mutex);
lock(&cpuctx_mutex);
lock(&event->child_mutex);
lock(event_mutex);

*** DEADLOCK ***

2 locks held by syz-executor.3/14307:
#0: (&ctx->mutex){+.+.}, at: [<ffffffff816d7b9d>] perf_event_release_kernel+0x1fd/0x880 kernel/events/core.c:4396
#1: (&event->child_mutex){+.+.}, at: [<ffffffff816d7ba7>] perf_event_release_kernel+0x207/0x880 kernel/events/core.c:4397

stack backtrace:
CPU: 0 PID: 14307 Comm: syz-executor.3 Not tainted 4.14.169-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x142/0x197 lib/dump_stack.c:58
print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
check_prev_add kernel/locking/lockdep.c:1901 [inline]
check_prevs_add kernel/locking/lockdep.c:2018 [inline]
validate_chain kernel/locking/lockdep.c:2460 [inline]
__lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8093
_free_event+0x330/0xe70 kernel/events/core.c:4238
free_event+0x38/0x50 kernel/events/core.c:4265
perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
perf_release+0x37/0x50 kernel/events/core.c:4435
__fput+0x275/0x7a0 fs/file_table.c:210
____fput+0x16/0x20 fs/file_table.c:244
task_work_run+0x114/0x190 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:191 [inline]
exit_to_usermode_loop+0x1da/0x220 arch/x86/entry/common.c:164
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x414f31
RSP: 002b:00007fff90edc6a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000006 RCX: 0000000000414f31
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005
RBP: 0000000000000000 R08: ffffffffffffffff R09: ffffffffffffffff
R10: 00007fff90edc780 R11: 0000000000000293 R12: 000000000075c070
R13: 000000000005b7ca R14: 0000000000763758 R15: 000000000075c07c


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,
Feb 3, 2020, 8:47:13 AM2/3/20
to syzkaller...@googlegroups.com
syzbot has found a reproducer for the following crash on:

HEAD commit: 9fa690a2 Linux 4.14.169
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=138874b5e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=eb55b601e76e3476
dashboard link: https://syzkaller.appspot.com/bug?extid=f1c8028dcc896ca525b5
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15102da5e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=120c8901e00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+f1c802...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
4.14.169-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor964/8248 is trying to acquire lock:
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0xa1a/0x2cd0 kernel/exit.c:858
do_group_exit+0x111/0x330 kernel/exit.c:955
get_signal+0x381/0x1cd0 kernel/signal.c:2418
do_signal+0x86/0x19a0 arch/x86/kernel/signal.c:814
exit_to_usermode_loop+0x15c/0x220 arch/x86/entry/common.c:160
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
event_mutex --> &cpuctx_mutex --> &event->child_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&event->child_mutex);
lock(&cpuctx_mutex);
lock(&event->child_mutex);
lock(event_mutex);

*** DEADLOCK ***

2 locks held by syz-executor964/8248:
#0: (&ctx->mutex){+.+.}, at: [<ffffffff816d7b9d>] perf_event_release_kernel+0x1fd/0x880 kernel/events/core.c:4396
#1: (&event->child_mutex){+.+.}, at: [<ffffffff816d7ba7>] perf_event_release_kernel+0x207/0x880 kernel/events/core.c:4397

stack backtrace:
CPU: 0 PID: 8248 Comm: syz-executor964 Not tainted 4.14.169-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x142/0x197 lib/dump_stack.c:58
print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
check_prev_add kernel/locking/lockdep.c:1901 [inline]
check_prevs_add kernel/locking/lockdep.c:2018 [inline]
validate_chain kernel/locking/lockdep.c:2460 [inline]
__lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8093
_free_event+0x330/0xe70 kernel/events/core.c:4238
free_event+0x38/0x50 kernel/events/core.c:4265
perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
perf_release+0x37/0x50 kernel/events/core.c:4435
__fput+0x275/0x7a0 fs/file_table.c:210
____fput+0x16/0x20 fs/file_table.c:244
task_work_run+0x114/0x190 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0xa1a/0x2cd0 kernel/exit.c:858
do_group_exit+0x111/0x330 kernel/exit.c:955
get_signal+0x381/0x1cd0 kernel/signal.c:2418
do_signal+0x86/0x19a0 arch/x86/kernel/signal.c:814
exit_to_usermode_loop+0x15c/0x220 arch/x86/entry/common.c:160
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x446969
RSP: 002b:00007f2a61e38db8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000006dbc38 RCX: 0000000000446969
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00000000006dbc38
RBP: 00000000006dbc30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dbc3c
R13: 00007ffed2cc5d3f R14: 00007f2a61e399c0 R15: 000000000000002d

Reply all
Reply to author
Forward
0 new messages