possible deadlock in static_key_slow_dec (3)

6 views
Skip to first unread message

syzbot

unread,
Oct 4, 2021, 5:46:25 PM10/4/21
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: c2276d585654 Linux 4.19.208
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=15943deb300000
kernel config: https://syzkaller.appspot.com/x/.config?x=9080caca9deef589
dashboard link: https://syzkaller.appspot.com/bug?extid=63b69a71f0d5917ca2b7
compiler: gcc version 10.2.1 20210110 (Debian 10.2.1-6)

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+63b69a...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
4.19.208-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.2/26450 is trying to acquire lock:
00000000c897dc42 (cpu_hotplug_lock.rw_sem){++++}, at: __static_key_slow_dec kernel/jump_label.c:222 [inline]
00000000c897dc42 (cpu_hotplug_lock.rw_sem){++++}, at: static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:237

but task is already holding lock:
00000000066072f4 (&mm->mmap_sem){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:355

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&mm->mmap_sem){++++}:
perf_copy_attr+0xab/0x7a0 kernel/events/core.c:10275
_perf_ioctl+0x6aa/0x2300 kernel/events/core.c:5150
perf_ioctl+0x55/0x80 kernel/events/core.c:5177
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:501 [inline]
do_vfs_ioctl+0xcdb/0x12e0 fs/ioctl.c:688
ksys_ioctl+0x9b/0xc0 fs/ioctl.c:705
__do_sys_ioctl fs/ioctl.c:712 [inline]
__se_sys_ioctl fs/ioctl.c:710 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:710
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (&cpuctx_mutex){+.+.}:
perf_event_init_cpu+0xc4/0x170 kernel/events/core.c:11828
perf_event_init+0x309/0x34e kernel/events/core.c:11875
start_kernel+0x5b1/0x911 init/main.c:644
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243

-> #1 (pmus_lock){+.+.}:
perf_event_init_cpu+0x2c/0x170 kernel/events/core.c:11822
cpuhp_invoke_callback+0x201/0x1b80 kernel/cpu.c:170
cpuhp_up_callbacks kernel/cpu.c:584 [inline]
_cpu_up+0x25c/0x540 kernel/cpu.c:1192
do_cpu_up+0xdd/0x1b0 kernel/cpu.c:1228
smp_init+0x1ed/0x202 kernel/smp.c:578
kernel_init_freeable+0x62b/0xab7 init/main.c:1138
kernel_init+0xd/0x1ba init/main.c:1062
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #0 (cpu_hotplug_lock.rw_sem){++++}:
percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
cpus_read_lock+0x3b/0xd0 kernel/cpu.c:290
__static_key_slow_dec kernel/jump_label.c:222 [inline]
static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:237
sw_perf_event_destroy+0x8a/0x120 kernel/events/core.c:8311
_free_event+0x32c/0x1150 kernel/events/core.c:4484
put_event kernel/events/core.c:4578 [inline]
perf_mmap_close+0x6f6/0xea0 kernel/events/core.c:5582
remove_vma+0xa9/0x170 mm/mmap.c:176
remove_vma_list mm/mmap.c:2550 [inline]
do_munmap+0x6f9/0xde0 mm/mmap.c:2786
mmap_region+0x2a3/0x16b0 mm/mmap.c:1700
do_mmap+0x8e8/0x1080 mm/mmap.c:1530
do_mmap_pgoff include/linux/mm.h:2329 [inline]
vm_mmap_pgoff+0x197/0x200 mm/util.c:357
ksys_mmap_pgoff+0x298/0x5a0 mm/mmap.c:1580
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
cpu_hotplug_lock.rw_sem --> &cpuctx_mutex --> &mm->mmap_sem

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&mm->mmap_sem);
lock(&cpuctx_mutex);
lock(&mm->mmap_sem);
lock(cpu_hotplug_lock.rw_sem);

*** DEADLOCK ***

1 lock held by syz-executor.2/26450:
#0: 00000000066072f4 (&mm->mmap_sem){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:355

stack backtrace:
CPU: 0 PID: 26450 Comm: syz-executor.2 Not tainted 4.19.208-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1222
check_prev_add kernel/locking/lockdep.c:1866 [inline]
check_prevs_add kernel/locking/lockdep.c:1979 [inline]
validate_chain kernel/locking/lockdep.c:2420 [inline]
__lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3416
lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3908
percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
cpus_read_lock+0x3b/0xd0 kernel/cpu.c:290
__static_key_slow_dec kernel/jump_label.c:222 [inline]
static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:237
sw_perf_event_destroy+0x8a/0x120 kernel/events/core.c:8311
_free_event+0x32c/0x1150 kernel/events/core.c:4484
put_event kernel/events/core.c:4578 [inline]
perf_mmap_close+0x6f6/0xea0 kernel/events/core.c:5582
remove_vma+0xa9/0x170 mm/mmap.c:176
remove_vma_list mm/mmap.c:2550 [inline]
do_munmap+0x6f9/0xde0 mm/mmap.c:2786
mmap_region+0x2a3/0x16b0 mm/mmap.c:1700
do_mmap+0x8e8/0x1080 mm/mmap.c:1530
do_mmap_pgoff include/linux/mm.h:2329 [inline]
vm_mmap_pgoff+0x197/0x200 mm/util.c:357
ksys_mmap_pgoff+0x298/0x5a0 mm/mmap.c:1580
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f28b3fec8d9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f28b1521188 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 00007f28b40f10e0 RCX: 00007f28b3fec8d9
RDX: 0000000000000000 RSI: 0000000000002000 RDI: 0000000020ffd000
RBP: 00007f28b4046cb4 R08: 0000000000000004 R09: 0000000000000000
R10: 0000200000000011 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffc36092c4f R14: 00007f28b1521300 R15: 0000000000022000
audit: type=1804 audit(1633383920.256:425): pid=26467 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2447/bus" dev="sda1" ino=15583 res=1
audit: type=1800 audit(1633383920.276:426): pid=26467 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=collect_data cause=failed(directio) comm="syz-executor.0" name="bus" dev="sda1" ino=15583 res=0
kauditd_printk_skb: 46 callbacks suppressed
audit: type=1804 audit(1633383922.696:473): pid=26641 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/130/bus" dev="sda1" ino=14980 res=1
audit: type=1804 audit(1633383922.696:474): pid=26641 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/130/bus" dev="sda1" ino=14980 res=1
audit: type=1804 audit(1633383922.696:475): pid=26641 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/130/bus" dev="sda1" ino=14980 res=1
audit: type=1804 audit(1633383922.806:476): pid=26593 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2451/bus" dev="sda1" ino=15713 res=1
audit: type=1804 audit(1633383922.896:477): pid=26656 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/131/bus" dev="sda1" ino=15593 res=1
audit: type=1804 audit(1633383922.896:478): pid=26656 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/131/bus" dev="sda1" ino=15593 res=1
audit: type=1804 audit(1633383922.896:479): pid=26656 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/131/bus" dev="sda1" ino=15593 res=1
audit: type=1804 audit(1633383923.336:480): pid=26682 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/132/bus" dev="sda1" ino=15745 res=1
audit: type=1804 audit(1633383923.366:481): pid=26682 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/132/bus" dev="sda1" ino=15745 res=1
audit: type=1804 audit(1633383923.366:482): pid=26682 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.2" name="/root/syzkaller-testdir470260520/syzkaller.IE6zsI/132/bus" dev="sda1" ino=15745 res=1
TCP: request_sock_TCPv6: Possible SYN flooding on port 20002. Sending cookies. Check SNMP counters.
kauditd_printk_skb: 44 callbacks suppressed
audit: type=1804 audit(1633383927.986:527): pid=26940 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2466/bus" dev="sda1" ino=14264 res=1
audit: type=1804 audit(1633383927.986:528): pid=26940 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2466/bus" dev="sda1" ino=14264 res=1
audit: type=1804 audit(1633383927.986:529): pid=26940 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2466/bus" dev="sda1" ino=14264 res=1
audit: type=1804 audit(1633383928.136:530): pid=26953 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2467/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383928.136:531): pid=26953 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2467/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383928.136:532): pid=26953 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2467/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383928.306:533): pid=26969 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2468/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383928.306:534): pid=26969 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=ToMToU comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2468/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383928.306:535): pid=26969 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2468/bus" dev="sda1" ino=14740 res=1
audit: type=1804 audit(1633383929.037:536): pid=26994 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=invalid_pcr cause=open_writers comm="syz-executor.0" name="/root/syzkaller-testdir589852850/syzkaller.t3NQyh/2469/bus" dev="sda1" ino=14962 res=1


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,
Mar 22, 2022, 1:24:15 AM3/22/22
to syzkaller...@googlegroups.com
Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.
Reply all
Reply to author
Forward
0 new messages