[v6.1] possible deadlock in static_key_slow_inc

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 42616e0f09fb Linux 6.1.15
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=114e80dac80000
kernel config: https://syzkaller.appspot.com/x/.config?x=650737f7e9682672
dashboard link: https://syzkaller.appspot.com/bug?extid=9be0d52892bc93e4de1d
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/f10713d1fd0f/disk-42616e0f.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5a1307bb774e/vmlinux-42616e0f.xz
kernel image: https://storage.googleapis.com/syzbot-assets/388238a30fe4/Image-42616e0f.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9be0d5...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.1.15-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.0/30328 is trying to acquire lock:
ffff8000157b5c10 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_slow_inc+0x1c/0x38 kernel/jump_label.c:158

but task is already holding lock:
ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_change_state kernel/cgroup/legacy_freezer.c:387 [inline]
ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0xc0/0x3b8 kernel/cgroup/legacy_freezer.c:426

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (freezer_mutex){+.+.}-{3:3}:
__mutex_lock_common+0x190/0x21a0 kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
freezer_attach+0xa8/0x268 kernel/cgroup/legacy_freezer.c:163
cgroup_migrate_execute+0x6bc/0xd18 kernel/cgroup/cgroup.c:2615
cgroup_migrate+0x18c/0x1a4 kernel/cgroup/cgroup.c:2873
cgroup_attach_task+0x504/0xafc kernel/cgroup/cgroup.c:2906
__cgroup1_procs_write+0x2fc/0x404 kernel/cgroup/cgroup-v1.c:523
cgroup1_procs_write+0x38/0x4c kernel/cgroup/cgroup-v1.c:536
cgroup_file_write+0x258/0x56c kernel/cgroup/cgroup.c:4057
kernfs_fop_write_iter+0x334/0x48c fs/kernfs/file.c:330
call_write_iter include/linux/fs.h:2205 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

-> #1 (cgroup_threadgroup_rwsem){++++}-{0:0}:
percpu_down_write+0x78/0x320 kernel/locking/percpu-rwsem.c:227
cgroup_attach_lock kernel/cgroup/cgroup.c:2431 [inline]
cgroup_procs_write_start+0x1ac/0x5bc kernel/cgroup/cgroup.c:2935
__cgroup_procs_write+0xec/0x410 kernel/cgroup/cgroup.c:5135
cgroup_procs_write+0x34/0x60 kernel/cgroup/cgroup.c:5171
cgroup_file_write+0x258/0x56c kernel/cgroup/cgroup.c:4057
kernfs_fop_write_iter+0x334/0x48c fs/kernfs/file.c:330
call_write_iter include/linux/fs.h:2205 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

-> #0 (cpu_hotplug_lock){++++}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3097 [inline]
check_prevs_add kernel/locking/lockdep.c:3216 [inline]
validate_chain kernel/locking/lockdep.c:3831 [inline]
__lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5055
lock_acquire+0x2f8/0x8dc kernel/locking/lockdep.c:5668
percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
cpus_read_lock+0x70/0x29c kernel/cpu.c:310
static_key_slow_inc+0x1c/0x38 kernel/jump_label.c:158
freezer_apply_state+0x18c/0x3d0 kernel/cgroup/legacy_freezer.c:353
freezer_write+0x24c/0x3b8 kernel/cgroup/legacy_freezer.c:426
cgroup_file_write+0x258/0x56c kernel/cgroup/cgroup.c:4057
kernfs_fop_write_iter+0x334/0x48c fs/kernfs/file.c:330
call_write_iter include/linux/fs.h:2205 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581

other info that might help us debug this:

Chain exists of:
cpu_hotplug_lock --> cgroup_threadgroup_rwsem --> freezer_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(freezer_mutex);
lock(cgroup_threadgroup_rwsem);
lock(freezer_mutex);
lock(cpu_hotplug_lock);

*** DEADLOCK ***

5 locks held by syz-executor.0/30328:
#0: ffff0000d34bd768 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xd8/0x104 fs/file.c:1046
#1: ffff0000d798e460 (sb_writers#9){.+.+}-{0:0}, at: vfs_write+0x244/0x914 fs/read_write.c:580
#2: ffff000104486888 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1c8/0x48c fs/kernfs/file.c:321
#3: ffff0000d6baabd0 (kn->active#62){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x1e4/0x48c fs/kernfs/file.c:322
#4: ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_change_state kernel/cgroup/legacy_freezer.c:387 [inline]
#4: ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0xc0/0x3b8 kernel/cgroup/legacy_freezer.c:426

stack backtrace:
CPU: 1 PID: 30328 Comm: syz-executor.0 Not tainted 6.1.15-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
Call trace:
dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
dump_stack+0x1c/0x58 lib/dump_stack.c:113
print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2055
check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2177
check_prev_add kernel/locking/lockdep.c:3097 [inline]
check_prevs_add kernel/locking/lockdep.c:3216 [inline]
validate_chain kernel/locking/lockdep.c:3831 [inline]
__lock_acquire+0x3338/0x764c kernel/locking/lockdep.c:5055
lock_acquire+0x2f8/0x8dc kernel/locking/lockdep.c:5668
percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
cpus_read_lock+0x70/0x29c kernel/cpu.c:310
static_key_slow_inc+0x1c/0x38 kernel/jump_label.c:158
freezer_apply_state+0x18c/0x3d0 kernel/cgroup/legacy_freezer.c:353
freezer_write+0x24c/0x3b8 kernel/cgroup/legacy_freezer.c:426
cgroup_file_write+0x258/0x56c kernel/cgroup/cgroup.c:4057
kernfs_fop_write_iter+0x334/0x48c fs/kernfs/file.c:330
call_write_iter include/linux/fs.h:2205 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 42616e0f09fb Linux 6.1.15
git tree: linux-6.1.y

console output: https://syzkaller.appspot.com/x/log.txt?x=12a294ccc80000

kernel config: https://syzkaller.appspot.com/x/.config?x=650737f7e9682672
dashboard link: https://syzkaller.appspot.com/bug?extid=9be0d52892bc93e4de1d
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13d65ba2c80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=148b4f5ac80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/f10713d1fd0f/disk-42616e0f.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5a1307bb774e/vmlinux-42616e0f.xz
kernel image: https://storage.googleapis.com/syzbot-assets/388238a30fe4/Image-42616e0f.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9be0d5...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.1.15-syzkaller #0 Not tainted
------------------------------------------------------

syz-executor255/4316 is trying to acquire lock:

4 locks held by syz-executor255/4316:
#0: ffff0000d945a460 (sb_writers#9){.+.+}-{0:0}, at: vfs_write+0x244/0x914 fs/read_write.c:580
#1: ffff0000c9df3c88 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1c8/0x48c fs/kernfs/file.c:321
#2: ffff0000c1a86748 (kn->active#51){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x1e4/0x48c fs/kernfs/file.c:322
#3: ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_change_state kernel/cgroup/legacy_freezer.c:387 [inline]
#3: ffff800015948048 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0xc0/0x3b8 kernel/cgroup/legacy_freezer.c:426

stack backtrace:
CPU: 1 PID: 4316 Comm: syz-executor255 Not tainted 6.1.15-syzkaller #0

[syzbot]

syzbot suspects this issue was fixed by commit:

commit 0d7a4e6589a02fd87d286f7f0713ba6d8fc344e6
Author: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
Date: Sun Jun 11 13:48:12 2023 +0000

cgroup,freezer: hold cpu_hotplug_lock before freezer_mutex in freezer_css_{online,offline}()

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=110bc603a80000
start commit: 2f3918bc53fb Linux 6.1.33
git tree: linux-6.1.y
kernel config: https://syzkaller.appspot.com/x/.config?x=668ab7dd51e152ad
dashboard link: https://syzkaller.appspot.com/bug?extid=9be0d52892bc93e4de1d
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=110241fd280000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14851b43280000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: cgroup,freezer: hold cpu_hotplug_lock before freezer_mutex in freezer_css_{online,offline}()

For information about bisection process see: https://goo.gl/tpsmEJ#bisection