[syzbot] [mm?] possible deadlock in __mmap_lock_do_trace_start_locking

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=13601320980000
kernel config: https://syzkaller.appspot.com/x/.config?x=6fb1be60a193d440
dashboard link: https://syzkaller.appspot.com/bug?extid=6ff90931779bcdfc840c
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/3f355021a085/disk-443574b0.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/44cf4de7472a/vmlinux-443574b0.xz
kernel image: https://storage.googleapis.com/syzbot-assets/a99a36c7ad65/bzImage-443574b0.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+6ff909...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-05236-g443574b03387 #0 Not tainted
------------------------------------------------------
syz-executor.2/5452 is trying to acquire lock:
ffff8880b9538528 (lock#10){+.+.}-{2:2}, at: local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
ffff8880b9538528 (lock#10){+.+.}-{2:2}, at: __mmap_lock_do_trace_start_locking+0x83/0x5f0 mm/mmap_lock.c:230

but task is already holding lock:
ffff8880b953e158 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:559

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&rq->__lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
_raw_spin_lock_nested+0x31/0x40 kernel/locking/spinlock.c:378
raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:559
raw_spin_rq_lock kernel/sched/sched.h:1385 [inline]
rq_lock kernel/sched/sched.h:1699 [inline]
task_fork_fair+0x61/0x1e0 kernel/sched/fair.c:12629
sched_cgroup_fork+0x37c/0x410 kernel/sched/core.c:4845
copy_process+0x2217/0x3df0 kernel/fork.c:2498
kernel_clone+0x21e/0x8d0 kernel/fork.c:2796
user_mode_thread+0x132/0x1a0 kernel/fork.c:2874
rest_init+0x27/0x300 init/main.c:695
arch_call_rest_init+0xe/0x10 init/main.c:831
start_kernel+0x47a/0x500 init/main.c:1077
x86_64_start_reservations+0x2a/0x30 arch/x86/kernel/head64.c:509
x86_64_start_kernel+0x99/0xa0 arch/x86/kernel/head64.c:490
common_startup_64+0x13e/0x147

-> #1 (&p->pi_lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
class_raw_spinlock_irqsave_constructor include/linux/spinlock.h:553 [inline]
try_to_wake_up+0xb0/0x1470 kernel/sched/core.c:4262
rcu_read_unlock_special+0x3db/0x550 kernel/rcu/tree_plugin.h:655
__rcu_read_unlock+0xa1/0x110 kernel/rcu/tree_plugin.h:426
rcu_read_unlock include/linux/rcupdate.h:782 [inline]
get_mem_cgroup_from_mm+0x1a3/0x2a0 mm/memcontrol.c:1112
get_mm_memcg_path+0x1b/0x600 mm/mmap_lock.c:202
__mmap_lock_do_trace_start_locking+0x134/0x5f0 mm/mmap_lock.c:230
__mmap_lock_trace_start_locking include/linux/mmap_lock.h:29 [inline]
mmap_read_trylock include/linux/mmap_lock.h:164 [inline]
get_mmap_lock_carefully mm/memory.c:5497 [inline]
lock_mm_and_find_vma+0x1f1/0x2f0 mm/memory.c:5557
do_user_addr_fault arch/x86/mm/fault.c:1385 [inline]
handle_page_fault arch/x86/mm/fault.c:1505 [inline]
exc_page_fault+0x1a9/0x890 arch/x86/mm/fault.c:1563
asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:623
rep_movs_alternative+0x4a/0x70 arch/x86/lib/copy_user_64.S:65
copy_user_generic arch/x86/include/asm/uaccess_64.h:110 [inline]
raw_copy_from_user arch/x86/include/asm/uaccess_64.h:125 [inline]
copy_from_user_iter lib/iov_iter.c:55 [inline]
iterate_ubuf include/linux/iov_iter.h:29 [inline]
iterate_and_advance2 include/linux/iov_iter.h:245 [inline]
iterate_and_advance include/linux/iov_iter.h:271 [inline]
__copy_from_iter lib/iov_iter.c:249 [inline]
_copy_from_iter+0x257/0x1cf0 lib/iov_iter.c:260
copy_from_iter include/linux/uio.h:204 [inline]
copy_from_iter_full include/linux/uio.h:211 [inline]
skb_do_copy_data_nocache include/net/sock.h:2231 [inline]
skb_copy_to_page_nocache include/net/sock.h:2257 [inline]
kcm_sendmsg+0xcca/0x27c0 net/kcm/kcmsock.c:858
sock_sendmsg_nosec net/socket.c:730 [inline]
__sock_sendmsg+0x221/0x270 net/socket.c:745
sock_write_iter+0x2dd/0x400 net/socket.c:1160
call_write_iter include/linux/fs.h:2108 [inline]
new_sync_write fs/read_write.c:497 [inline]
vfs_write+0xa84/0xcb0 fs/read_write.c:590
ksys_write+0x1a0/0x2c0 fs/read_write.c:643
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #0 (lock#10){+.+.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
__mmap_lock_do_trace_start_locking+0x9c/0x5f0 mm/mmap_lock.c:230
__mmap_lock_trace_start_locking include/linux/mmap_lock.h:29 [inline]
mmap_read_trylock include/linux/mmap_lock.h:164 [inline]
stack_map_get_build_id_offset+0x98d/0x9d0 kernel/bpf/stackmap.c:141
__bpf_get_stack+0x4ad/0x5a0 kernel/bpf/stackmap.c:449
____bpf_get_stack_raw_tp kernel/trace/bpf_trace.c:1985 [inline]
bpf_get_stack_raw_tp+0x1a3/0x240 kernel/trace/bpf_trace.c:1975
bpf_prog_5c8cb7646b59f521+0x42/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_tlb_flush+0xfa/0x120 include/trace/events/tlb.h:38
switch_mm_irqs_off+0x7cb/0xae0
context_switch kernel/sched/core.c:5393 [inline]
__schedule+0x1054/0x4a20 kernel/sched/core.c:6736
__schedule_loop kernel/sched/core.c:6813 [inline]
schedule+0x14b/0x320 kernel/sched/core.c:6828
exit_to_user_mode_loop kernel/entry/common.c:96 [inline]
exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
irqentry_exit_to_user_mode+0xe7/0x270 kernel/entry/common.c:225
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702

other info that might help us debug this:

Chain exists of:
lock#10 --> &p->pi_lock --> &rq->__lock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&rq->__lock);
lock(&p->pi_lock);
lock(&rq->__lock);
lock(lock#10);

*** DEADLOCK ***

2 locks held by syz-executor.2/5452:
#0: ffff8880b953e158 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:559
#1: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#1: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#1: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2380 [inline]
#1: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x114/0x420 kernel/trace/bpf_trace.c:2420

stack backtrace:
CPU: 1 PID: 5452 Comm: syz-executor.2 Not tainted 6.8.0-syzkaller-05236-g443574b03387 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2e0 lib/dump_stack.c:106
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
__mmap_lock_do_trace_start_locking+0x9c/0x5f0 mm/mmap_lock.c:230
__mmap_lock_trace_start_locking include/linux/mmap_lock.h:29 [inline]
mmap_read_trylock include/linux/mmap_lock.h:164 [inline]
stack_map_get_build_id_offset+0x98d/0x9d0 kernel/bpf/stackmap.c:141
__bpf_get_stack+0x4ad/0x5a0 kernel/bpf/stackmap.c:449
____bpf_get_stack_raw_tp kernel/trace/bpf_trace.c:1985 [inline]
bpf_get_stack_raw_tp+0x1a3/0x240 kernel/trace/bpf_trace.c:1975
bpf_prog_5c8cb7646b59f521+0x42/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_tlb_flush+0xfa/0x120 include/trace/events/tlb.h:38
switch_mm_irqs_off+0x7cb/0xae0
context_switch kernel/sched/core.c:5393 [inline]
__schedule+0x1054/0x4a20 kernel/sched/core.c:6736
__schedule_loop kernel/sched/core.c:6813 [inline]
schedule+0x14b/0x320 kernel/sched/core.c:6828
exit_to_user_mode_loop kernel/entry/common.c:96 [inline]
exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
irqentry_exit_to_user_mode+0xe7/0x270 kernel/entry/common.c:225
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0033:0x7fe38f639ba5
Code: Unable to access opcode bytes at 0x7fe38f639b7b.
RSP: 002b:00007ffe0d7da5f0 EFLAGS: 00000a87
RAX: 00000000984f6378 RBX: 00007fe38f7ac018 RCX: 0000000000000415
RDX: ffffffff81408dcc RSI: 0000000080000000 RDI: 00000000984f6378
RBP: 0000000081408000 R08: 0000001b31220000 R09: 0000000000000dcd
R10: 0000000081372dd1 R11: 0000000000000000 R12: 0000000000000415
R13: 0000000000000001 R14: 00007fe38f7a0000 R15: ffffffff81408dcc
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 36534d3c5453 tcp: use signed arithmetic in tcp_rtx_probe0_..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=110e9dde980000
kernel config: https://syzkaller.appspot.com/x/.config?x=333ebe38d43c42e2

dashboard link: https://syzkaller.appspot.com/bug?extid=6ff90931779bcdfc840c
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1585acfa980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17bdb7ee980000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/4e648f638b5f/disk-36534d3c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/bbe0d41240f1/vmlinux-36534d3c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/17eb17ecd214/bzImage-36534d3c.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+6ff909...@syzkaller.appspotmail.com

============================================

WARNING: possible recursive locking detected
6.10.0-rc2-syzkaller-00242-g36534d3c5453 #0 Not tainted
--------------------------------------------
syz-executor181/5090 is trying to acquire lock:
ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: __mmap_lock_do_trace_start_locking+0x83/0x620 mm/mmap_lock.c:230

but task is already holding lock:

ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: __mmap_lock_do_trace_start_locking+0x83/0x620 mm/mmap_lock.c:230

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0

----
lock(lock#9);
lock(lock#9);

*** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by syz-executor181/5090:
#0: ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
#0: ffff8880b9538828 (lock#9){+.+.}-{2:2}, at: __mmap_lock_do_trace_start_locking+0x83/0x620 mm/mmap_lock.c:230
#1: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
#1: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
#1: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: get_mem_cgroup_from_mm+0x38/0x2a0 mm/memcontrol.c:1265
#2: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: trace_call_bpf+0xbc/0x8a0

stack backtrace:
CPU: 1 PID: 5090 Comm: syz-executor181 Not tainted 6.10.0-rc2-syzkaller-00242-g36534d3c5453 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024

Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]

dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_deadlock kernel/locking/lockdep.c:3062 [inline]
validate_chain+0x15d3/0x5900 kernel/locking/lockdep.c:3856
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
local_lock_acquire include/linux/local_lock_internal.h:29 [inline]
__mmap_lock_do_trace_start_locking+0x9c/0x620 mm/mmap_lock.c:230
__mmap_lock_trace_start_locking include/linux/mmap_lock.h:29 [inline]
mmap_read_trylock include/linux/mmap_lock.h:162 [inline]
stack_map_get_build_id_offset+0x98a/0x9d0 kernel/bpf/stackmap.c:141
__bpf_get_stack+0x4ad/0x5a0 kernel/bpf/stackmap.c:449
bpf_prog_e6cf5f9c69743609+0x42/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
__bpf_prog_run include/linux/filter.h:691 [inline]
bpf_prog_run include/linux/filter.h:698 [inline]
bpf_prog_run_array include/linux/bpf.h:2103 [inline]
trace_call_bpf+0x369/0x8a0 kernel/trace/bpf_trace.c:147
perf_trace_run_bpf_submit+0x7c/0x1d0 kernel/events/core.c:10269
perf_trace_lock+0x388/0x490 include/trace/events/lock.h:50
trace_lock_release include/trace/events/lock.h:69 [inline]
lock_release+0x986/0x9f0 kernel/locking/lockdep.c:5765
rcu_lock_release include/linux/rcupdate.h:339 [inline]
rcu_read_unlock include/linux/rcupdate.h:812 [inline]
get_mem_cgroup_from_mm+0x1ad/0x2a0 mm/memcontrol.c:1271
get_mm_memcg_path+0x1b/0x600 mm/mmap_lock.c:202
__mmap_lock_do_trace_start_locking+0x134/0x620 mm/mmap_lock.c:230
__mmap_lock_trace_start_locking include/linux/mmap_lock.h:29 [inline]
mmap_read_lock include/linux/mmap_lock.h:143 [inline]
acct_collect+0x7e7/0x830 kernel/acct.c:563
do_exit+0x936/0x27e0 kernel/exit.c:853
do_group_exit+0x207/0x2c0 kernel/exit.c:1023
__do_sys_exit_group kernel/exit.c:1034 [inline]
__se_sys_exit_group kernel/exit.c:1032 [inline]
__x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1032
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f9c47742279
Code: 90 49 c7 c0 b8 ff ff ff be e7 00 00 00 ba 3c 00 00 00 eb 12 0f 1f 44 00 00 89 d0 0f 05 48 3d 00 f0 ff ff 77 1c f4 89 f0 0f 05 <48> 3d 00 f0 ff ff 76 e7 f7 d8 64 41 89 00 eb df 0f 1f 80 00 00 00
RSP: 002b:00007ffd321ae558 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9c47742279
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 00007f9c477bd2b0 R08: ffffffffffffffb8 R09: 00000000000000a0
R10: 00000000000000a0 R11: 0000000000000246 R12: 00007f9c477bd2b0
R13: 0000000000000000 R14: 00007f9c477bdd20 R15: 00007f9c47713400
</TASK>


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

[syzbot]

syzbot has bisected this issue to:

commit 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2
Author: Jesper Dangaard Brouer <ha...@kernel.org>
Date: Wed May 1 14:04:11 2024 +0000

cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=16695261980000
start commit: 36534d3c5453 tcp: use signed arithmetic in tcp_rtx_probe0_..
git tree: bpf
final oops: https://syzkaller.appspot.com/x/report.txt?x=15695261980000
console output: https://syzkaller.appspot.com/x/log.txt?x=11695261980000

kernel config: https://syzkaller.appspot.com/x/.config?x=333ebe38d43c42e2
dashboard link: https://syzkaller.appspot.com/bug?extid=6ff90931779bcdfc840c

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1585acfa980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17bdb7ee980000

Reported-by: syzbot+6ff909...@syzkaller.appspotmail.com
Fixes: 21c38a3bd4ee ("cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Waiman Long]

+static __always_inline
+unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu,
+                                    struct cgroup *cgrp, const bool
fast_path)
+{
+       unsigned long flags;
+       bool contended;
+
+       /*
+        * The _irqsave() is needed because cgroup_rstat_lock is
+        * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring
+        * this lock with the _irq() suffix only disables interrupts on
+        * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables
+        * interrupts on both configurations. The _irqsave() ensures
+        * that interrupts are always disabled and later restored.
+        */
+       contended = !raw_spin_trylock_irqsave(cpu_lock, flags);
+       if (contended) {
+               if (fast_path)
+ trace_cgroup_rstat_cpu_lock_contended_fastpath(cgrp, cp>
+               else
+                       trace_cgroup_rstat_cpu_lock_contended(cgrp, cpu,
conten>
+
+               raw_spin_lock_irqsave(cpu_lock, flags);
+       }

I believe the problem may be caused by the fact that
trace_cgroup_rstat_cpu_lock_contended*() can be called with IRQ enabled.
I had suggested before IRQ should be disabled first before doing any
trace operation. See

https://lore.kernel.org/linux-mm/203fdb35-f4cf-4754...@redhat.com/

Doing so may be able to resolve this possible deadlock.

Cheers,
Longman