[v5.15] possible deadlock in do_qc

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: b925f60c6ee7 Linux 5.15.157
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=152e80a7180000
kernel config: https://syzkaller.appspot.com/x/.config?x=2a1cb0d51cbb9dfb
dashboard link: https://syzkaller.appspot.com/bug?extid=216240cf171bef980ebb
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/07f51426f82f/disk-b925f60c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/d41cc73399aa/vmlinux-b925f60c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/0f542c09e64b/bzImage-b925f60c.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+216240...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
5.15.157-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.4/5271 is trying to acquire lock:
ffff888078b84ae0 (&sdp->sd_quota_mutex){+.+.}-{3:3}, at: do_qc+0xc8/0x720 fs/gfs2/quota.c:678

but task is already holding lock:
ffff8880546e5080 (&ip->i_rw_mutex){++++}-{3:3}, at: sweep_bh_for_rgrps fs/gfs2/bmap.c:1514 [inline]
ffff8880546e5080 (&ip->i_rw_mutex){++++}-{3:3}, at: punch_hole+0x27ee/0x3ae0 fs/gfs2/bmap.c:1838

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&ip->i_rw_mutex){++++}-{3:3}:
lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
down_read+0x45/0x2e0 kernel/locking/rwsem.c:1498
__gfs2_iomap_get+0x178/0x1480 fs/gfs2/bmap.c:858
gfs2_iomap_get fs/gfs2/bmap.c:1398 [inline]
gfs2_block_map+0x292/0x830 fs/gfs2/bmap.c:1213
bh_get+0x26d/0x6a0 fs/gfs2/quota.c:385
qdsb_get+0x209/0x3b0 fs/gfs2/quota.c:518
gfs2_quota_hold+0x19d/0x5c0 fs/gfs2/quota.c:604
punch_hole+0xeb6/0x3ae0 fs/gfs2/bmap.c:1799
gfs2_iomap_end+0x511/0x6d0 fs/gfs2/bmap.c:1159
iomap_iter+0x222/0xe30 fs/iomap/iter.c:62
iomap_file_buffered_write+0x748/0x8a0 fs/iomap/buffered-io.c:814
gfs2_file_buffered_write+0x3e3/0x850 fs/gfs2/file.c:1057
gfs2_file_write_iter+0xcdc/0xf10 fs/gfs2/file.c:1144
call_write_iter include/linux/fs.h:2172 [inline]
new_sync_write fs/read_write.c:507 [inline]
vfs_write+0xacf/0xe50 fs/read_write.c:594
ksys_write+0x1a2/0x2c0 fs/read_write.c:647
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #0 (&sdp->sd_quota_mutex){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
__lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
__mutex_lock_common+0x1da/0x25a0 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0x17/0x20 kernel/locking/mutex.c:743
do_qc+0xc8/0x720 fs/gfs2/quota.c:678
gfs2_quota_change+0x2db/0x8c0 fs/gfs2/quota.c:1295
punch_hole+0x3457/0x3ae0 fs/gfs2/bmap.c:1939
gfs2_iomap_end+0x511/0x6d0 fs/gfs2/bmap.c:1159
iomap_iter+0x222/0xe30 fs/iomap/iter.c:62
iomap_file_buffered_write+0x748/0x8a0 fs/iomap/buffered-io.c:814
gfs2_file_buffered_write+0x3e3/0x850 fs/gfs2/file.c:1057
gfs2_file_write_iter+0xcdc/0xf10 fs/gfs2/file.c:1144
call_write_iter include/linux/fs.h:2172 [inline]
new_sync_write fs/read_write.c:507 [inline]
vfs_write+0xacf/0xe50 fs/read_write.c:594
ksys_write+0x1a2/0x2c0 fs/read_write.c:647
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x66/0xd0

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&ip->i_rw_mutex);
lock(&sdp->sd_quota_mutex);
lock(&ip->i_rw_mutex);
lock(&sdp->sd_quota_mutex);

*** DEADLOCK ***

6 locks held by syz-executor.4/5271:
#0: ffff88805b2cc5f0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x2cb/0x380 fs/file.c:1088
#1: ffff888056028460 (sb_writers#26){.+.+}-{0:0}, at: vfs_write+0x29a/0xe50 fs/read_write.c:590
#2: ffff8880546e4be8 (&sb->s_type->i_mutex_key#35){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:789 [inline]
#2: ffff8880546e4be8 (&sb->s_type->i_mutex_key#35){+.+.}-{3:3}, at: gfs2_file_write_iter+0x373/0xf10 fs/gfs2/file.c:1122
#3: ffff888056028650 (sb_internal#6){.+.+}-{0:0}, at: gfs2_trans_begin+0x6d/0xe0 fs/gfs2/trans.c:118
#4: ffff888078b850a8 (&sdp->sd_log_flush_lock){.+.+}-{3:3}, at: __gfs2_trans_begin+0x55a/0x940 fs/gfs2/trans.c:87
#5: ffff8880546e5080 (&ip->i_rw_mutex){++++}-{3:3}, at: sweep_bh_for_rgrps fs/gfs2/bmap.c:1514 [inline]
#5: ffff8880546e5080 (&ip->i_rw_mutex){++++}-{3:3}, at: punch_hole+0x27ee/0x3ae0 fs/gfs2/bmap.c:1838

stack backtrace:
CPU: 0 PID: 5271 Comm: syz-executor.4 Not tainted 5.15.157-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
__lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
__mutex_lock_common+0x1da/0x25a0 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0x17/0x20 kernel/locking/mutex.c:743
do_qc+0xc8/0x720 fs/gfs2/quota.c:678
gfs2_quota_change+0x2db/0x8c0 fs/gfs2/quota.c:1295
punch_hole+0x3457/0x3ae0 fs/gfs2/bmap.c:1939
gfs2_iomap_end+0x511/0x6d0 fs/gfs2/bmap.c:1159
iomap_iter+0x222/0xe30 fs/iomap/iter.c:62
iomap_file_buffered_write+0x748/0x8a0 fs/iomap/buffered-io.c:814
gfs2_file_buffered_write+0x3e3/0x850 fs/gfs2/file.c:1057
gfs2_file_write_iter+0xcdc/0xf10 fs/gfs2/file.c:1144
call_write_iter include/linux/fs.h:2172 [inline]
new_sync_write fs/read_write.c:507 [inline]
vfs_write+0xacf/0xe50 fs/read_write.c:594
ksys_write+0x1a2/0x2c0 fs/read_write.c:647
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7ffa31aedea9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffa300600c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00007ffa31c1bf80 RCX: 00007ffa31aedea9
RDX: 0000000000200600 RSI: 0000000000000000 RDI: 0000000000000005
RBP: 00007ffa31b3a4a4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007ffa31c1bf80 R15: 00007ffda4843958
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: f2295faba5e8 Linux 6.1.88
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=124a62f8980000
kernel config: https://syzkaller.appspot.com/x/.config?x=508163b9ab79dc25
dashboard link: https://syzkaller.appspot.com/bug?extid=bdc96b32d35cce7a6064

compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:

disk image: https://storage.googleapis.com/syzbot-assets/db6c44dd5d81/disk-f2295fab.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/96a1c7aa1507/vmlinux-f2295fab.xz
kernel image: https://storage.googleapis.com/syzbot-assets/84353702f58a/Image-f2295fab.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by: syzbot+bdc96b...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected

6.1.88-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.1/4341 is trying to acquire lock:
ffff0000f8128ae0 (&sdp->sd_quota_mutex){+.+.}-{3:3}, at: do_qc+0xbc/0x664 fs/gfs2/quota.c:682

but task is already holding lock:

ffff0000f23205e8 (&ip->i_rw_mutex){++++}-{3:3}, at: sweep_bh_for_rgrps fs/gfs2/bmap.c:1512 [inline]
ffff0000f23205e8 (&ip->i_rw_mutex){++++}-{3:3}, at: punch_hole+0x1fb0/0x2ff0 fs/gfs2/bmap.c:1836

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&ip->i_rw_mutex){++++}-{3:3}:

down_read+0x64/0x308 kernel/locking/rwsem.c:1520
__gfs2_iomap_get+0x138/0x1058 fs/gfs2/bmap.c:856
gfs2_iomap_get+0xd8/0x154 fs/gfs2/bmap.c:1396
bh_get+0x210/0x6b8 fs/gfs2/quota.c:385
qdsb_get+0x1e0/0x398 fs/gfs2/quota.c:522
gfs2_quota_hold+0x16c/0x558 fs/gfs2/quota.c:608
punch_hole+0xc14/0x2ff0 fs/gfs2/bmap.c:1797
gfs2_iomap_end+0x448/0x618 fs/gfs2/bmap.c:1157
iomap_iter+0x1e0/0x112c fs/iomap/iter.c:62
iomap_file_buffered_write+0x734/0x8c0 fs/iomap/buffered-io.c:822
gfs2_file_buffered_write+0x49c/0x800 fs/gfs2/file.c:1056
gfs2_file_write_iter+0x3d4/0xc3c fs/gfs2/file.c:1157
call_write_iter include/linux/fs.h:2265 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585

-> #0 (&sdp->sd_quota_mutex){+.+.}-{3:3}:

check_prev_add kernel/locking/lockdep.c:3090 [inline]
check_prevs_add kernel/locking/lockdep.c:3209 [inline]
validate_chain kernel/locking/lockdep.c:3825 [inline]
__lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
__mutex_lock_common+0x190/0x21a0 kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
do_qc+0xbc/0x664 fs/gfs2/quota.c:682
gfs2_quota_change+0x238/0x718 fs/gfs2/quota.c:1295
punch_hole+0x2aa8/0x2ff0 fs/gfs2/bmap.c:1937
gfs2_iomap_end+0x448/0x618 fs/gfs2/bmap.c:1157
iomap_iter+0x1e0/0x112c fs/iomap/iter.c:62
iomap_file_buffered_write+0x734/0x8c0 fs/iomap/buffered-io.c:822
gfs2_file_buffered_write+0x49c/0x800 fs/gfs2/file.c:1056
gfs2_file_write_iter+0x3d4/0xc3c fs/gfs2/file.c:1157
call_write_iter include/linux/fs.h:2265 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&ip->i_rw_mutex);
lock(&sdp->sd_quota_mutex);
lock(&ip->i_rw_mutex);
lock(&sdp->sd_quota_mutex);

*** DEADLOCK ***

6 locks held by syz-executor.1/4341:
#0: ffff0000d322a368 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x124/0x16c fs/file.c:1062
#1: ffff0000d50fa460 (sb_writers#14){.+.+}-{0:0}, at: vfs_write+0x244/0x914 fs/read_write.c:580
#2: ffff0000f2320150 (&sb->s_type->i_mutex_key#22){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:758 [inline]
#2: ffff0000f2320150 (&sb->s_type->i_mutex_key#22){+.+.}-{3:3}, at: gfs2_file_write_iter+0x2cc/0xc3c fs/gfs2/file.c:1114
#3: ffff0000d50fa650 (sb_internal#4){.+.+}-{0:0}, at: gfs2_trans_begin+0xa8/0x11c fs/gfs2/trans.c:118
#4: ffff0000f8129058 (&sdp->sd_log_flush_lock){.+.+}-{3:3}, at: __gfs2_trans_begin+0x524/0x91c fs/gfs2/trans.c:87
#5: ffff0000f23205e8 (&ip->i_rw_mutex){++++}-{3:3}, at: sweep_bh_for_rgrps fs/gfs2/bmap.c:1512 [inline]
#5: ffff0000f23205e8 (&ip->i_rw_mutex){++++}-{3:3}, at: punch_hole+0x1fb0/0x2ff0 fs/gfs2/bmap.c:1836

stack backtrace:
CPU: 0 PID: 4341 Comm: syz-executor.1 Not tainted 6.1.88-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024

Call trace:
dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
dump_stack+0x1c/0x5c lib/dump_stack.c:113
print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2048
check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2170
check_prev_add kernel/locking/lockdep.c:3090 [inline]
check_prevs_add kernel/locking/lockdep.c:3209 [inline]
validate_chain kernel/locking/lockdep.c:3825 [inline]
__lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
__mutex_lock_common+0x190/0x21a0 kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
do_qc+0xbc/0x664 fs/gfs2/quota.c:682
gfs2_quota_change+0x238/0x718 fs/gfs2/quota.c:1295
punch_hole+0x2aa8/0x2ff0 fs/gfs2/bmap.c:1937
gfs2_iomap_end+0x448/0x618 fs/gfs2/bmap.c:1157
iomap_iter+0x1e0/0x112c fs/iomap/iter.c:62
iomap_file_buffered_write+0x734/0x8c0 fs/iomap/buffered-io.c:822
gfs2_file_buffered_write+0x49c/0x800 fs/gfs2/file.c:1056
gfs2_file_write_iter+0x3d4/0xc3c fs/gfs2/file.c:1157
call_write_iter include/linux/fs.h:2265 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x610/0x914 fs/read_write.c:584
ksys_write+0x15c/0x26c fs/read_write.c:637
__do_sys_write fs/read_write.c:649 [inline]
__se_sys_write fs/read_write.c:646 [inline]
__arm64_sys_write+0x7c/0x90 fs/read_write.c:646
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585