[syzbot] [gfs2?] kernel BUG in gfs2_quota_cleanup

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 65d6e954e378 Merge tag 'gfs2-v6.5-rc5-fixes' of git://git...
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=150b1d58680000
kernel config: https://syzkaller.appspot.com/x/.config?x=cbf8b29a87b8a830
dashboard link: https://syzkaller.appspot.com/bug?extid=3b6e67ac2b646da57862
compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/a239a86394dd/disk-65d6e954.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/1d7af042d4c5/vmlinux-65d6e954.xz
kernel image: https://storage.googleapis.com/syzbot-assets/cdb5df52c1e3/bzImage-65d6e954.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+3b6e67...@syzkaller.appspotmail.com

RBP: 00007ffd0d57a210 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000246 R12: 00007ffd0d57b2d0
R13: 00007fbf640c73b9 R14: 00000000002ca96c R15: 0000000000000003
</TASK>
------------[ cut here ]------------
kernel BUG at fs/gfs2/quota.c:1485!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 5074 Comm: syz-executor.4 Not tainted 6.5.0-syzkaller-11938-g65d6e954e378 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
RIP: 0010:gfs2_quota_cleanup+0x5cc/0x6e0 fs/gfs2/quota.c:1485
Code: ff e8 e8 3b e1 fd e8 73 40 e1 fd 49 8d b7 70 12 00 00 89 ea 48 c7 c7 a0 16 da 8a e8 be 5b c4 fd e9 1b fe ff ff e8 54 40 e1 fd <0f> 0b 4c 89 e7 e8 2a 3d 36 fe e9 ef fd ff ff e8 40 3d 36 fe e9 14
RSP: 0018:ffffc900038bfaf0 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff888076948200 RSI: ffffffff83a5622c RDI: 0000000000000007
RBP: dffffc0000000000 R08: 0000000000000007 R09: 0000000000000000
R10: 0000000000000001 R11: 000000000008c8e8 R12: 0000000000000000
R13: 0000000000000000 R14: ffffc900038bfb40 R15: ffff888036950000
FS: 0000555556145480(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68df122866 CR3: 000000003214a000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
gfs2_make_fs_ro+0x11e/0x350 fs/gfs2/super.c:574
gfs2_put_super+0x66a/0x760 fs/gfs2/super.c:606
generic_shutdown_super+0x161/0x3c0 fs/super.c:693
kill_block_super+0x3b/0x70 fs/super.c:1646
gfs2_kill_sb+0x361/0x410 fs/gfs2/ops_fstype.c:1811
deactivate_locked_super+0x9a/0x170 fs/super.c:481
deactivate_super+0xde/0x100 fs/super.c:514
cleanup_mnt+0x222/0x3d0 fs/namespace.c:1254
task_work_run+0x14d/0x240 kernel/task_work.c:179
resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
exit_to_user_mode_prepare+0x210/0x240 kernel/entry/common.c:204
__syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
syscall_exit_to_user_mode+0x1d/0x60 kernel/entry/common.c:296
do_syscall_64+0x44/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fbf6407de17
Code: b0 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 c7 c2 b0 ff ff ff f7 d8 64 89 02 b8
RSP: 002b:00007ffd0d57a158 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fbf6407de17
RDX: 0000000000000000 RSI: 000000000000000a RDI: 00007ffd0d57a210
RBP: 00007ffd0d57a210 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000246 R12: 00007ffd0d57b2d0
R13: 00007fbf640c73b9 R14: 00000000002ca96c R15: 0000000000000003
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:gfs2_quota_cleanup+0x5cc/0x6e0 fs/gfs2/quota.c:1485
Code: ff e8 e8 3b e1 fd e8 73 40 e1 fd 49 8d b7 70 12 00 00 89 ea 48 c7 c7 a0 16 da 8a e8 be 5b c4 fd e9 1b fe ff ff e8 54 40 e1 fd <0f> 0b 4c 89 e7 e8 2a 3d 36 fe e9 ef fd ff ff e8 40 3d 36 fe e9 14
RSP: 0018:ffffc900038bfaf0 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff888076948200 RSI: ffffffff83a5622c RDI: 0000000000000007
RBP: dffffc0000000000 R08: 0000000000000007 R09: 0000000000000000
R10: 0000000000000001 R11: 000000000008c8e8 R12: 0000000000000000
R13: 0000000000000000 R14: ffffc900038bfb40 R15: ffff888036950000
FS: 0000555556145480(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68df122866 CR3: 000000003214a000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 994d5c58e50e Merge tag 'hardening-v6.7-rc4' of git://git.k..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=165bc8d2e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=c2c74446ab4f0028
dashboard link: https://syzkaller.appspot.com/bug?extid=3b6e67ac2b646da57862
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1268c086e80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=164b3faae80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/dceb288ab9b9/disk-994d5c58.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/00a42d6cc05c/vmlinux-994d5c58.xz
kernel image: https://storage.googleapis.com/syzbot-assets/039fd7f1b283/bzImage-994d5c58.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/4022e7b58a1d/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+3b6e67...@syzkaller.appspotmail.com

gfs2: fsid=syz:syz.0: first mount done, others may mount
------------[ cut here ]------------
kernel BUG at fs/gfs2/quota.c:1508!

invalid opcode: 0000 [#1] PREEMPT SMP KASAN

CPU: 0 PID: 5060 Comm: syz-executor505 Not tainted 6.7.0-rc3-syzkaller-00134-g994d5c58e50e #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
RIP: 0010:gfs2_quota_cleanup+0x6b5/0x6c0 fs/gfs2/quota.c:1508
Code: fe e9 cf fd ff ff 44 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c 2d fe ff ff 4c 89 ef e8 b6 19 23 fe e9 20 fe ff ff e8 ec 11 c7 fd 90 <0f> 0b e8 84 9c 4f 07 0f 1f 40 00 66 0f 1f 00 55 41 57 41 56 41 54
RSP: 0018:ffffc9000409f9e0 EFLAGS: 00010293
RAX: ffffffff83c76854 RBX: 0000000000000002 RCX: ffff888026001dc0
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: ffffc9000409fb00 R08: ffffffff83c762b0 R09: 1ffff1100fd38015
R10: dffffc0000000000 R11: ffffed100fd38016 R12: dffffc0000000000
R13: ffff88807e9c0828 R14: ffff888014693580 R15: ffff88807e9c0000
FS: 0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f16d1bd70f8 CR3: 0000000027199000 CR4: 00000000003506f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>

gfs2_put_super+0x2e1/0x940 fs/gfs2/super.c:611
generic_shutdown_super+0x13a/0x2c0 fs/super.c:696
kill_block_super+0x44/0x90 fs/super.c:1667
deactivate_locked_super+0xc1/0x130 fs/super.c:484
cleanup_mnt+0x426/0x4c0 fs/namespace.c:1256
task_work_run+0x24a/0x300 kernel/task_work.c:180
exit_task_work include/linux/task_work.h:38 [inline]
do_exit+0xa34/0x2750 kernel/exit.c:871
do_group_exit+0x206/0x2c0 kernel/exit.c:1021
__do_sys_exit_group kernel/exit.c:1032 [inline]
__se_sys_exit_group kernel/exit.c:1030 [inline]
__x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1030
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x45/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f16d1b4b789
Code: Unable to access opcode bytes at 0x7f16d1b4b75f.
RSP: 002b:00007ffcad68ef38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f16d1b4b789
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000001
RBP: 00007f16d1bd92d0 R08: ffffffffffffffb8 R09: 00007ffcad68f010
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f16d1bd92d0
R13: 0000000000000000 R14: 00007f16d1bda040 R15: 00007f16d1b19cc0

</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---

RIP: 0010:gfs2_quota_cleanup+0x6b5/0x6c0 fs/gfs2/quota.c:1508
Code: fe e9 cf fd ff ff 44 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c 2d fe ff ff 4c 89 ef e8 b6 19 23 fe e9 20 fe ff ff e8 ec 11 c7 fd 90 <0f> 0b e8 84 9c 4f 07 0f 1f 40 00 66 0f 1f 00 55 41 57 41 56 41 54
RSP: 0018:ffffc9000409f9e0 EFLAGS: 00010293
RAX: ffffffff83c76854 RBX: 0000000000000002 RCX: ffff888026001dc0
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: ffffc9000409fb00 R08: ffffffff83c762b0 R09: 1ffff1100fd38015
R10: dffffc0000000000 R11: ffffed100fd38016 R12: dffffc0000000000
R13: ffff88807e9c0828 R14: ffff888014693580 R15: ffff88807e9c0000
FS: 0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f16d1bd70f8 CR3: 000000000d730000 CR4: 00000000003506f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

[Edward Adam Davis]

please test kernel BUG in gfs2_quota_cleanup

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 994d5c58e50e

diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c
index 95dae7838b4e..af32dd8a72fa 100644
--- a/fs/gfs2/quota.c
+++ b/fs/gfs2/quota.c
@@ -1505,7 +1505,8 @@ void gfs2_quota_cleanup(struct gfs2_sbd *sdp)
LIST_HEAD(dispose);
int count;

- BUG_ON(test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));
+ BUG_ON(!test_bit(SDF_NORECOVERY, &sdp->sd_flags) &&
+ test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));

spin_lock(&qd_lock);
list_for_each_entry(qd, &sdp->sd_quota_list, qd_list) {

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+3b6e67...@syzkaller.appspotmail.com

Tested on:

commit: 994d5c58 Merge tag 'hardening-v6.7-rc4' of git://git.k..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=159d7eb4e80000

kernel config: https://syzkaller.appspot.com/x/.config?x=c2c74446ab4f0028
dashboard link: https://syzkaller.appspot.com/bug?extid=3b6e67ac2b646da57862
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

patch: https://syzkaller.appspot.com/x/patch.diff?x=15c45f62e80000

Note: testing is done by a robot and is best-effort only.

[Edward Adam Davis]

[Syz report]

...

[pid 5060] fsconfig(4, FSCONFIG_CMD_RECONFIGURE, NULL, NULL, 0) = 0
[pid 5060] exit_group(1) = ?
...

[Analysis]
When the task exits, it will execute cleanup_mnt() to recycle the mounted gfs2
file system, but it performs a system call fsconfig(4, FSCONFIG_CMD_RECONFIGURE,
NULL, NULL, 0) before executing the task exit operation.

This will execute the following kernel path to complete the setting of
SDF_JOURNAL_LIVE for sd_flags:

SYSCALL_DEFINE5(fsconfig, ..)->
vfs_fsconfig_locked()->
vfs_cmd_reconfigure()->
gfs2_reconfigure()->
gfs2_make_fs_rw()->
set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);

[Fix]
Add SDF_NORECOVERY check in gfs2_quota_cleanup() to avoid checking
SDF_JOURNAL_LIVE on the path where gfs2 is being unmounted.

Reported-and-tested-by: syzbot+3b6e67...@syzkaller.appspotmail.com
Signed-off-by: Edward Adam Davis <ead...@qq.com>
---
fs/gfs2/quota.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c
index 95dae7838b4e..af32dd8a72fa 100644
--- a/fs/gfs2/quota.c
+++ b/fs/gfs2/quota.c
@@ -1505,7 +1505,8 @@ void gfs2_quota_cleanup(struct gfs2_sbd *sdp)
LIST_HEAD(dispose);
int count;

- BUG_ON(test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));
+ BUG_ON(!test_bit(SDF_NORECOVERY, &sdp->sd_flags) &&
+ test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));

spin_lock(&qd_lock);
list_for_each_entry(qd, &sdp->sd_quota_list, qd_list) {

--
2.43.0

[syzbot]

syzbot has bisected this issue to:

commit bdcb8aa434c6d36b5c215d02a9ef07551be25a37
Author: Juntong Deng <junton...@outlook.com>
Date: Sun Oct 29 21:10:06 2023 +0000

gfs2: Fix slab-use-after-free in gfs2_qd_dealloc

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=169c7b52e80000
start commit: 994d5c58e50e Merge tag 'hardening-v6.7-rc4' of git://git.k..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=159c7b52e80000
console output: https://syzkaller.appspot.com/x/log.txt?x=119c7b52e80000

kernel config: https://syzkaller.appspot.com/x/.config?x=c2c74446ab4f0028
dashboard link: https://syzkaller.appspot.com/bug?extid=3b6e67ac2b646da57862

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1268c086e80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=164b3faae80000

Reported-by: syzbot+3b6e67...@syzkaller.appspotmail.com
Fixes: bdcb8aa434c6 ("gfs2: Fix slab-use-after-free in gfs2_qd_dealloc")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Andreas Gruenbacher]

On Sat, Dec 2, 2023 at 10:34 AM Edward Adam Davis <ead...@qq.com> wrote:
> [Analysis]
> When the task exits, it will execute cleanup_mnt() to recycle the mounted gfs2
> file system, but it performs a system call fsconfig(4, FSCONFIG_CMD_RECONFIGURE,
> NULL, NULL, 0) before executing the task exit operation.
>
> This will execute the following kernel path to complete the setting of
> SDF_JOURNAL_LIVE for sd_flags:
>
> SYSCALL_DEFINE5(fsconfig, ..)->
> vfs_fsconfig_locked()->
> vfs_cmd_reconfigure()->
> gfs2_reconfigure()->
> gfs2_make_fs_rw()->
> set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
>
> [Fix]
> Add SDF_NORECOVERY check in gfs2_quota_cleanup() to avoid checking
> SDF_JOURNAL_LIVE on the path where gfs2 is being unmounted.

Thanks for this fix, I've applied it and added the following tag:

Fixes: f66af88e3321 ("gfs2: Stop using gfs2_make_fs_ro for withdraw")

>
> Reported-and-tested-by: syzbot+3b6e67...@syzkaller.appspotmail.com
> Signed-off-by: Edward Adam Davis <ead...@qq.com>
> ---
> fs/gfs2/quota.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c
> index 95dae7838b4e..af32dd8a72fa 100644
> --- a/fs/gfs2/quota.c
> +++ b/fs/gfs2/quota.c
> @@ -1505,7 +1505,8 @@ void gfs2_quota_cleanup(struct gfs2_sbd *sdp)
> LIST_HEAD(dispose);
> int count;
>
> - BUG_ON(test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));
> + BUG_ON(!test_bit(SDF_NORECOVERY, &sdp->sd_flags) &&
> + test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags));
>
> spin_lock(&qd_lock);
> list_for_each_entry(qd, &sdp->sd_quota_list, qd_list) {
> --
> 2.43.0

Thanks,
Andreas