[syzbot] [jfs?] INFO: task hung in lmLogClose (3)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 707081b61156 Merge branch 'for-next/core', remote-tracking..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=126c764e180000
kernel config: https://syzkaller.appspot.com/x/.config?x=caeac3f3565b057a
dashboard link: https://syzkaller.appspot.com/bug?extid=c824290332add8067111
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=101e05be180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=151ec2a5180000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/6cad68bf7532/disk-707081b6.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/1a27e5400778/vmlinux-707081b6.xz
kernel image: https://storage.googleapis.com/syzbot-assets/67dfc53755d0/Image-707081b6.gz.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/e14a3937c758/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c82429...@syzkaller.appspotmail.com

INFO: task syz-executor394:6204 blocked for more than 143 seconds.
Not tainted 6.8.0-rc7-syzkaller-g707081b61156 #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor394 state:D stack:0 pid:6204 tgid:6204 ppid:6201 flags:0x0000000c
Call trace:
__switch_to+0x314/0x560 arch/arm64/kernel/process.c:553
context_switch kernel/sched/core.c:5400 [inline]
__schedule+0x1498/0x24b4 kernel/sched/core.c:6727
__schedule_loop kernel/sched/core.c:6802 [inline]
schedule+0xb8/0x19c kernel/sched/core.c:6817
schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:6874
__mutex_lock_common+0xbd8/0x21a0 kernel/locking/mutex.c:684
__mutex_lock kernel/locking/mutex.c:752 [inline]
mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:804
lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444
jfs_umount+0x274/0x360 fs/jfs/jfs_umount.c:114
jfs_put_super+0x90/0x188 fs/jfs/super.c:194
generic_shutdown_super+0x128/0x2b8 fs/super.c:641
kill_block_super+0x44/0x90 fs/super.c:1675
deactivate_locked_super+0xc4/0x12c fs/super.c:472
deactivate_super+0xe0/0x100 fs/super.c:505
cleanup_mnt+0x34c/0x3dc fs/namespace.c:1267
__cleanup_mnt+0x20/0x30 fs/namespace.c:1274
task_work_run+0x230/0x2e0 kernel/task_work.c:180
resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
do_notify_resume+0x178/0x1f4 arch/arm64/kernel/entry-common.c:151
exit_to_user_mode_prepare arch/arm64/kernel/entry-common.c:169 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:178 [inline]
el0_svc+0xac/0x168 arch/arm64/kernel/entry-common.c:713
el0t_64_sync_handler+0x84/0xfc arch/arm64/kernel/entry-common.c:730
el0t_64_sync+0x190/0x194 arch/arm64/kernel/entry.S:598
INFO: task syz-executor394:6205 blocked for more than 143 seconds.
Not tainted 6.8.0-rc7-syzkaller-g707081b61156 #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor394 state:D stack:0 pid:6205 tgid:6205 ppid:6201 flags:0x0000000c
Call trace:
__switch_to+0x314/0x560 arch/arm64/kernel/process.c:553
context_switch kernel/sched/core.c:5400 [inline]
__schedule+0x1498/0x24b4 kernel/sched/core.c:6727
__schedule_loop kernel/sched/core.c:6802 [inline]
schedule+0xb8/0x19c kernel/sched/core.c:6817
schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:6874
__mutex_lock_common+0xbd8/0x21a0 kernel/locking/mutex.c:684
__mutex_lock kernel/locking/mutex.c:752 [inline]
mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:804
lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444
jfs_umount+0x274/0x360 fs/jfs/jfs_umount.c:114
jfs_put_super+0x90/0x188 fs/jfs/super.c:194
generic_shutdown_super+0x128/0x2b8 fs/super.c:641
kill_block_super+0x44/0x90 fs/super.c:1675
deactivate_locked_super+0xc4/0x12c fs/super.c:472
deactivate_super+0xe0/0x100 fs/super.c:505
cleanup_mnt+0x34c/0x3dc fs/namespace.c:1267
__cleanup_mnt+0x20/0x30 fs/namespace.c:1274
task_work_run+0x230/0x2e0 kernel/task_work.c:180
resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
do_notify_resume+0x178/0x1f4 arch/arm64/kernel/entry-common.c:151
exit_to_user_mode_prepare arch/arm64/kernel/entry-common.c:169 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:178 [inline]
el0_svc+0xac/0x168 arch/arm64/kernel/entry-common.c:713
el0t_64_sync_handler+0x84/0xfc arch/arm64/kernel/entry-common.c:730
el0t_64_sync+0x190/0x194 arch/arm64/kernel/entry.S:598

Showing all locks held in the system:
1 lock held by khungtaskd/29:
#0: ffff80008ee74ac0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0xc/0x44 include/linux/rcupdate.h:297
2 locks held by getty/5927:
#0: ffff0000d2e810a0 (&tty->ldisc_sem){++++}-{0:0}, at: ldsem_down_read+0x3c/0x4c drivers/tty/tty_ldsem.c:340
#1: ffff800093fe02f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x41c/0x1228 drivers/tty/n_tty.c:2201
2 locks held by syz-executor394/6202:
#0: ffff0000d98900e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock fs/super.c:56 [inline]
#0: ffff0000d98900e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
#0: ffff0000d98900e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xd8/0x100 fs/super.c:504
#1: ffff80008f2586e8 (jfs_log_mutex){+.+.}-{3:3}, at: lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444
1 lock held by syz-executor394/6203:
#0: ffff0000c25c00e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock fs/super.c:56 [inline]
#0: ffff0000c25c00e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
#0: ffff0000c25c00e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xd8/0x100 fs/super.c:504
2 locks held by syz-executor394/6204:
#0: ffff0000d60740e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock fs/super.c:56 [inline]
#0: ffff0000d60740e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
#0: ffff0000d60740e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xd8/0x100 fs/super.c:504
#1: ffff80008f2586e8 (jfs_log_mutex){+.+.}-{3:3}, at: lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444
2 locks held by syz-executor394/6205:
#0: ffff0000d95420e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock fs/super.c:56 [inline]
#0: ffff0000d95420e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
#0: ffff0000d95420e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xd8/0x100 fs/super.c:504
#1: ffff80008f2586e8 (jfs_log_mutex){+.+.}-{3:3}, at: lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444
2 locks held by syz-executor394/6206:
#0: ffff0000d5cb80e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock fs/super.c:56 [inline]
#0: ffff0000d5cb80e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
#0: ffff0000d5cb80e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xd8/0x100 fs/super.c:504
#1: ffff80008f2586e8 (jfs_log_mutex){+.+.}-{3:3}, at: lmLogClose+0xc8/0x4d4 fs/jfs/jfs_logmgr.c:1444

=============================================



---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Edward Adam Davis]

please test hung in lmLogClose

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

diff --git a/fs/jfs/jfs_incore.h b/fs/jfs/jfs_incore.h
index 10934f9a11be..56d336a49985 100644
--- a/fs/jfs/jfs_incore.h
+++ b/fs/jfs/jfs_incore.h
@@ -197,6 +197,7 @@ struct jfs_sb_info {
kgid_t gid; /* gid to override on-disk gid */
uint umask; /* umask to override on-disk umask */
uint minblks_trim; /* minimum blocks, for online trim */
+ struct mutex simutex;
};

/* jfs_sb_info commit_state */
diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c
index 73389c68e251..b5609a7618e5 100644
--- a/fs/jfs/jfs_logmgr.c
+++ b/fs/jfs/jfs_logmgr.c
@@ -155,7 +155,6 @@ do { \
*/
static LIST_HEAD(jfs_external_logs);
static struct jfs_log *dummy_log;
-static DEFINE_MUTEX(jfs_log_mutex);

/*
* forward references
@@ -1068,19 +1067,19 @@ int lmLogOpen(struct super_block *sb)
if (sbi->mntflag & JFS_INLINELOG)
return open_inline_log(sb);

- mutex_lock(&jfs_log_mutex);
+ mutex_lock(&sbi->simutex);
list_for_each_entry(log, &jfs_external_logs, journal_list) {
if (file_bdev(log->bdev_file)->bd_dev == sbi->logdev) {
if (!uuid_equal(&log->uuid, &sbi->loguuid)) {
jfs_warn("wrong uuid on JFS journal");
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return -EINVAL;
}
/*
* add file system to log active file system list
*/
if ((rc = lmLogFileSystem(log, sbi, 1))) {
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return rc;
}
goto journal_found;
@@ -1088,7 +1087,7 @@ int lmLogOpen(struct super_block *sb)
}

if (!(log = kzalloc(sizeof(struct jfs_log), GFP_KERNEL))) {
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return -ENOMEM;
}
INIT_LIST_HEAD(&log->sb_list);
@@ -1130,7 +1129,7 @@ int lmLogOpen(struct super_block *sb)
sbi->log = log;
LOG_UNLOCK(log);

- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return 0;

/*
@@ -1144,7 +1143,7 @@ int lmLogOpen(struct super_block *sb)
fput(bdev_file);

free: /* free log descriptor */
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
kfree(log);

jfs_warn("lmLogOpen: exit(%d)", rc);
@@ -1187,12 +1186,13 @@ static int open_inline_log(struct super_block *sb)
static int open_dummy_log(struct super_block *sb)
{
int rc;
+ struct jfs_sb_info *sbi = JFS_SBI(sb);

- mutex_lock(&jfs_log_mutex);
+ mutex_lock(&sbi->simutex);
if (!dummy_log) {
dummy_log = kzalloc(sizeof(struct jfs_log), GFP_KERNEL);
if (!dummy_log) {
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return -ENOMEM;
}
INIT_LIST_HEAD(&dummy_log->sb_list);
@@ -1205,7 +1205,7 @@ static int open_dummy_log(struct super_block *sb)
if (rc) {
kfree(dummy_log);
dummy_log = NULL;
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
return rc;
}
}
@@ -1214,7 +1214,7 @@ static int open_dummy_log(struct super_block *sb)
list_add(&JFS_SBI(sb)->log_list, &dummy_log->sb_list);
JFS_SBI(sb)->log = dummy_log;
LOG_UNLOCK(dummy_log);
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);

return 0;
}
@@ -1441,7 +1441,7 @@ int lmLogClose(struct super_block *sb)

jfs_info("lmLogClose: log:0x%p", log);

- mutex_lock(&jfs_log_mutex);
+ mutex_lock(&sbi->simutex);
LOG_LOCK(log);
list_del(&sbi->log_list);
LOG_UNLOCK(log);
@@ -1490,7 +1490,7 @@ int lmLogClose(struct super_block *sb)
kfree(log);

out:
- mutex_unlock(&jfs_log_mutex);
+ mutex_unlock(&sbi->simutex);
jfs_info("lmLogClose: exit(%d)", rc);
return rc;
}
diff --git a/fs/jfs/super.c b/fs/jfs/super.c
index e1be21ca5d6e..23628ca3990c 100644
--- a/fs/jfs/super.c
+++ b/fs/jfs/super.c
@@ -504,6 +504,7 @@ static int jfs_fill_super(struct super_block *sb, void *data, int silent)
sbi->uid = INVALID_UID;
sbi->gid = INVALID_GID;
sbi->umask = -1;
+ mutex_init(&sbi->simutex);

/* initialize the mount flag and determine the default error handler */
flag = JFS_ERR_REMOUNT_RO;

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+c82429...@syzkaller.appspotmail.com

Tested on:

commit: 712e1425 Merge tag 'xfs-6.9-fixes-1' of git://git.kern..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=10f2512d180000
kernel config: https://syzkaller.appspot.com/x/.config?x=ffb07c9738b12db3

dashboard link: https://syzkaller.appspot.com/bug?extid=c824290332add8067111
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

patch: https://syzkaller.appspot.com/x/patch.diff?x=121abb76180000

Note: testing is done by a robot and is best-effort only.

[Edward Adam Davis]

[Syzbot reported]

[Fix]
The jfs_log_mutex cannot distinguish different superblocks, resulting in multiple
different superblocks competing for jfs_log_mutex.

Reported-and-tested-by: syzbot+c82429...@syzkaller.appspotmail.com
Signed-off-by: Edward Adam Davis <ead...@qq.com>
---
fs/jfs/jfs_incore.h | 1 +
fs/jfs/jfs_logmgr.c | 26 +++++++++++++-------------
fs/jfs/super.c | 1 +
3 files changed, 15 insertions(+), 13 deletions(-)

--
2.43.0