[v5.15] possible deadlock in btrfs_run_dev_stats

0 views
Skip to first unread message

syzbot

unread,
May 1, 2024, 10:20:52 PMMay 1
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: b925f60c6ee7 Linux 5.15.157
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=146cb2a0980000
kernel config: https://syzkaller.appspot.com/x/.config?x=802c8e49b2826f05
dashboard link: https://syzkaller.appspot.com/bug?extid=ee7f25e27b96da12819f
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/add896a77889/disk-b925f60c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/e24f83a46999/vmlinux-b925f60c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/de040a426c29/Image-b925f60c.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+ee7f25...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
5.15.157-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.3/3981 is trying to acquire lock:
ffff0000d250c4e0 (&fs_devs->device_list_mutex){+.+.}-{3:3}, at: btrfs_run_dev_stats+0xe4/0xe24 fs/btrfs/volumes.c:7941

but task is already holding lock:
ffff0000ead3c720 (&fs_info->tree_log_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (&fs_info->tree_log_mutex){+.+.}-{3:3}:
__mutex_lock_common+0x194/0x2154 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0xa4/0xf8 kernel/locking/mutex.c:743
btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367
btrfs_sync_fs+0x678/0xc94 fs/btrfs/super.c:1454
sync_filesystem+0x1a0/0x218 fs/sync.c:66
generic_shutdown_super+0x70/0x29c fs/super.c:448
kill_anon_super+0x4c/0x74 fs/super.c:1076
btrfs_kill_super+0x40/0x58 fs/btrfs/super.c:2391
deactivate_locked_super+0xb8/0x13c fs/super.c:335
deactivate_super+0x108/0x128 fs/super.c:366
cleanup_mnt+0x3c0/0x474 fs/namespace.c:1143
__cleanup_mnt+0x20/0x30 fs/namespace.c:1150
task_work_run+0x130/0x1e4 kernel/task_work.c:164
exit_task_work include/linux/task_work.h:32 [inline]
do_exit+0x670/0x20bc kernel/exit.c:872
do_group_exit+0x110/0x268 kernel/exit.c:994
__do_sys_exit_group kernel/exit.c:1005 [inline]
__se_sys_exit_group kernel/exit.c:1003 [inline]
__wake_up_parent+0x0/0x60 kernel/exit.c:1003
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #3 (&fs_info->reloc_mutex){+.+.}-{3:3}:
__mutex_lock_common+0x194/0x2154 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0xa4/0xf8 kernel/locking/mutex.c:743
btrfs_record_root_in_trans fs/btrfs/transaction.c:511 [inline]
start_transaction+0x450/0x1480 fs/btrfs/transaction.c:748
btrfs_join_transaction+0x30/0x40 fs/btrfs/transaction.c:791
btrfs_dirty_inode+0xb4/0x1c0 fs/btrfs/inode.c:6428
btrfs_update_time+0x248/0x270 fs/btrfs/inode.c:6470
inode_update_time fs/inode.c:1829 [inline]
touch_atime+0x434/0xa4c fs/inode.c:1902
file_accessed include/linux/fs.h:2516 [inline]
iterate_dir+0x400/0x4e4 fs/readdir.c:70
__do_sys_getdents64 fs/readdir.c:369 [inline]
__se_sys_getdents64 fs/readdir.c:354 [inline]
__arm64_sys_getdents64+0x1c4/0x4c4 fs/readdir.c:354
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #2 (sb_internal#2){.+.+}-{0:0}:
percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
__sb_start_write include/linux/fs.h:1811 [inline]
sb_start_intwrite include/linux/fs.h:1928 [inline]
start_transaction+0x644/0x1480 fs/btrfs/transaction.c:678
btrfs_join_transaction+0x30/0x40 fs/btrfs/transaction.c:791
btrfs_dirty_inode+0xb4/0x1c0 fs/btrfs/inode.c:6428
btrfs_update_time+0x248/0x270 fs/btrfs/inode.c:6470
inode_update_time fs/inode.c:1829 [inline]
touch_atime+0x434/0xa4c fs/inode.c:1902
file_accessed include/linux/fs.h:2516 [inline]
iterate_dir+0x400/0x4e4 fs/readdir.c:70
__do_sys_getdents64 fs/readdir.c:369 [inline]
__se_sys_getdents64 fs/readdir.c:354 [inline]
__arm64_sys_getdents64+0x1c4/0x4c4 fs/readdir.c:354
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #1 (&type->i_mutex_dir_key#8){++++}-{3:3}:
down_read+0xc0/0x398 kernel/locking/rwsem.c:1498
inode_lock_shared include/linux/fs.h:799 [inline]
lookup_slow+0x50/0x84 fs/namei.c:1679
walk_component+0x394/0x4cc fs/namei.c:1976
lookup_last fs/namei.c:2431 [inline]
path_lookupat+0x13c/0x3d0 fs/namei.c:2455
filename_lookup+0x1c4/0x4c8 fs/namei.c:2484
kern_path+0x4c/0x194 fs/namei.c:2582
lookup_bdev+0xc0/0x25c block/bdev.c:979
device_matched fs/btrfs/volumes.c:568 [inline]
btrfs_free_stale_devices+0x658/0x9ec fs/btrfs/volumes.c:608
btrfs_forget_devices+0x5c/0x98 fs/btrfs/volumes.c:1388
btrfs_control_ioctl+0x12c/0x248 fs/btrfs/super.c:2451
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:874 [inline]
__se_sys_ioctl fs/ioctl.c:860 [inline]
__arm64_sys_ioctl+0x14c/0x1c8 fs/ioctl.c:860
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #0 (&fs_devs->device_list_mutex){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x32d4/0x7638 kernel/locking/lockdep.c:5012
lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5623
__mutex_lock_common+0x194/0x2154 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0xa4/0xf8 kernel/locking/mutex.c:743
btrfs_run_dev_stats+0xe4/0xe24 fs/btrfs/volumes.c:7941
commit_cowonly_roots+0x17c/0x808 fs/btrfs/transaction.c:1290
btrfs_commit_transaction+0xef4/0x30f0 fs/btrfs/transaction.c:2392
btrfs_sync_fs+0x678/0xc94 fs/btrfs/super.c:1454
sync_filesystem+0x1a0/0x218 fs/sync.c:66
generic_shutdown_super+0x70/0x29c fs/super.c:448
kill_anon_super+0x4c/0x74 fs/super.c:1076
btrfs_kill_super+0x40/0x58 fs/btrfs/super.c:2391
deactivate_locked_super+0xb8/0x13c fs/super.c:335
deactivate_super+0x108/0x128 fs/super.c:366
cleanup_mnt+0x3c0/0x474 fs/namespace.c:1143
__cleanup_mnt+0x20/0x30 fs/namespace.c:1150
task_work_run+0x130/0x1e4 kernel/task_work.c:164
exit_task_work include/linux/task_work.h:32 [inline]
do_exit+0x670/0x20bc kernel/exit.c:872
do_group_exit+0x110/0x268 kernel/exit.c:994
__do_sys_exit_group kernel/exit.c:1005 [inline]
__se_sys_exit_group kernel/exit.c:1003 [inline]
__wake_up_parent+0x0/0x60 kernel/exit.c:1003
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

other info that might help us debug this:

Chain exists of:
&fs_devs->device_list_mutex --> &fs_info->reloc_mutex --> &fs_info->tree_log_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&fs_info->tree_log_mutex);
lock(&fs_info->reloc_mutex);
lock(&fs_info->tree_log_mutex);
lock(&fs_devs->device_list_mutex);

*** DEADLOCK ***

3 locks held by syz-executor.3/3981:
#0: ffff0000c155c0e0 (&type->s_umount_key#51){+.+.}-{3:3}, at: deactivate_super+0x100/0x128 fs/super.c:365
#1: ffff0000ead3cc90 (&fs_info->reloc_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xd9c/0x30f0 fs/btrfs/transaction.c:2317
#2: ffff0000ead3c720 (&fs_info->tree_log_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367

stack backtrace:
CPU: 0 PID: 3981 Comm: syz-executor.3 Not tainted 5.15.157-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call trace:
dump_backtrace+0x0/0x530 arch/arm64/kernel/stacktrace.c:152
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:216
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
dump_stack+0x1c/0x58 lib/dump_stack.c:113
print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2011
check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2133
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x32d4/0x7638 kernel/locking/lockdep.c:5012
lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5623
__mutex_lock_common+0x194/0x2154 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0xa4/0xf8 kernel/locking/mutex.c:743
btrfs_run_dev_stats+0xe4/0xe24 fs/btrfs/volumes.c:7941
commit_cowonly_roots+0x17c/0x808 fs/btrfs/transaction.c:1290
btrfs_commit_transaction+0xef4/0x30f0 fs/btrfs/transaction.c:2392
btrfs_sync_fs+0x678/0xc94 fs/btrfs/super.c:1454
sync_filesystem+0x1a0/0x218 fs/sync.c:66
generic_shutdown_super+0x70/0x29c fs/super.c:448
kill_anon_super+0x4c/0x74 fs/super.c:1076
btrfs_kill_super+0x40/0x58 fs/btrfs/super.c:2391
deactivate_locked_super+0xb8/0x13c fs/super.c:335
deactivate_super+0x108/0x128 fs/super.c:366
cleanup_mnt+0x3c0/0x474 fs/namespace.c:1143
__cleanup_mnt+0x20/0x30 fs/namespace.c:1150
task_work_run+0x130/0x1e4 kernel/task_work.c:164
exit_task_work include/linux/task_work.h:32 [inline]
do_exit+0x670/0x20bc kernel/exit.c:872
do_group_exit+0x110/0x268 kernel/exit.c:994
__do_sys_exit_group kernel/exit.c:1005 [inline]
__se_sys_exit_group kernel/exit.c:1003 [inline]
__wake_up_parent+0x0/0x60 kernel/exit.c:1003
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:608
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

syzbot

unread,
May 1, 2024, 11:17:37 PMMay 1
to syzkaller...@googlegroups.com
syzbot has found a reproducer for the following issue on:

HEAD commit: b925f60c6ee7 Linux 5.15.157
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=140e8c40980000
kernel config: https://syzkaller.appspot.com/x/.config?x=802c8e49b2826f05
dashboard link: https://syzkaller.appspot.com/bug?extid=ee7f25e27b96da12819f
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=145fac40980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11eeb317180000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/add896a77889/disk-b925f60c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/e24f83a46999/vmlinux-b925f60c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/de040a426c29/Image-b925f60c.gz.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/e700c8eb0e1a/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+ee7f25...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
5.15.157-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor327/3960 is trying to acquire lock:
ffff0000cf2f48e0 (&fs_devs->device_list_mutex){+.+.}-{3:3}, at: btrfs_run_dev_stats+0xe4/0xe24 fs/btrfs/volumes.c:7941

but task is already holding lock:
ffff0000d6200720 (&fs_info->tree_log_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (&fs_info->tree_log_mutex){+.+.}-{3:3}:
__mutex_lock_common+0x194/0x2154 kernel/locking/mutex.c:596
__mutex_lock kernel/locking/mutex.c:729 [inline]
mutex_lock_nested+0xa4/0xf8 kernel/locking/mutex.c:743
btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367
btrfs_sync_fs+0x678/0xc94 fs/btrfs/super.c:1454
sync_filesystem+0x1a0/0x218 fs/sync.c:66
generic_shutdown_super+0x70/0x29c fs/super.c:448
kill_anon_super+0x4c/0x74 fs/super.c:1076
btrfs_kill_super+0x40/0x58 fs/btrfs/super.c:2391
deactivate_locked_super+0xb8/0x13c fs/super.c:335
deactivate_super+0x108/0x128 fs/super.c:366
cleanup_mnt+0x3c0/0x474 fs/namespace.c:1143
__cleanup_mnt+0x20/0x30 fs/namespace.c:1150
task_work_run+0x130/0x1e4 kernel/task_work.c:164
tracehook_notify_resume include/linux/tracehook.h:189 [inline]
do_notify_resume+0x262c/0x32b8 arch/arm64/kernel/signal.c:946
prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:609
-> #1 (&type->i_mutex_dir_key#6){++++}-{3:3}:
tracehook_notify_resume include/linux/tracehook.h:189 [inline]
do_notify_resume+0x262c/0x32b8 arch/arm64/kernel/signal.c:946
prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:609
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

other info that might help us debug this:

Chain exists of:
&fs_devs->device_list_mutex --> &fs_info->reloc_mutex --> &fs_info->tree_log_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&fs_info->tree_log_mutex);
lock(&fs_info->reloc_mutex);
lock(&fs_info->tree_log_mutex);
lock(&fs_devs->device_list_mutex);

*** DEADLOCK ***

3 locks held by syz-executor327/3960:
#0: ffff0000dc3ee0e0 (&type->s_umount_key#41){+.+.}-{3:3}, at: deactivate_super+0x100/0x128 fs/super.c:365
#1: ffff0000d6200c90 (&fs_info->reloc_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xd9c/0x30f0 fs/btrfs/transaction.c:2317
#2: ffff0000d6200720 (&fs_info->tree_log_mutex){+.+.}-{3:3}, at: btrfs_commit_transaction+0xe34/0x30f0 fs/btrfs/transaction.c:2367

stack backtrace:
CPU: 0 PID: 3960 Comm: syz-executor327 Not tainted 5.15.157-syzkaller #0
tracehook_notify_resume include/linux/tracehook.h:189 [inline]
do_notify_resume+0x262c/0x32b8 arch/arm64/kernel/signal.c:946
prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:609
el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.
Reply all
Reply to author
Forward
0 new messages