possible deadlock in btrfs_run_dev_stats

5 views
Skip to first unread message

syzbot

unread,
Nov 29, 2022, 3:35:48 PM11/29/22
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: 3f8a27f9e27b Linux 4.19.211
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=1576165b880000
kernel config: https://syzkaller.appspot.com/x/.config?x=9b9277b418617afe
dashboard link: https://syzkaller.appspot.com/bug?extid=307e0a07468114fb5e2e
compiler: gcc version 10.2.1 20210110 (Debian 10.2.1-6)

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/98c0bdb4abb3/disk-3f8a27f9.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/ea228ff02669/vmlinux-3f8a27f9.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+307e0a...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
4.19.211-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.2/8141 is trying to acquire lock:
0000000096052743 (&fs_devs->device_list_mutex){+.+.}, at: btrfs_run_dev_stats+0xbb/0xa80 fs/btrfs/volumes.c:7111

but task is already holding lock:
00000000f3c75895 (&fs_info->tree_log_mutex){+.+.}, at: btrfs_commit_transaction+0x8c2/0x2480 fs/btrfs/transaction.c:2176

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #6 (&fs_info->tree_log_mutex){+.+.}:
btrfs_commit_transaction+0x8c2/0x2480 fs/btrfs/transaction.c:2176
btrfs_sync_fs+0x120/0x490 fs/btrfs/super.c:1279
__sync_filesystem fs/sync.c:39 [inline]
sync_filesystem+0x1b5/0x250 fs/sync.c:67
generic_shutdown_super+0x70/0x370 fs/super.c:442
kill_anon_super+0x36/0x60 fs/super.c:1032
btrfs_kill_super+0x49/0x550 fs/btrfs/super.c:2221
deactivate_locked_super+0x94/0x160 fs/super.c:329
deactivate_super+0x174/0x1a0 fs/super.c:360
cleanup_mnt+0x1a8/0x290 fs/namespace.c:1098
task_work_run+0x148/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #5 (&fs_info->reloc_mutex){+.+.}:
btrfs_record_root_in_trans+0x122/0x190 fs/btrfs/transaction.c:394
start_transaction+0x234/0xf90 fs/btrfs/transaction.c:586
btrfs_create+0x105/0x670 fs/btrfs/inode.c:6683
lookup_open+0x893/0x1a20 fs/namei.c:3235
do_last fs/namei.c:3327 [inline]
path_openat+0x1094/0x2df0 fs/namei.c:3537
do_filp_open+0x18c/0x3f0 fs/namei.c:3567
do_sys_open+0x3b3/0x520 fs/open.c:1085
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #4 (sb_internal#4){.+.+}:
sb_start_intwrite include/linux/fs.h:1626 [inline]
start_transaction+0xa37/0xf90 fs/btrfs/transaction.c:528
cow_file_range_inline.constprop.0+0x2f8/0xf90 fs/btrfs/inode.c:309
cow_file_range.constprop.0+0x2af/0xa30 fs/btrfs/inode.c:1010
run_delalloc_nocow+0x8fe/0x14d0 fs/btrfs/inode.c:1597
btrfs_run_delalloc_range+0xb11/0xdb0 fs/btrfs/inode.c:1653
writepage_delalloc+0x1d0/0x510 fs/btrfs/extent_io.c:3269
__extent_writepage+0x447/0xcc0 fs/btrfs/extent_io.c:3522
extent_write_cache_pages.constprop.0+0x6e6/0x1120 fs/btrfs/extent_io.c:4107
extent_writepages+0x104/0x1a0 fs/btrfs/extent_io.c:4232
do_writepages+0xe5/0x290 mm/page-writeback.c:2344
__writeback_single_inode+0x10c/0x11d0 fs/fs-writeback.c:1385
writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
wb_writeback+0x28d/0xcc0 fs/fs-writeback.c:1820
wb_do_writeback fs/fs-writeback.c:1965 [inline]
wb_workfn+0x29b/0x1250 fs/fs-writeback.c:2006
process_one_work+0x864/0x1570 kernel/workqueue.c:2153
worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #3 ((work_completion)(&(&wb->dwork)->work)){+.+.}:
wb_shutdown+0x172/0x210 mm/backing-dev.c:374
bdi_unregister+0x169/0x610 mm/backing-dev.c:946
del_gendisk+0x7f6/0xa80 block/genhd.c:788
loop_remove drivers/block/loop.c:2066 [inline]
loop_control_ioctl drivers/block/loop.c:2165 [inline]
loop_control_ioctl+0x3b1/0x480 drivers/block/loop.c:2131
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:501 [inline]
do_vfs_ioctl+0xcdb/0x12e0 fs/ioctl.c:688
ksys_ioctl+0x9b/0xc0 fs/ioctl.c:705
__do_sys_ioctl fs/ioctl.c:712 [inline]
__se_sys_ioctl fs/ioctl.c:710 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:710
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (loop_ctl_mutex){+.+.}:
lo_open+0x19/0xd0 drivers/block/loop.c:1771
__blkdev_get+0x372/0x1480 fs/block_dev.c:1494
blkdev_get+0xb0/0x940 fs/block_dev.c:1627
blkdev_open+0x202/0x290 fs/block_dev.c:1788
do_dentry_open+0x4aa/0x1160 fs/open.c:796
do_last fs/namei.c:3421 [inline]
path_openat+0x793/0x2df0 fs/namei.c:3537
do_filp_open+0x18c/0x3f0 fs/namei.c:3567
do_sys_open+0x3b3/0x520 fs/open.c:1085
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&bdev->bd_mutex){+.+.}:
blkdev_put+0x30/0x520 fs/block_dev.c:1839
btrfs_close_bdev fs/btrfs/volumes.c:1033 [inline]
btrfs_close_one_device fs/btrfs/volumes.c:1057 [inline]
close_fs_devices.part.0+0x24d/0x8e0 fs/btrfs/volumes.c:1085
close_fs_devices fs/btrfs/volumes.c:1117 [inline]
btrfs_close_devices+0x95/0x1f0 fs/btrfs/volumes.c:1103
open_ctree+0x26b/0x61e0 fs/btrfs/disk-io.c:3326
btrfs_fill_super fs/btrfs/super.c:1209 [inline]
btrfs_mount_root+0x12e5/0x1830 fs/btrfs/super.c:1613
mount_fs+0xa3/0x310 fs/super.c:1261
vfs_kern_mount.part.0+0x68/0x470 fs/namespace.c:961
vfs_kern_mount+0x3c/0x60 fs/namespace.c:951
btrfs_mount+0x23a/0xaa0 fs/btrfs/super.c:1681
mount_fs+0xa3/0x310 fs/super.c:1261
vfs_kern_mount.part.0+0x68/0x470 fs/namespace.c:961
vfs_kern_mount fs/namespace.c:951 [inline]
do_new_mount fs/namespace.c:2492 [inline]
do_mount+0x115c/0x2f50 fs/namespace.c:2822
ksys_mount+0xcf/0x130 fs/namespace.c:3038
__do_sys_mount fs/namespace.c:3052 [inline]
__se_sys_mount fs/namespace.c:3049 [inline]
__x64_sys_mount+0xba/0x150 fs/namespace.c:3049
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&fs_devs->device_list_mutex){+.+.}:
__mutex_lock_common kernel/locking/mutex.c:937 [inline]
__mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
btrfs_run_dev_stats+0xbb/0xa80 fs/btrfs/volumes.c:7111
commit_cowonly_roots+0x1ce/0xc30 fs/btrfs/transaction.c:1172
btrfs_commit_transaction+0x94a/0x2480 fs/btrfs/transaction.c:2218
btrfs_sync_fs+0x120/0x490 fs/btrfs/super.c:1279
__sync_filesystem fs/sync.c:39 [inline]
sync_filesystem+0x1b5/0x250 fs/sync.c:67
generic_shutdown_super+0x70/0x370 fs/super.c:442
kill_anon_super+0x36/0x60 fs/super.c:1032
btrfs_kill_super+0x49/0x550 fs/btrfs/super.c:2221
deactivate_locked_super+0x94/0x160 fs/super.c:329
deactivate_super+0x174/0x1a0 fs/super.c:360
cleanup_mnt+0x1a8/0x290 fs/namespace.c:1098
task_work_run+0x148/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
&fs_devs->device_list_mutex --> &fs_info->reloc_mutex --> &fs_info->tree_log_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&fs_info->tree_log_mutex);
lock(&fs_info->reloc_mutex);
lock(&fs_info->tree_log_mutex);
lock(&fs_devs->device_list_mutex);

*** DEADLOCK ***

3 locks held by syz-executor.2/8141:
#0: 00000000da02ce95 (&type->s_umount_key#91){+.+.}, at: deactivate_super+0x16c/0x1a0 fs/super.c:359
#1: 00000000205c4c14 (&fs_info->reloc_mutex){+.+.}, at: btrfs_commit_transaction+0x80b/0x2480 fs/btrfs/transaction.c:2120
#2: 00000000f3c75895 (&fs_info->tree_log_mutex){+.+.}, at: btrfs_commit_transaction+0x8c2/0x2480 fs/btrfs/transaction.c:2176

stack backtrace:
CPU: 0 PID: 8141 Comm: syz-executor.2 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1222
check_prev_add kernel/locking/lockdep.c:1866 [inline]
check_prevs_add kernel/locking/lockdep.c:1979 [inline]
validate_chain kernel/locking/lockdep.c:2420 [inline]
__lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3416
lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3908
__mutex_lock_common kernel/locking/mutex.c:937 [inline]
__mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
btrfs_run_dev_stats+0xbb/0xa80 fs/btrfs/volumes.c:7111
commit_cowonly_roots+0x1ce/0xc30 fs/btrfs/transaction.c:1172
btrfs_commit_transaction+0x94a/0x2480 fs/btrfs/transaction.c:2218
btrfs_sync_fs+0x120/0x490 fs/btrfs/super.c:1279
__sync_filesystem fs/sync.c:39 [inline]
sync_filesystem+0x1b5/0x250 fs/sync.c:67
generic_shutdown_super+0x70/0x370 fs/super.c:442
kill_anon_super+0x36/0x60 fs/super.c:1032
btrfs_kill_super+0x49/0x550 fs/btrfs/super.c:2221
deactivate_locked_super+0x94/0x160 fs/super.c:329
deactivate_super+0x174/0x1a0 fs/super.c:360
cleanup_mnt+0x1a8/0x290 fs/namespace.c:1098
task_work_run+0x148/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7fc66126b547
Code: ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffc2dec3188 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fc66126b547
RDX: 00007ffc2dec325c RSI: 000000000000000a RDI: 00007ffc2dec3250
RBP: 00007ffc2dec3250 R08: 00000000ffffffff R09: 00007ffc2dec3020
R10: 00005555562f3903 R11: 0000000000000246 R12: 00007fc6612c4b24
R13: 00007ffc2dec4310 R14: 00005555562f3810 R15: 00007ffc2dec4350
x_tables: duplicate underflow at hook 3
x_tables: duplicate underflow at hook 3
print_req_error: 18 callbacks suppressed
print_req_error: I/O error, dev loop2, sector 8064
print_req_error: I/O error, dev loop2, sector 8064
buffer_io_error: 3 callbacks suppressed
Buffer I/O error on dev loop2p1, logical block 1008, async page read
BTRFS info (device loop1): using free space tree
BTRFS info (device loop1): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by syz-executor.3 (20193)
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by systemd-udevd (20224)
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
x_tables: duplicate underflow at hook 3
x_tables: duplicate underflow at hook 3
print_req_error: I/O error, dev loop2, sector 8064
print_req_error: I/O error, dev loop2, sector 8064
Buffer I/O error on dev loop2p1, logical block 1008, async page read
x_tables: duplicate underflow at hook 3
x_tables: duplicate underflow at hook 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
x_tables: duplicate underflow at hook 3
x_tables: duplicate underflow at hook 3


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
Reply all
Reply to author
Forward
0 new messages