possible deadlock in fat_write_inode

5 views
Skip to first unread message

syzbot

unread,
Dec 24, 2022, 7:59:44 PM12/24/22
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: 3f8a27f9e27b Linux 4.19.211
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=10027b44480000
kernel config: https://syzkaller.appspot.com/x/.config?x=9b9277b418617afe
dashboard link: https://syzkaller.appspot.com/bug?extid=34f4e24356c17bbf1ccb
compiler: gcc version 10.2.1 20210110 (Debian 10.2.1-6)

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/98c0bdb4abb3/disk-3f8a27f9.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/ea228ff02669/vmlinux-3f8a27f9.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+34f4e2...@syzkaller.appspotmail.com

IPVS: ftp: loaded support on port[0] = 21
======================================================
IPVS: ftp: loaded support on port[0] = 21
WARNING: possible circular locking dependency detected
4.19.211-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:7/9609 is trying to acquire lock:
00000000d1a8feba (&sbi->s_lock){+.+.}, at: fat_write_inode fs/fat/inode.c:911 [inline]
00000000d1a8feba (&sbi->s_lock){+.+.}, at: fat_write_inode+0xfa/0x180 fs/fat/inode.c:904

but task is already holding lock:
00000000c7ee5ebf ((work_completion)(&(&wb->dwork)->work)){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #6 ((work_completion)(&(&wb->dwork)->work)){+.+.}:
wb_shutdown+0x172/0x210 mm/backing-dev.c:374
bdi_unregister+0x169/0x610 mm/backing-dev.c:946
del_gendisk+0x7f6/0xa80 block/genhd.c:788
loop_remove drivers/block/loop.c:2066 [inline]
loop_control_ioctl drivers/block/loop.c:2165 [inline]
loop_control_ioctl+0x3b1/0x480 drivers/block/loop.c:2131
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:501 [inline]
do_vfs_ioctl+0xcdb/0x12e0 fs/ioctl.c:688
ksys_ioctl+0x9b/0xc0 fs/ioctl.c:705
__do_sys_ioctl fs/ioctl.c:712 [inline]
__se_sys_ioctl fs/ioctl.c:710 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:710
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #5 (loop_ctl_mutex){+.+.}:
lo_open+0x19/0xd0 drivers/block/loop.c:1771
__blkdev_get+0x372/0x1480 fs/block_dev.c:1494
blkdev_get+0xb0/0x940 fs/block_dev.c:1627
blkdev_open+0x202/0x290 fs/block_dev.c:1788
do_dentry_open+0x4aa/0x1160 fs/open.c:796
do_last fs/namei.c:3421 [inline]
path_openat+0x793/0x2df0 fs/namei.c:3537
do_filp_open+0x18c/0x3f0 fs/namei.c:3567
do_sys_open+0x3b3/0x520 fs/open.c:1085
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #4 (&bdev->bd_mutex){+.+.}:
blkdev_put+0x30/0x520 fs/block_dev.c:1839
btrfs_close_bdev fs/btrfs/volumes.c:1033 [inline]
btrfs_close_one_device fs/btrfs/volumes.c:1057 [inline]
close_fs_devices.part.0+0x24d/0x8e0 fs/btrfs/volumes.c:1085
close_fs_devices fs/btrfs/volumes.c:1117 [inline]
btrfs_close_devices+0x95/0x1f0 fs/btrfs/volumes.c:1103
close_ctree+0x3c8/0x850 fs/btrfs/disk-io.c:4047
generic_shutdown_super+0x144/0x370 fs/super.c:456
kill_anon_super+0x36/0x60 fs/super.c:1032
btrfs_kill_super+0x49/0x550 fs/btrfs/super.c:2221
deactivate_locked_super+0x94/0x160 fs/super.c:329
deactivate_super+0x174/0x1a0 fs/super.c:360
cleanup_mnt+0x1a8/0x290 fs/namespace.c:1098
task_work_run+0x148/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #3 (&fs_devs->device_list_mutex){+.+.}:
btrfs_finish_chunk_alloc+0x27b/0xf90 fs/btrfs/volumes.c:4938
btrfs_create_pending_block_groups+0x242/0x590 fs/btrfs/extent-tree.c:10134
__btrfs_end_transaction+0x21a/0xb00 fs/btrfs/transaction.c:855
flush_space+0xa41/0xee0 fs/btrfs/extent-tree.c:4861
btrfs_async_reclaim_metadata_space+0x466/0x1050 fs/btrfs/extent-tree.c:4977
process_one_work+0x864/0x1570 kernel/workqueue.c:2153
worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #2 (sb_internal#2){.+.+}:
sb_start_intwrite include/linux/fs.h:1626 [inline]
start_transaction+0xa37/0xf90 fs/btrfs/transaction.c:528
btrfs_dirty_inode+0xe3/0x210 fs/btrfs/inode.c:6165
btrfs_update_time+0x33b/0x3d0 fs/btrfs/inode.c:6207
update_time fs/inode.c:1675 [inline]
touch_atime+0x23c/0x2a0 fs/inode.c:1746
file_accessed include/linux/fs.h:2123 [inline]
btrfs_file_mmap+0x11b/0x160 fs/btrfs/file.c:2274
call_mmap include/linux/fs.h:1826 [inline]
mmap_region+0xc94/0x16b0 mm/mmap.c:1757
do_mmap+0x8e8/0x1080 mm/mmap.c:1530
do_mmap_pgoff include/linux/mm.h:2329 [inline]
vm_mmap_pgoff+0x197/0x200 mm/util.c:357
ksys_mmap_pgoff+0x298/0x5a0 mm/mmap.c:1580
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&mm->mmap_sem){++++}:
_copy_to_user+0x29/0x100 lib/usercopy.c:25
copy_to_user include/linux/uaccess.h:155 [inline]
filldir64+0x26e/0x430 fs/readdir.c:324
dir_emit_dot include/linux/fs.h:3432 [inline]
dir_emit_dots include/linux/fs.h:3443 [inline]
__fat_readdir+0xef1/0x1490 fs/fat/dir.c:575
iterate_dir+0x473/0x5c0 fs/readdir.c:51
ksys_getdents64+0x175/0x2b0 fs/readdir.c:357
__do_sys_getdents64 fs/readdir.c:376 [inline]
__se_sys_getdents64 fs/readdir.c:373 [inline]
__x64_sys_getdents64+0x6f/0xb0 fs/readdir.c:373
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&sbi->s_lock){+.+.}:
__mutex_lock_common kernel/locking/mutex.c:937 [inline]
__mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
fat_write_inode fs/fat/inode.c:911 [inline]
fat_write_inode+0xfa/0x180 fs/fat/inode.c:904
write_inode fs/fs-writeback.c:1244 [inline]
__writeback_single_inode+0x733/0x11d0 fs/fs-writeback.c:1442
writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
wb_writeback+0x28d/0xcc0 fs/fs-writeback.c:1820
wb_do_writeback fs/fs-writeback.c:1965 [inline]
wb_workfn+0x29b/0x1250 fs/fs-writeback.c:2006
process_one_work+0x864/0x1570 kernel/workqueue.c:2153
worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

other info that might help us debug this:

Chain exists of:
&sbi->s_lock --> loop_ctl_mutex --> (work_completion)(&(&wb->dwork)->work)

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock((work_completion)(&(&wb->dwork)->work));
lock(loop_ctl_mutex);
lock((work_completion)(&(&wb->dwork)->work));
lock(&sbi->s_lock);

*** DEADLOCK ***

2 locks held by kworker/u4:7/9609:
#0: 0000000058a0c2fe ((wq_completion)"writeback"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2124
#1: 00000000c7ee5ebf ((work_completion)(&(&wb->dwork)->work)){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128

stack backtrace:
CPU: 1 PID: 9609 Comm: kworker/u4:7 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: writeback wb_workfn (flush-7:3)
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1222
check_prev_add kernel/locking/lockdep.c:1866 [inline]
check_prevs_add kernel/locking/lockdep.c:1979 [inline]
validate_chain kernel/locking/lockdep.c:2420 [inline]
__lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3416
lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3908
__mutex_lock_common kernel/locking/mutex.c:937 [inline]
__mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
fat_write_inode fs/fat/inode.c:911 [inline]
fat_write_inode+0xfa/0x180 fs/fat/inode.c:904
write_inode fs/fs-writeback.c:1244 [inline]
__writeback_single_inode+0x733/0x11d0 fs/fs-writeback.c:1442
writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
wb_writeback+0x28d/0xcc0 fs/fs-writeback.c:1820
wb_do_writeback fs/fs-writeback.c:1965 [inline]
wb_workfn+0x29b/0x1250 fs/fs-writeback.c:2006
process_one_work+0x864/0x1570 kernel/workqueue.c:2153
worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
BTRFS info (device loop3): enabling inode map caching
BTRFS warning (device loop3): excessive commit interval 622039222
BTRFS warning (device <unknown>): duplicate device /dev/loop2 devid 1 generation 8 scanned by syz-executor.2 (15262)
BTRFS warning (device <unknown>): duplicate device /dev/loop2 devid 1 generation 8 scanned by systemd-udevd (15291)
BTRFS info (device loop3): force zlib compression, level 3
BTRFS info (device loop3): using free space tree
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by systemd-udevd (15289)
BTRFS info (device loop3): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15278)
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop3): enabling inode map caching
BTRFS warning (device loop3): excessive commit interval 622039222
BTRFS info (device loop3): force zlib compression, level 3
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15366)
BTRFS info (device loop3): using free space tree
BTRFS info (device loop3): has skinny extents
IPVS: ftp: loaded support on port[0] = 21
kauditd_printk_skb: 61 callbacks suppressed
audit: type=1800 audit(1671929936.963:134): pid=15470 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=collect_data cause=failed comm="syz-executor.1" name="memory.events" dev="sda1" ino=13943 res=0
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy1 wpan1: encryption failed: -22
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15451)
BTRFS info (device loop2): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by systemd-udevd (15493)
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by syz-executor.3 (15474)
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by systemd-udevd (15528)
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): found reiserfs format "3.6" with non-standard journal
REISERFS (device loop5): using ordered data mode
ieee802154 phy0 wpan0: encryption failed: -22
reiserfs: using flush barriers
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): journal params: device loop5, size 512, journal first block 18, max trans len 256, max batch 225, max commit age 30, max trans age 30
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): checking transaction log (loop5)
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): Using r5 hash to sort names
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): Created .reiserfs_priv - reserved for xattr storage.
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): enabling inode map caching
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS warning (device loop2): excessive commit interval 622039222
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): force zlib compression, level 3
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): using free space tree
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): has skinny extents
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop3): found reiserfs format "3.6" with non-standard journal
REISERFS (device loop3): using ordered data mode
reiserfs: using flush barriers
REISERFS (device loop3): journal params: device loop3, size 512, journal first block 18, max trans len 256, max batch 225, max commit age 30, max trans age 30
REISERFS (device loop3): checking transaction log (loop3)
REISERFS (device loop3): Using r5 hash to sort names
BTRFS info (device loop2): enabling inode map caching
REISERFS (device loop3): Created .reiserfs_priv - reserved for xattr storage.
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by syz-executor.3 (15937)


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
Reply all
Reply to author
Forward
0 new messages