Hello,
syzbot has tested the proposed patch but the reproducer still triggered
crash:
possible deadlock in __filemap_fdatawrite_range
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.4.0-rc5-next-20191028+ #0 Not tainted
-----------------------------------------------------
khugepaged/1083 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
ffff888090de7928 (&sb->s_type->i_lock_key#23){+.+.}, at: spin_lock
include/linux/spinlock.h:338 [inline]
ffff888090de7928 (&sb->s_type->i_lock_key#23){+.+.}, at:
wbc_attach_fdatawrite_inode include/linux/writeback.h:266 [inline]
ffff888090de7928 (&sb->s_type->i_lock_key#23){+.+.}, at:
__filemap_fdatawrite_range+0x26e/0x3b0 mm/filemap.c:420
and this task is already holding:
ffff888090de7ab0 (&(&xa->xa_lock)->rlock#4){..-.}, at: spin_lock_irq
include/linux/spinlock.h:363 [inline]
ffff888090de7ab0 (&(&xa->xa_lock)->rlock#4){..-.}, at:
collapse_file+0x24d/0x4580 mm/khugepaged.c:1524
which would create a new lock dependency:
(&(&xa->xa_lock)->rlock#4){..-.} -> (&sb->s_type->i_lock_key#23){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&xa->xa_lock)->rlock#4){..-.}
... which became SOFTIRQ-irq-safe at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x95/0xcd kernel/locking/spinlock.c:159
test_clear_page_writeback+0x1da/0x11b0 mm/page-writeback.c:2728
end_page_writeback+0x244/0x530 mm/filemap.c:1339
end_buffer_async_write+0x679/0x980 fs/buffer.c:349
end_bio_bh_io_sync+0xed/0x140 fs/buffer.c:3015
bio_endio+0x609/0xaf0 block/bio.c:1818
req_bio_endio block/blk-core.c:271 [inline]
blk_update_request+0x49e/0x10d0 block/blk-core.c:1491
scsi_end_request+0x7f/0x830 drivers/scsi/scsi_lib.c:579
scsi_io_completion+0x20a/0x1420 drivers/scsi/scsi_lib.c:963
scsi_finish_command+0x3b7/0x670 drivers/scsi/scsi.c:228
scsi_softirq_done+0x326/0x3b0 drivers/scsi/scsi_lib.c:1477
blk_done_softirq+0x2fe/0x4d0 block/blk-softirq.c:37
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
do_IRQ+0xe3/0x280 arch/x86/kernel/irq.c:263
ret_from_intr+0x0/0x36
native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:60
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:571
default_idle_call+0x84/0xb0 kernel/sched/idle.c:94
cpuidle_idle_call kernel/sched/idle.c:154 [inline]
do_idle+0x3b7/0x6e0 kernel/sched/idle.c:263
cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:355
start_secondary+0x2f4/0x410 arch/x86/kernel/smpboot.c:264
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242
to a SOFTIRQ-irq-unsafe lock:
(&sb->s_type->i_lock_key#23){+.+.}
... which became SOFTIRQ-irq-unsafe at:
...
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
iget_locked+0x320/0x4b0 fs/inode.c:1193
__ext4_iget+0x265/0x3e20 fs/ext4/inode.c:4835
ext4_fill_super+0x7b11/0xcdb0 fs/ext4/super.c:4489
mount_bdev+0x304/0x3c0 fs/super.c:1415
ext4_mount+0x35/0x40 fs/ext4/super.c:6039
legacy_get_tree+0x108/0x220 fs/fs_context.c:647
vfs_get_tree+0x8e/0x300 fs/super.c:1545
do_new_mount fs/namespace.c:2822 [inline]
do_mount+0x135a/0x1b50 fs/namespace.c:3142
ksys_mount+0xdb/0x150 fs/namespace.c:3351
do_mount_root+0x35/0x1d3 init/do_mounts.c:393
mount_block_root+0x353/0x61d init/do_mounts.c:422
mount_root+0x283/0x2cd init/do_mounts.c:612
prepare_namespace+0x26f/0x2ae init/do_mounts.c:671
kernel_init_freeable+0x5a0/0x5b9 init/main.c:1210
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&sb->s_type->i_lock_key#23);
local_irq_disable();
lock(&(&xa->xa_lock)->rlock#4);
lock(&sb->s_type->i_lock_key#23);
<Interrupt>
lock(&(&xa->xa_lock)->rlock#4);
*** DEADLOCK ***
1 lock held by khugepaged/1083:
#0: ffff888090de7ab0 (&(&xa->xa_lock)->rlock#4){..-.}, at: spin_lock_irq
include/linux/spinlock.h:363 [inline]
#0: ffff888090de7ab0 (&(&xa->xa_lock)->rlock#4){..-.}, at:
collapse_file+0x24d/0x4580 mm/khugepaged.c:1524
the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&(&xa->xa_lock)->rlock#4){..-.} {
IN-SOFTIRQ-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock_irqsave
include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x95/0xcd
kernel/locking/spinlock.c:159
test_clear_page_writeback+0x1da/0x11b0
mm/page-writeback.c:2728
end_page_writeback+0x244/0x530 mm/filemap.c:1339
end_buffer_async_write+0x679/0x980 fs/buffer.c:349
end_bio_bh_io_sync+0xed/0x140 fs/buffer.c:3015
bio_endio+0x609/0xaf0 block/bio.c:1818
req_bio_endio block/blk-core.c:271 [inline]
blk_update_request+0x49e/0x10d0 block/blk-core.c:1491
scsi_end_request+0x7f/0x830 drivers/scsi/scsi_lib.c:579
scsi_io_completion+0x20a/0x1420
drivers/scsi/scsi_lib.c:963
scsi_finish_command+0x3b7/0x670 drivers/scsi/scsi.c:228
scsi_softirq_done+0x326/0x3b0
drivers/scsi/scsi_lib.c:1477
blk_done_softirq+0x2fe/0x4d0 block/blk-softirq.c:37
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
do_IRQ+0xe3/0x280 arch/x86/kernel/irq.c:263
ret_from_intr+0x0/0x36
native_safe_halt+0xe/0x10
arch/x86/include/asm/irqflags.h:60
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:571
default_idle_call+0x84/0xb0 kernel/sched/idle.c:94
cpuidle_idle_call kernel/sched/idle.c:154 [inline]
do_idle+0x3b7/0x6e0 kernel/sched/idle.c:263
cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:355
start_secondary+0x2f4/0x410
arch/x86/kernel/smpboot.c:264
secondary_startup_64+0xa4/0xb0
arch/x86/kernel/head_64.S:242
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:128
[inline]
_raw_spin_lock_irq+0x60/0x80
kernel/locking/spinlock.c:167
spin_lock_irq include/linux/spinlock.h:363 [inline]
__add_to_page_cache_locked+0x677/0xec0 mm/filemap.c:877
add_to_page_cache_lru+0x1d8/0x790 mm/filemap.c:943
do_read_cache_page+0x9fc/0x2140 mm/filemap.c:2770
read_cache_page+0x5e/0x70 mm/filemap.c:2874
read_mapping_page include/linux/pagemap.h:396 [inline]
read_dev_sector+0x71/0x310 block/partition-generic.c:667
read_part_sector block/partitions/check.h:38 [inline]
adfspart_check_ICS+0x12d/0xc90
block/partitions/acorn.c:361
check_partition+0x3bc/0x6ce block/partitions/check.c:167
rescan_partitions+0x230/0xa30
block/partition-generic.c:531
__blkdev_get+0xbae/0x1600 fs/block_dev.c:1599
blkdev_get+0x47/0x2c0 fs/block_dev.c:1707
register_disk block/genhd.c:655 [inline]
__device_add_disk+0xabf/0x1230 block/genhd.c:745
device_add_disk+0x2b/0x40 block/genhd.c:763
add_disk include/linux/genhd.h:429 [inline]
brd_init+0x237/0x41c drivers/block/brd.c:514
do_one_initcall+0x120/0x81a init/main.c:938
do_initcall_level init/main.c:1006 [inline]
do_initcalls init/main.c:1014 [inline]
do_basic_setup init/main.c:1031 [inline]
kernel_init_freeable+0x4ca/0x5b9 init/main.c:1191
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
}
... key at: [<ffffffff8ab268c0>] __key.18618+0x0/0x40
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
wbc_attach_fdatawrite_inode include/linux/writeback.h:266 [inline]
__filemap_fdatawrite_range+0x26e/0x3b0 mm/filemap.c:420
__filemap_fdatawrite mm/filemap.c:429 [inline]
filemap_flush+0x24/0x30 mm/filemap.c:456
collapse_file+0x3b28/0x4580 mm/khugepaged.c:1609
khugepaged_scan_file mm/khugepaged.c:1890 [inline]
khugepaged_scan_mm_slot mm/khugepaged.c:1988 [inline]
khugepaged_do_scan mm/khugepaged.c:2072 [inline]
khugepaged+0x2da9/0x4360 mm/khugepaged.c:2117
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
the dependencies between the lock to be acquired
and SOFTIRQ-irq-unsafe lock:
-> (&sb->s_type->i_lock_key#23){+.+.} {
HARDIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142
[inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
iget_locked+0x320/0x4b0 fs/inode.c:1193
__ext4_iget+0x265/0x3e20 fs/ext4/inode.c:4835
ext4_fill_super+0x7b11/0xcdb0 fs/ext4/super.c:4489
mount_bdev+0x304/0x3c0 fs/super.c:1415
ext4_mount+0x35/0x40 fs/ext4/super.c:6039
legacy_get_tree+0x108/0x220 fs/fs_context.c:647
vfs_get_tree+0x8e/0x300 fs/super.c:1545
do_new_mount fs/namespace.c:2822 [inline]
do_mount+0x135a/0x1b50 fs/namespace.c:3142
ksys_mount+0xdb/0x150 fs/namespace.c:3351
do_mount_root+0x35/0x1d3 init/do_mounts.c:393
mount_block_root+0x353/0x61d init/do_mounts.c:422
mount_root+0x283/0x2cd init/do_mounts.c:612
prepare_namespace+0x26f/0x2ae init/do_mounts.c:671
kernel_init_freeable+0x5a0/0x5b9 init/main.c:1210
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
SOFTIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142
[inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
iget_locked+0x320/0x4b0 fs/inode.c:1193
__ext4_iget+0x265/0x3e20 fs/ext4/inode.c:4835
ext4_fill_super+0x7b11/0xcdb0 fs/ext4/super.c:4489
mount_bdev+0x304/0x3c0 fs/super.c:1415
ext4_mount+0x35/0x40 fs/ext4/super.c:6039
legacy_get_tree+0x108/0x220 fs/fs_context.c:647
vfs_get_tree+0x8e/0x300 fs/super.c:1545
do_new_mount fs/namespace.c:2822 [inline]
do_mount+0x135a/0x1b50 fs/namespace.c:3142
ksys_mount+0xdb/0x150 fs/namespace.c:3351
do_mount_root+0x35/0x1d3 init/do_mounts.c:393
mount_block_root+0x353/0x61d init/do_mounts.c:422
mount_root+0x283/0x2cd init/do_mounts.c:612
prepare_namespace+0x26f/0x2ae init/do_mounts.c:671
kernel_init_freeable+0x5a0/0x5b9 init/main.c:1210
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142
[inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
iget_locked+0x320/0x4b0 fs/inode.c:1193
__ext4_iget+0x265/0x3e20 fs/ext4/inode.c:4835
ext4_fill_super+0x7b11/0xcdb0 fs/ext4/super.c:4489
mount_bdev+0x304/0x3c0 fs/super.c:1415
ext4_mount+0x35/0x40 fs/ext4/super.c:6039
legacy_get_tree+0x108/0x220 fs/fs_context.c:647
vfs_get_tree+0x8e/0x300 fs/super.c:1545
do_new_mount fs/namespace.c:2822 [inline]
do_mount+0x135a/0x1b50 fs/namespace.c:3142
ksys_mount+0xdb/0x150 fs/namespace.c:3351
do_mount_root+0x35/0x1d3 init/do_mounts.c:393
mount_block_root+0x353/0x61d init/do_mounts.c:422
mount_root+0x283/0x2cd init/do_mounts.c:612
prepare_namespace+0x26f/0x2ae init/do_mounts.c:671
kernel_init_freeable+0x5a0/0x5b9 init/main.c:1210
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
}
... key at: [<ffffffff890ded28>] ext4_fs_type+0xa8/0x100
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
wbc_attach_fdatawrite_inode include/linux/writeback.h:266 [inline]
__filemap_fdatawrite_range+0x26e/0x3b0 mm/filemap.c:420
__filemap_fdatawrite mm/filemap.c:429 [inline]
filemap_flush+0x24/0x30 mm/filemap.c:456
collapse_file+0x3b28/0x4580 mm/khugepaged.c:1609
khugepaged_scan_file mm/khugepaged.c:1890 [inline]
khugepaged_scan_mm_slot mm/khugepaged.c:1988 [inline]
khugepaged_do_scan mm/khugepaged.c:2072 [inline]
khugepaged+0x2da9/0x4360 mm/khugepaged.c:2117
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
stack backtrace:
CPU: 0 PID: 1083 Comm: khugepaged Not tainted 5.4.0-rc5-next-20191028+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
print_bad_irq_dependency kernel/locking/lockdep.c:2095 [inline]
check_irq_usage.cold+0x586/0x6fe kernel/locking/lockdep.c:2293
check_prev_add kernel/locking/lockdep.c:2480 [inline]
check_prevs_add kernel/locking/lockdep.c:2581 [inline]
validate_chain kernel/locking/lockdep.c:2971 [inline]
__lock_acquire+0x25b4/0x4a00 kernel/locking/lockdep.c:3955
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
wbc_attach_fdatawrite_inode include/linux/writeback.h:266 [inline]
__filemap_fdatawrite_range+0x26e/0x3b0 mm/filemap.c:420
__filemap_fdatawrite mm/filemap.c:429 [inline]
filemap_flush+0x24/0x30 mm/filemap.c:456
collapse_file+0x3b28/0x4580 mm/khugepaged.c:1609
khugepaged_scan_file mm/khugepaged.c:1890 [inline]
khugepaged_scan_mm_slot mm/khugepaged.c:1988 [inline]
khugepaged_do_scan mm/khugepaged.c:2072 [inline]
khugepaged+0x2da9/0x4360 mm/khugepaged.c:2117
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
BUG: sleeping function called from invalid context at
include/linux/percpu-rwsem.h:38
in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 1083, name:
khugepaged
INFO: lockdep is turned off.
irq event stamp: 58
hardirqs last enabled at (57): [<ffffffff81a6f157>] rmqueue
mm/page_alloc.c:3304 [inline]
hardirqs last enabled at (57): [<ffffffff81a6f157>]
get_page_from_freelist+0x3437/0x4330 mm/page_alloc.c:3692
hardirqs last disabled at (58): [<ffffffff8755523a>] __raw_spin_lock_irq
include/linux/spinlock_api_smp.h:126 [inline]
hardirqs last disabled at (58): [<ffffffff8755523a>]
_raw_spin_lock_irq+0x3a/0x80 kernel/locking/spinlock.c:167
softirqs last enabled at (0): [<ffffffff8143e8a2>]
copy_process+0x1822/0x6880 kernel/fork.c:2019
softirqs last disabled at (0): [<0000000000000000>] 0x0
Preemption disabled at:
[<ffffffff81b2c5ad>] spin_lock_irq include/linux/spinlock.h:363 [inline]
[<ffffffff81b2c5ad>] collapse_file+0x24d/0x4580 mm/khugepaged.c:1524
CPU: 0 PID: 1083 Comm: khugepaged Not tainted 5.4.0-rc5-next-20191028+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
___might_sleep.cold+0x1fb/0x23e kernel/sched/core.c:6788
__might_sleep+0x95/0x190 kernel/sched/core.c:6741
percpu_down_read include/linux/percpu-rwsem.h:38 [inline]
ext4_writepages+0x1cb/0x2e70 fs/ext4/inode.c:2728
do_writepages+0xfa/0x2a0 mm/page-writeback.c:2344
__filemap_fdatawrite_range+0x2bc/0x3b0 mm/filemap.c:421
__filemap_fdatawrite mm/filemap.c:429 [inline]
filemap_flush+0x24/0x30 mm/filemap.c:456
collapse_file+0x3b28/0x4580 mm/khugepaged.c:1609
khugepaged_scan_file mm/khugepaged.c:1890 [inline]
khugepaged_scan_mm_slot mm/khugepaged.c:1988 [inline]
khugepaged_do_scan mm/khugepaged.c:2072 [inline]
khugepaged+0x2da9/0x4360 mm/khugepaged.c:2117
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Tested on:
commit: 8de721f7 thp: proper flush
git tree:
https://github.com/liu-song-6/linux.git thp-fix-20191028
console output:
https://syzkaller.appspot.com/x/log.txt?x=12fa8b74e00000
kernel config:
https://syzkaller.appspot.com/x/.config?x=cb86688f30db053d
dashboard link:
https://syzkaller.appspot.com/bug?extid=efb9e48b9fbdc49bb34a