[syzbot] INFO: task hung in blk_freeze_queue (3)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: a41a877bc12d Merge branch 'for-next/fixes' into for-kernelci
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=1233470b080000
kernel config: https://syzkaller.appspot.com/x/.config?x=5cea15779c42821c
dashboard link: https://syzkaller.appspot.com/bug?extid=38e6c55d4969a14c1534
compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10f59dad080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=160918e5080000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+38e6c5...@syzkaller.appspotmail.com

INFO: task syz-executor245:3083 blocked for more than 143 seconds.
Not tainted 6.0.0-rc2-syzkaller-16455-ga41a877bc12d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor245 state:D stack: 0 pid: 3083 ppid: 3071 flags:0x00000001
Call trace:
__switch_to+0x180/0x28c arch/arm64/kernel/process.c:557
context_switch kernel/sched/core.c:5182 [inline]
__schedule+0x414/0x570 kernel/sched/core.c:6494
schedule+0x64/0xa4 kernel/sched/core.c:6570
blk_mq_freeze_queue_wait+0x94/0xf4 block/blk-mq.c:180
blk_freeze_queue+0x94/0xa8 block/blk-mq.c:207
blk_mq_freeze_queue+0x20/0x30 block/blk-mq.c:216
nbd_add_socket+0x78/0x26c drivers/block/nbd.c:1123
__nbd_ioctl+0x80/0x320 drivers/block/nbd.c:1464
nbd_ioctl+0xec/0x13c drivers/block/nbd.c:1521
blkdev_ioctl+0x250/0x764 block/ioctl.c:614
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__arm64_sys_ioctl+0xd0/0x140 fs/ioctl.c:856
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall arch/arm64/kernel/syscall.c:52 [inline]
el0_svc_common+0x138/0x220 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x48/0x154 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x150 arch/arm64/kernel/entry-common.c:624
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:642
el0t_64_sync+0x18c/0x190
INFO: task udevd:3084 blocked for more than 143 seconds.
Not tainted 6.0.0-rc2-syzkaller-16455-ga41a877bc12d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:udevd state:D stack: 0 pid: 3084 ppid: 2560 flags:0x00000000
Call trace:
__switch_to+0x180/0x28c arch/arm64/kernel/process.c:557
context_switch kernel/sched/core.c:5182 [inline]
__schedule+0x414/0x570 kernel/sched/core.c:6494
schedule+0x64/0xa4 kernel/sched/core.c:6570
io_schedule+0x38/0x104 kernel/sched/core.c:8714
folio_wait_bit_common+0x458/0x9f8 mm/filemap.c:1298
folio_put_wait_locked mm/filemap.c:1472 [inline]
do_read_cache_folio+0x78/0x550 mm/filemap.c:3501
read_cache_folio+0x40/0x54 mm/filemap.c:3551
read_mapping_folio include/linux/pagemap.h:762 [inline]
read_part_sector+0x60/0x1d8 block/partitions/core.c:714
adfspart_check_ICS+0x4c/0x38c block/partitions/acorn.c:360
check_partition block/partitions/core.c:146 [inline]
blk_add_partitions block/partitions/core.c:599 [inline]
bdev_disk_changed+0x340/0x63c block/partitions/core.c:685
blkdev_get_whole+0x1b0/0x1f8 block/bdev.c:684
blkdev_get_by_dev+0x178/0x2fc block/bdev.c:821
blkdev_open+0x84/0xd8 block/fops.c:485
do_dentry_open+0x318/0x688 fs/open.c:878
vfs_open+0x38/0x48 fs/open.c:1014
do_open fs/namei.c:3557 [inline]
path_openat+0xe34/0x11c4 fs/namei.c:3691
do_filp_open+0xdc/0x1b8 fs/namei.c:3718
do_sys_openat2+0xb8/0x22c fs/open.c:1311
do_sys_open fs/open.c:1327 [inline]
__do_sys_openat fs/open.c:1343 [inline]
__se_sys_openat fs/open.c:1338 [inline]
__arm64_sys_openat+0xb0/0xe0 fs/open.c:1338
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall arch/arm64/kernel/syscall.c:52 [inline]
el0_svc_common+0x138/0x220 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x48/0x154 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x150 arch/arm64/kernel/entry-common.c:624
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:642
el0t_64_sync+0x18c/0x190

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/10:
#0: ffff80000d4a3568 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x3c/0x450 kernel/rcu/tasks.h:507
1 lock held by rcu_tasks_trace/11:
#0: ffff80000d4a3bb8 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x3c/0x450 kernel/rcu/tasks.h:507
1 lock held by khungtaskd/26:
#0: ffff80000d4a3440 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x4/0x48 include/linux/rcupdate.h:279
2 locks held by getty/2712:
#0: ffff0000c730a898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x28/0x58 drivers/tty/tty_ldisc.c:244
#1: ffff80000f67e2f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x19c/0x88c drivers/tty/n_tty.c:2177
1 lock held by syz-executor245/3083:
#0: ffff0000c5ccd998 (&nbd->config_lock){+.+.}-{3:3}, at: nbd_ioctl+0x70/0x13c drivers/block/nbd.c:1514
1 lock held by udevd/3084:
#0: ffff0000c5daacc8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812

=============================================



---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

[Hillf Danton]

On Wed, 31 Aug 2022 07:35:32 -0700

> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: a41a877bc12d Merge branch 'for-next/fixes' into for-kernelci
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=1233470b080000
> kernel config: https://syzkaller.appspot.com/x/.config?x=5cea15779c42821c
> dashboard link: https://syzkaller.appspot.com/bug?extid=38e6c55d4969a14c1534
> compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> userspace arch: arm64
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10f59dad080000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=160918e5080000

Fix typo.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

--- x/block/genhd.c
+++ g/block/genhd.c
@@ -602,7 +602,7 @@ void del_gendisk(struct gendisk *disk)
* Prevent new I/O from crossing bio_queue_enter().
*/
blk_queue_start_drain(q);
- blk_mq_freeze_queue_wait(q);
+ blk_freeze_queue(q);

if (!(disk->flags & GENHD_FL_HIDDEN)) {
sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
--

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
INFO: task hung in blkdev_put

INFO: task syz-executor.0:3518 blocked for more than 143 seconds.
Not tainted 6.0.0-rc3-syzkaller-00299-gd895ec7938c4-dirty #0

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

task:syz-executor.0 state:D stack: 0 pid: 3518 ppid: 3499 flags:0x0000000d

Call trace:
__switch_to+0x180/0x28c arch/arm64/kernel/process.c:557
context_switch kernel/sched/core.c:5182 [inline]
__schedule+0x414/0x570 kernel/sched/core.c:6494
schedule+0x64/0xa4 kernel/sched/core.c:6570

schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:6629
__mutex_lock_common+0x74c/0xc28 kernel/locking/mutex.c:679
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
blkdev_put+0x78/0x270 block/bdev.c:910
blkdev_close+0x24/0x38 block/fops.c:499
__fput+0x198/0x3bc fs/file_table.c:320
____fput+0x20/0x30 fs/file_table.c:353
task_work_run+0xc4/0x208 kernel/task_work.c:177
get_signal+0xac8/0xb30 kernel/signal.c:2634
do_signal+0x128/0x424 arch/arm64/kernel/signal.c:1071
do_notify_resume+0xc0/0x1d0 arch/arm64/kernel/signal.c:1124
prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:137 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:142 [inline]
el0_svc+0x9c/0x150 arch/arm64/kernel/entry-common.c:625

el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:642
el0t_64_sync+0x18c/0x190

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/10:
#0: ffff80000d4a3568 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x3c/0x450 kernel/rcu/tasks.h:507
1 lock held by rcu_tasks_trace/11:
#0: ffff80000d4a3bb8 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x3c/0x450 kernel/rcu/tasks.h:507
1 lock held by khungtaskd/26:
#0: ffff80000d4a3440 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x4/0x48 include/linux/rcupdate.h:279

2 locks held by getty/2709:
#0: ffff0000c6c9a098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x28/0x58 drivers/tty/tty_ldisc.c:244

#1: ffff80000f67e2f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x19c/0x88c drivers/tty/n_tty.c:2177

1 lock held by udevd/3495:
#0: ffff0000c4b1e4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812
1 lock held by syz-executor.0/3518:
#0: ffff0000c4b1e4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_put+0x78/0x270 block/bdev.c:910
1 lock held by syz-executor.0/3553:
#0: ffff0000c4b1e4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812
1 lock held by syz-executor.0/3579:
#0: ffff0000c4b1e4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812

=============================================



Tested on:

commit: d895ec79 Merge tag 'block-6.0-2022-09-02' of git://git..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=134b2be7080000
kernel config: https://syzkaller.appspot.com/x/.config?x=57b9bfeca947ab90

dashboard link: https://syzkaller.appspot.com/bug?extid=38e6c55d4969a14c1534
compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

patch: https://syzkaller.appspot.com/x/patch.diff?x=12ca9c55080000

[Hillf Danton]

On Wed, 31 Aug 2022 07:35:32 -0700

> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: a41a877bc12d Merge branch 'for-next/fixes' into for-kernelci
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=1233470b080000
> kernel config: https://syzkaller.appspot.com/x/.config?x=5cea15779c42821c
> dashboard link: https://syzkaller.appspot.com/bug?extid=38e6c55d4969a14c1534
> compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> userspace arch: arm64
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10f59dad080000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=160918e5080000

Fix typo.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

--- x/block/genhd.c
+++ g/block/genhd.c
@@ -602,7 +602,7 @@ void del_gendisk(struct gendisk *disk)
* Prevent new I/O from crossing bio_queue_enter().
*/
blk_queue_start_drain(q);
- blk_mq_freeze_queue_wait(q);
+ blk_freeze_queue(q);

if (!(disk->flags & GENHD_FL_HIDDEN)) {
sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");

@@ -641,13 +641,13 @@ void del_gendisk(struct gendisk *disk)
rq_qos_exit(q);
blk_mq_unquiesce_queue(q);

+ __blk_mq_unfreeze_queue(q, true);
/*
* If the disk does not own the queue, allow using passthrough requests
* again. Else leave the queue frozen to fail all I/O.
*/
if (!test_bit(GD_OWNS_QUEUE, &disk->state)) {
blk_queue_flag_clear(QUEUE_FLAG_INIT_DONE, q);
- __blk_mq_unfreeze_queue(q, true);
} else {
if (queue_is_mq(q))
blk_mq_exit_queue(q);
--

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
INFO: task hung in blkdev_put

INFO: task syz-executor.0:3527 blocked for more than 143 seconds.

Not tainted 6.0.0-rc3-syzkaller-00299-gd895ec7938c4-dirty #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

task:syz-executor.0 state:D stack: 0 pid: 3527 ppid: 3506 flags:0x0000000d

#0: ffff0000c6319898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x28/0x58 drivers/tty/tty_ldisc.c:244

#1: ffff80000f67e2f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x19c/0x88c drivers/tty/n_tty.c:2177

1 lock held by udevd/3505:
#0: ffff0000c1f45cc8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812
1 lock held by syz-executor.0/3527:
#0: ffff0000c1f45cc8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_put+0x78/0x270 block/bdev.c:910
1 lock held by syz-executor.0/3563:
#0: ffff0000c1f45cc8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812
1 lock held by syz-executor.0/3585:
#0: ffff0000c1f45cc8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0xfc/0x2fc block/bdev.c:812

=============================================



Tested on:

commit: d895ec79 Merge tag 'block-6.0-2022-09-02' of git://git..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

console output: https://syzkaller.appspot.com/x/log.txt?x=101b868b080000
kernel config: https://syzkaller.appspot.com/x/.config?x=57b9bfeca947ab90

dashboard link: https://syzkaller.appspot.com/bug?extid=38e6c55d4969a14c1534
compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64

patch: https://syzkaller.appspot.com/x/patch.diff?x=1138e4b7080000