[v6.1] possible deadlock in nvme_sync_queues

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: e4f7d8be268e Linux 6.1.182
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=17527279580000
kernel config: https://syzkaller.appspot.com/x/.config?x=872c04466179833f
dashboard link: https://syzkaller.appspot.com/bug?extid=78ec14d4fb9d3a2a1282
compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/7cd9b1875376/disk-e4f7d8be.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/4b5d35d18c31/vmlinux-e4f7d8be.xz
kernel image: https://storage.googleapis.com/syzbot-assets/d5ee64c2af77/Image-e4f7d8be.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+78ec14...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Tainted: G M
------------------------------------------------------
kworker/u4:7/4433 is trying to acquire lock:
ffff0000cff10210 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}, at: __flush_work+0xe4/0x1ec kernel/workqueue.c:3069

but task is already holding lock:
ffff0000cfd10528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
ffff0000cfd10528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&ctrl->namespaces_rwsem){++++}-{3:3}:
down_read+0x64/0x300 kernel/locking/rwsem.c:1520
nvme_start_freeze+0x2c/0xc0 drivers/nvme/host/core.c:5416
nvme_dev_disable+0x1b8/0x5ec drivers/nvme/host/pci.c:2747
nvme_timeout+0x628/0xa88 drivers/nvme/host/pci.c:-1
blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
__sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
bt_for_each block/blk-mq-tag.c:327 [inline]
blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
kthread+0x254/0x2e0 kernel/kthread.c:376
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #1 (&dev->shutdown_lock){+.+.}-{3:3}:
__mutex_lock_common+0x1a0/0x1f3c kernel/locking/mutex.c:603
__mutex_lock kernel/locking/mutex.c:747 [inline]
mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799
nvme_dev_disable+0x5c/0x5ec drivers/nvme/host/pci.c:2735
nvme_timeout+0x628/0xa88 drivers/nvme/host/pci.c:-1
blk_mq_rq_timed_out block/blk-mq.c:1566 [inline]
blk_mq_handle_expired+0x148/0x270 block/blk-mq.c:1634
bt_iter+0x2e4/0x3bc block/blk-mq-tag.c:295
__sbitmap_for_each_set include/linux/sbitmap.h:286 [inline]
sbitmap_for_each_set+0x2b8/0x558 include/linux/sbitmap.h:307
bt_for_each block/blk-mq-tag.c:327 [inline]
blk_mq_queue_tag_busy_iter+0x440/0x730 block/blk-mq-tag.c:536
blk_mq_timeout_work+0x16c/0x300 block/blk-mq.c:1676
process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
kthread+0x254/0x2e0 kernel/kthread.c:376
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

-> #0 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5676
__flush_work+0x10c/0x1ec kernel/workqueue.c:3072
__cancel_work_timer+0x2f8/0x450 kernel/workqueue.c:3163
cancel_work_sync+0x24/0x38 kernel/workqueue.c:3199
blk_sync_queue+0x28/0x38 block/blk-core.c:232
nvme_sync_io_queues drivers/nvme/host/core.c:5467 [inline]
nvme_sync_queues+0x70/0xf4 drivers/nvme/host/core.c:5474
nvme_reset_work+0x13c/0x477c drivers/nvme/host/pci.c:2890
process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
kthread+0x254/0x2e0 kernel/kthread.c:376
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850

other info that might help us debug this:

Chain exists of:
(work_completion)(&q->timeout_work) --> &dev->shutdown_lock --> &ctrl->namespaces_rwsem

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&ctrl->namespaces_rwsem);
lock(&dev->shutdown_lock);
lock(&ctrl->namespaces_rwsem);
lock((work_completion)(&q->timeout_work));

*** DEADLOCK ***

3 locks held by kworker/u4:7/4433:
#0: ffff0000cfd92138 ((wq_completion)nvme-reset-wq){+.+.}-{0:0}, at: process_one_work+0x69c/0x13bc kernel/workqueue.c:-1
#1: ffff800021457c40 ((work_completion)(&dev->ctrl.reset_work)){+.+.}-{0:0}, at: process_one_work+0x6e8/0x13bc kernel/workqueue.c:2267
#2: ffff0000cfd10528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
#2: ffff0000cfd10528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

stack backtrace:
CPU: 0 PID: 4433 Comm: kworker/u4:7 Tainted: G M syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/07/2026
Workqueue: nvme-reset-wq nvme_reset_work
Call trace:
dump_backtrace+0x1c4/0x1f0 arch/arm64/kernel/stacktrace.c:158
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
__dump_stack+0x30/0x40 lib/dump_stack.c:88
dump_stack_lvl+0xf4/0x15c lib/dump_stack.c:106
dump_stack+0x1c/0x5c lib/dump_stack.c:113
print_circular_bug+0x148/0x1b0 kernel/locking/lockdep.c:2048
check_noncircular+0x264/0x2f8 kernel/locking/lockdep.c:2170
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2808/0x6634 kernel/locking/lockdep.c:5063
lock_acquire+0x20c/0x638 kernel/locking/lockdep.c:5676
__flush_work+0x10c/0x1ec kernel/workqueue.c:3072
__cancel_work_timer+0x2f8/0x450 kernel/workqueue.c:3163
cancel_work_sync+0x24/0x38 kernel/workqueue.c:3199
blk_sync_queue+0x28/0x38 block/blk-core.c:232
nvme_sync_io_queues drivers/nvme/host/core.c:5467 [inline]
nvme_sync_queues+0x70/0xf4 drivers/nvme/host/core.c:5474
nvme_reset_work+0x13c/0x477c drivers/nvme/host/pci.c:2890
process_one_work+0x7e4/0x13bc kernel/workqueue.c:2292
worker_thread+0x8cc/0xfe8 kernel/workqueue.c:2439
kthread+0x254/0x2e0 kernel/kthread.c:376
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:850
nvme nvme0: 2/0/0 default/read/poll queues


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: e4f7d8be268e Linux 6.1.182
git tree: linux-6.1.y

console output: https://syzkaller.appspot.com/x/log.txt?x=125316c6580000

kernel config: https://syzkaller.appspot.com/x/.config?x=872c04466179833f
dashboard link: https://syzkaller.appspot.com/bug?extid=78ec14d4fb9d3a2a1282
compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
userspace arch: arm64

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16079279580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=10b19a79580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/7cd9b1875376/disk-e4f7d8be.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/4b5d35d18c31/vmlinux-e4f7d8be.xz
kernel image: https://storage.googleapis.com/syzbot-assets/d5ee64c2af77/Image-e4f7d8be.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+78ec14...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected

syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:6/4466 is trying to acquire lock:
ffff0000cf8189c8 ((work_completion)(&q->timeout_work)){+.+.}-{0:0}, at: __flush_work+0xe4/0x1ec kernel/workqueue.c:3069

but task is already holding lock:

ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

3 locks held by kworker/u4:6/4466:
#0: ffff0000cfa0c938 ((wq_completion)nvme-reset-wq){+.+.}-{0:0}, at: process_one_work+0x69c/0x13bc kernel/workqueue.c:-1
#1: ffff800020fe7c40 ((work_completion)(&dev->ctrl.reset_work)){+.+.}-{0:0}, at: process_one_work+0x6e8/0x13bc kernel/workqueue.c:2267
#2: ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_io_queues drivers/nvme/host/core.c:5465 [inline]
#2: ffff0000cfbc0528 (&ctrl->namespaces_rwsem){++++}-{3:3}, at: nvme_sync_queues+0x2c/0xf4 drivers/nvme/host/core.c:5474

stack backtrace:
CPU: 1 PID: 4466 Comm: kworker/u4:6 Not tainted syzkaller #0

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.