[v6.1] possible deadlock in worker_thread

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: fb28aa725e05 Linux 6.1.180
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=15dd7e32580000
kernel config: https://syzkaller.appspot.com/x/.config?x=31dfefc4a14efea3
dashboard link: https://syzkaller.appspot.com/bug?extid=b4162d1817cac6b21ece
compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/9a6fe2a54d0f/disk-fb28aa72.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/742b23a1d931/vmlinux-fb28aa72.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b3f257c16d2e/bzImage-fb28aa72.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+b4162d...@syzkaller.appspotmail.com

nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
nvmet: ctrl 1 fatal error occurred!
======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:15/4411 is trying to acquire lock:
ffff8880764dc030 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: __flush_work+0x109/0xac0 kernel/workqueue.c:3072

but task is already holding lock:
ffffc900048cfd00 ((work_completion)(&queue->release_work)){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&queue->release_work)){+.+.}-{0:0}:
process_one_work+0x7ec/0x1160 kernel/workqueue.c:2268
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #1 ((wq_completion)nvmet-wq){+.+.}-{0:0}:
__flush_workqueue+0x165/0x1380 kernel/workqueue.c:2812
nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1891
nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:185
nvmet_execute_admin_connect+0x332/0x7e0 drivers/nvme/target/fabrics-cmd.c:266
nvmet_tcp_execute_request drivers/nvme/target/tcp.c:584 [inline]
nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1230 [inline]
nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1294 [inline]
nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1320 [inline]
nvmet_tcp_io_work+0x1a52/0x87b0 drivers/nvme/target/tcp.c:1370
process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #0 ((work_completion)(&queue->io_work)){+.+.}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2d18/0x7c40 kernel/locking/lockdep.c:5063
lock_acquire+0x1bb/0x4a0 kernel/locking/lockdep.c:5676
__flush_work+0x122/0xac0 kernel/workqueue.c:3072
__cancel_work_timer+0x3f4/0x560 kernel/workqueue.c:3163
nvmet_tcp_release_queue_work+0x2ef/0xf30 drivers/nvme/target/tcp.c:1534
process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

other info that might help us debug this:

Chain exists of:
(work_completion)(&queue->io_work) --> (wq_completion)nvmet-wq --> (work_completion)(&queue->release_work)

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock((work_completion)(&queue->release_work));
lock((wq_completion)nvmet-wq);
lock((work_completion)(&queue->release_work));
lock((work_completion)(&queue->io_work));

*** DEADLOCK ***

2 locks held by kworker/0:15/4411:
#0: ffff888025555538 ((wq_completion)nvmet-wq){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267
#1: ffffc900048cfd00 ((work_completion)(&queue->release_work)){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267

stack backtrace:
CPU: 0 PID: 4411 Comm: kworker/0:15 Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/24/2026
Workqueue: nvmet-wq nvmet_tcp_release_queue_work
Call Trace:
<TASK>
dump_stack_lvl+0x188/0x24e lib/dump_stack.c:106
check_noncircular+0x296/0x330 kernel/locking/lockdep.c:2170
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3830 [inline]
__lock_acquire+0x2d18/0x7c40 kernel/locking/lockdep.c:5063
lock_acquire+0x1bb/0x4a0 kernel/locking/lockdep.c:5676
__flush_work+0x122/0xac0 kernel/workqueue.c:3072
__cancel_work_timer+0x3f4/0x560 kernel/workqueue.c:3163
nvmet_tcp_release_queue_work+0x2ef/0xf30 drivers/nvme/target/tcp.c:1534
process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
kthread+0x29d/0x330 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 9c7634c60784 Linux 5.15.214
git tree: linux-5.15.y
console output: https://syzkaller.appspot.com/x/log.txt?x=127af29e580000
kernel config: https://syzkaller.appspot.com/x/.config?x=f161cbc9aef65db0
dashboard link: https://syzkaller.appspot.com/bug?extid=caf10f0d2e95c5109993

compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:

disk image: https://storage.googleapis.com/syzbot-assets/a77ef10eb83f/disk-9c7634c6.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/9e27d6d7ff73/vmlinux-9c7634c6.xz
kernel image: https://storage.googleapis.com/syzbot-assets/2f95795bf4ae/bzImage-9c7634c6.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by: syzbot+caf10f...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------

kworker/0:1H/264 is trying to acquire lock:
ffff88814753c938 ((wq_completion)nvmet-wq){+.+.}-{0:0}, at: flush_workqueue+0x14e/0x1390 kernel/workqueue.c:2830

but task is already holding lock:

ffffc9000253fd00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x79e/0xff0 kernel/workqueue.c:2285

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&queue->io_work)){+.+.}-{0:0}:
__flush_work+0x116/0x210 kernel/workqueue.c:3090
nvmet_tcp_release_queue_work+0x32f/0xdb0 drivers/nvme/target/tcp.c:1511
process_one_work+0x867/0xff0 kernel/workqueue.c:2310
worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
kthread+0x42e/0x520 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #1 ((work_completion)(&queue->release_work)){+.+.}-{0:0}:
process_one_work+0x7ba/0xff0 kernel/workqueue.c:2286
worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
kthread+0x42e/0x520 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #0 ((wq_completion)nvmet-wq){+.+.}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x2c66/0x7b50 kernel/locking/lockdep.c:5012
lock_acquire+0x19e/0x400 kernel/locking/lockdep.c:5623
flush_workqueue+0x16a/0x1390 kernel/workqueue.c:2830
nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1865
nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:154
nvmet_execute_admin_connect+0x332/0x750 drivers/nvme/target/fabrics-cmd.c:217
nvmet_tcp_execute_request drivers/nvme/target/tcp.c:588 [inline]
nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1211 [inline]
nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1273 [inline]
nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1299 [inline]
nvmet_tcp_io_work+0x19c4/0x8510 drivers/nvme/target/tcp.c:1349
process_one_work+0x867/0xff0 kernel/workqueue.c:2310
worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
kthread+0x42e/0x520 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:

(wq_completion)nvmet-wq --> (work_completion)(&queue->release_work) --> (work_completion)(&queue->io_work)

Possible unsafe locking scenario:

CPU0 CPU1
---- ----

lock((work_completion)(&queue->io_work));
lock((work_completion)(&queue->release_work));
lock((work_completion)(&queue->io_work));
lock((wq_completion)nvmet-wq);

*** DEADLOCK ***

2 locks held by kworker/0:1H/264:
#0: ffff88814753d538 ((wq_completion)nvmet_tcp_wq){+.+.}-{0:0}, at: process_one_work+0x75c/0xff0 kernel/workqueue.c:-1
#1: ffffc9000253fd00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x79e/0xff0 kernel/workqueue.c:2285

stack backtrace:
CPU: 0 PID: 264 Comm: kworker/0:1H Not tainted syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/24/2026

Workqueue: nvmet_tcp_wq nvmet_tcp_io_work
Call Trace:
<TASK>
dump_stack_lvl+0x188/0x250 lib/dump_stack.c:106
check_noncircular+0x296/0x330 kernel/locking/lockdep.c:2133
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x2c66/0x7b50 kernel/locking/lockdep.c:5012
lock_acquire+0x19e/0x400 kernel/locking/lockdep.c:5623
flush_workqueue+0x16a/0x1390 kernel/workqueue.c:2830
nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1865
nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:154
nvmet_execute_admin_connect+0x332/0x750 drivers/nvme/target/fabrics-cmd.c:217
nvmet_tcp_execute_request drivers/nvme/target/tcp.c:588 [inline]
nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1211 [inline]
nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1273 [inline]
nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1299 [inline]
nvmet_tcp_io_work+0x19c4/0x8510 drivers/nvme/target/tcp.c:1349
process_one_work+0x867/0xff0 kernel/workqueue.c:2310
worker_thread+0xad7/0x12a0 kernel/workqueue.c:2457
kthread+0x42e/0x520 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
</TASK>
nvmet: creating controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:047137bc-7dc0-410b-ba75-2d0d9998a3b2.