possible deadlock in flush_workqueue

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: dafd6344 Linux 4.19.79
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=17dfcacf600000
kernel config: https://syzkaller.appspot.com/x/.config?x=af1dc11a1b49548a
dashboard link: https://syzkaller.appspot.com/bug?extid=91a7242c038b5c070eb9
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+91a724...@syzkaller.appspotmail.com

block nbd5: Receive control failed (result -22)
block nbd5: shutting down sockets
kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env
============================================
WARNING: possible recursive locking detected
4.19.79 #0 Not tainted
--------------------------------------------
kworker/u5:1/9149 is trying to acquire lock:
kobject: 'loop3' (000000009af191bc): fill_kobj_path: path
= '/devices/virtual/block/loop3'
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
flush_workqueue+0xf7/0x14b0 kernel/workqueue.c:2652

but task is already holding lock:
kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path
= '/devices/virtual/block/loop4'
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
__write_once_size include/linux/compiler.h:220 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
atomic_long_set include/asm-generic/atomic-long.h:59 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
set_work_data kernel/workqueue.c:617 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline]
00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
process_one_work+0x87e/0x1750 kernel/workqueue.c:2124

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock((wq_completion)"knbd%d-recv"nbd->index);
lock((wq_completion)"knbd%d-recv"nbd->index);

*** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by kworker/u5:1/9149:
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
__write_once_size include/linux/compiler.h:220 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
atomic_long_set include/asm-generic/atomic-long.h:59 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
set_work_data kernel/workqueue.c:617 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline]
#0: 00000000dc1d0876 ((wq_completion)"knbd%d-recv"nbd->index){+.+.}, at:
process_one_work+0x87e/0x1750 kernel/workqueue.c:2124
EXT4-fs (loop4): VFS: Can't find ext4 filesystem
#1: 000000007f0b712b ((work_completion)(&args->work)){+.+.}, at:
process_one_work+0x8b4/0x1750 kernel/workqueue.c:2128
#2: 0000000025c76702 (&nbd->config_lock){+.+.}, at:
refcount_dec_and_mutex_lock lib/refcount.c:311 [inline]
#2: 0000000025c76702 (&nbd->config_lock){+.+.}, at:
refcount_dec_and_mutex_lock+0x56/0x90 lib/refcount.c:306

stack backtrace:
CPU: 1 PID: 9149 Comm: kworker/u5:1 Not tainted 4.19.79 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: knbd5-recv recv_work
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
print_deadlock_bug kernel/locking/lockdep.c:1759 [inline]
check_deadlock kernel/locking/lockdep.c:1803 [inline]
validate_chain kernel/locking/lockdep.c:2399 [inline]
__lock_acquire.cold+0x20f/0x4a7 kernel/locking/lockdep.c:3411
lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3903
flush_workqueue+0x126/0x14b0 kernel/workqueue.c:2655
drain_workqueue+0x1b4/0x470 kernel/workqueue.c:2820
destroy_workqueue+0x21/0x6b0 kernel/workqueue.c:4158
nbd_config_put+0x3cf/0x860 drivers/block/nbd.c:1132
recv_work+0x19b/0x200 drivers/block/nbd.c:740
process_one_work+0x989/0x1750 kernel/workqueue.c:2153
worker_thread+0x98/0xe40 kernel/workqueue.c:2296
kthread+0x354/0x420 kernel/kthread.c:246
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env
kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'loop0' (00000000bdb0fb12): kobject_uevent_env
kobject: 'loop0' (00000000bdb0fb12): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'loop2' (000000000c77fc03): kobject_uevent_env
kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env
kobject: 'loop2' (000000000c77fc03): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'loop4' (00000000ca9adf4f): kobject_uevent_env
kobject: 'loop4' (00000000ca9adf4f): fill_kobj_path: path
= '/devices/virtual/block/loop4'


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.