possible deadlock in cleanup_net

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 53fff24a Linux 4.19.156
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=179aab0c500000
kernel config: https://syzkaller.appspot.com/x/.config?x=d127f87fb0acc1f3
dashboard link: https://syzkaller.appspot.com/bug?extid=bf91330c3df22a4c2b7c
compiler: gcc (GCC) 10.1.0-syz 20200507

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+bf9133...@syzkaller.appspotmail.com

IPVS: ftp: loaded support on port[0] = 21
F2FS-fs (loop0): Mounted with checkpoint version = 7ad43cd7
======================================================
WARNING: possible circular locking dependency detected
4.19.156-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:7/9636 is trying to acquire lock:
0000000006a9e91b ((wq_completion)"events"){+.+.}, at: flush_workqueue+0xe8/0x13e0 kernel/workqueue.c:2660

but task is already holding lock:
00000000355e2e3a (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (pernet_ops_rwsem){++++}:
unregister_netdevice_notifier+0x7b/0x330 net/core/dev.c:1708
bcm_release+0x94/0x700 net/can/bcm.c:1525
__sock_release+0xcd/0x2a0 net/socket.c:579
sock_close+0x15/0x20 net/socket.c:1140
__fput+0x2ce/0x890 fs/file_table.c:278
task_work_run+0x148/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (&sb->s_type->i_mutex_key#13){+.+.}:
inode_lock include/linux/fs.h:748 [inline]
__sock_release+0x86/0x2a0 net/socket.c:578
sock_close+0x15/0x20 net/socket.c:1140
__fput+0x2ce/0x890 fs/file_table.c:278
delayed_fput+0x56/0x70 fs/file_table.c:304
process_one_work+0x864/0x1570 kernel/workqueue.c:2155
worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #1 ((delayed_fput_work).work){+.+.}:
worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #0 ((wq_completion)"events"){+.+.}:
flush_workqueue+0x117/0x13e0 kernel/workqueue.c:2663
IPVS: ftp: loaded support on port[0] = 21
flush_scheduled_work include/linux/workqueue.h:599 [inline]
tipc_exit_net+0x38/0x60 net/tipc/core.c:100
ops_exit_list+0xa5/0x150 net/core/net_namespace.c:153
cleanup_net+0x3b4/0x8b0 net/core/net_namespace.c:553
process_one_work+0x864/0x1570 kernel/workqueue.c:2155
worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

other info that might help us debug this:

Chain exists of:
(wq_completion)"events" --> &sb->s_type->i_mutex_key#13 --> pernet_ops_rwsem

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(pernet_ops_rwsem);
lock(&sb->s_type->i_mutex_key#13);
lock(pernet_ops_rwsem);
lock((wq_completion)"events");

*** DEADLOCK ***

3 locks held by kworker/u4:7/9636:
#0: 00000000218d7981 ((wq_completion)"%s""netns"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2126
#1: 000000003dcefd9a (net_cleanup_work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2130
#2: 00000000355e2e3a (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

stack backtrace:
CPU: 1 PID: 9636 Comm: kworker/u4:7 Not tainted 4.19.156-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1fc/0x2fe lib/dump_stack.c:118
print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1221
check_prev_add kernel/locking/lockdep.c:1865 [inline]
check_prevs_add kernel/locking/lockdep.c:1978 [inline]
validate_chain kernel/locking/lockdep.c:2419 [inline]
__lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3415
lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3907
flush_workqueue+0x117/0x13e0 kernel/workqueue.c:2663
flush_scheduled_work include/linux/workqueue.h:599 [inline]
tipc_exit_net+0x38/0x60 net/tipc/core.c:100
ops_exit_list+0xa5/0x150 net/core/net_namespace.c:153
cleanup_net+0x3b4/0x8b0 net/core/net_namespace.c:553
process_one_work+0x864/0x1570 kernel/workqueue.c:2155
worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
F2FS-fs (loop0): Found nat_bits in checkpoint
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
F2FS-fs (loop0): Mounted with checkpoint version = 7ad43cd7
F2FS-fs (loop0): Found nat_bits in checkpoint
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
F2FS-fs (loop0): Mounted with checkpoint version = 7ad43cd7
F2FS-fs (loop0): Found nat_bits in checkpoint
IPVS: ftp: loaded support on port[0] = 21
F2FS-fs (loop0): Mounted with checkpoint version = 7ad43cd7
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
usb usb1: usbfs: process 27627 (syz-executor.0) did not claim interface 0 before use
usb usb1: usbfs: process 27644 (syz-executor.0) did not claim interface 0 before use
usb usb1: usbfs: process 27648 (syz-executor.4) did not claim interface 0 before use


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: daefdc9e Linux 4.19.161
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12164345500000
kernel config: https://syzkaller.appspot.com/x/.config?x=5e8be1f59358cc24

dashboard link: https://syzkaller.appspot.com/bug?extid=bf91330c3df22a4c2b7c
compiler: gcc (GCC) 10.1.0-syz 20200507

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14aa0773500000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15ab4117500000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+bf9133...@syzkaller.appspotmail.com

IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21

======================================================
WARNING: possible circular locking dependency detected

4.19.161-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/110 is trying to acquire lock:
00000000f710179f ((wq_completion)"events"){+.+.}, at: flush_workqueue+0xe8/0x13e0 kernel/workqueue.c:2660

but task is already holding lock:

00000000a7c48411 (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (pernet_ops_rwsem){++++}:
unregister_netdevice_notifier+0x7b/0x330 net/core/dev.c:1708

raw_release+0x58/0x820 net/can/raw.c:358

flush_scheduled_work include/linux/workqueue.h:599 [inline]
tipc_exit_net+0x38/0x60 net/tipc/core.c:100
ops_exit_list+0xa5/0x150 net/core/net_namespace.c:153
cleanup_net+0x3b4/0x8b0 net/core/net_namespace.c:553
process_one_work+0x864/0x1570 kernel/workqueue.c:2155
worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
kthread+0x33f/0x460 kernel/kthread.c:259
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

other info that might help us debug this:

Chain exists of:
(wq_completion)"events" --> &sb->s_type->i_mutex_key#13 --> pernet_ops_rwsem

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(pernet_ops_rwsem);
lock(&sb->s_type->i_mutex_key#13);
lock(pernet_ops_rwsem);
lock((wq_completion)"events");

*** DEADLOCK ***

3 locks held by kworker/u4:3/110:
#0: 000000000251fa38 ((wq_completion)"%s""netns"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2126
#1: 00000000b133b634 (net_cleanup_work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2130
#2: 00000000a7c48411 (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

stack backtrace:
CPU: 0 PID: 110 Comm: kworker/u4:3 Not tainted 4.19.161-syzkaller #0

[syzbot]

syzbot suspects this issue was fixed by commit:

commit 3719acc161d5c1ce09912cc1c9eddc2c5faa3c66
Author: Tetsuo Handa <penguin...@i-love.sakura.ne.jp>
Date: Wed Aug 4 10:26:56 2021 +0000

Bluetooth: defer cleanup of resources in hci_unregister_dev()

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=161e8a6d300000
start commit: daefdc9eb24b Linux 4.19.161
git tree: linux-4.19.y

kernel config: https://syzkaller.appspot.com/x/.config?x=5e8be1f59358cc24
dashboard link: https://syzkaller.appspot.com/bug?extid=bf91330c3df22a4c2b7c

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14aa0773500000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15ab4117500000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: Bluetooth: defer cleanup of resources in hci_unregister_dev()

For information about bisection process see: https://goo.gl/tpsmEJ#bisection