[syzbot] possible deadlock in p9_req_put

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: ca688bff68bc Add linux-next specific files for 20220808
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=104c8666080000
kernel config: https://syzkaller.appspot.com/x/.config?x=4c20e006003cdecb
dashboard link: https://syzkaller.appspot.com/bug?extid=50f7e8d06c3768dd97f3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+50f7e8...@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
5.19.0-next-20220808-syzkaller #0 Not tainted
--------------------------------------------
kworker/0:5/3683 is trying to acquire lock:
ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: p9_tag_remove net/9p/client.c:367 [inline]
ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: p9_req_put net/9p/client.c:375 [inline]
ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: p9_req_put+0xc6/0x250 net/9p/client.c:372

but task is already holding lock:
ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline]
ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: p9_conn_cancel+0xaa/0x970 net/9p/trans_fd.c:192

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&clnt->lock);
lock(&clnt->lock);

*** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by kworker/0:5/3683:
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:636 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:663 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x87a/0x1610 kernel/workqueue.c:2260
#1: ffffc90005167da8 ((work_completion)(&m->rq)){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264
#2: ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline]
#2: ffff888026be2c18 (&clnt->lock){+.+.}-{2:2}, at: p9_conn_cancel+0xaa/0x970 net/9p/trans_fd.c:192

stack backtrace:
CPU: 0 PID: 3683 Comm: kworker/0:5 Not tainted 5.19.0-next-20220808-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
Workqueue: events p9_read_work
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
print_deadlock_bug kernel/locking/lockdep.c:2988 [inline]
check_deadlock kernel/locking/lockdep.c:3031 [inline]
validate_chain kernel/locking/lockdep.c:3816 [inline]
__lock_acquire.cold+0x116/0x3a7 kernel/locking/lockdep.c:5053
lock_acquire kernel/locking/lockdep.c:5666 [inline]
lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162
p9_tag_remove net/9p/client.c:367 [inline]
p9_req_put net/9p/client.c:375 [inline]
p9_req_put+0xc6/0x250 net/9p/client.c:372
p9_conn_cancel+0x640/0x970 net/9p/trans_fd.c:213
p9_read_work+0x514/0x10c0 net/9p/trans_fd.c:403
process_one_work+0x991/0x1610 kernel/workqueue.c:2289
worker_thread+0x665/0x1080 kernel/workqueue.c:2436
kthread+0x2e4/0x3a0 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: ca688bff68bc Add linux-next specific files for 20220808
git tree: linux-next

console+strace: https://syzkaller.appspot.com/x/log.txt?x=1408148e080000

kernel config: https://syzkaller.appspot.com/x/.config?x=4c20e006003cdecb
dashboard link: https://syzkaller.appspot.com/bug?extid=50f7e8d06c3768dd97f3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11f6ea66080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1640de8e080000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+50f7e8...@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
5.19.0-next-20220808-syzkaller #0 Not tainted
--------------------------------------------

kworker/1:2/1590 is trying to acquire lock:
ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: p9_tag_remove net/9p/client.c:367 [inline]
ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: p9_req_put net/9p/client.c:375 [inline]
ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: p9_req_put+0xc6/0x250 net/9p/client.c:372

but task is already holding lock:

ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline]
ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: p9_conn_cancel+0xaa/0x970 net/9p/trans_fd.c:192

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&clnt->lock);
lock(&clnt->lock);

*** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by kworker/1:2/1590:

#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:636 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:663 [inline]
#0: ffff888011864d38 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x87a/0x1610 kernel/workqueue.c:2260

#1: ffffc90006357da8 (p9_poll_work){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264
#2: ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline]
#2: ffff888079686c18 (&clnt->lock){+.+.}-{2:2}, at: p9_conn_cancel+0xaa/0x970 net/9p/trans_fd.c:192

stack backtrace:
CPU: 1 PID: 1590 Comm: kworker/1:2 Not tainted 5.19.0-next-20220808-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022

Workqueue: events p9_poll_workfn

Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
print_deadlock_bug kernel/locking/lockdep.c:2988 [inline]
check_deadlock kernel/locking/lockdep.c:3031 [inline]
validate_chain kernel/locking/lockdep.c:3816 [inline]
__lock_acquire.cold+0x116/0x3a7 kernel/locking/lockdep.c:5053
lock_acquire kernel/locking/lockdep.c:5666 [inline]
lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162
p9_tag_remove net/9p/client.c:367 [inline]
p9_req_put net/9p/client.c:375 [inline]
p9_req_put+0xc6/0x250 net/9p/client.c:372
p9_conn_cancel+0x640/0x970 net/9p/trans_fd.c:213

p9_poll_mux net/9p/trans_fd.c:627 [inline]
p9_poll_workfn+0x25d/0x4e0 net/9p/trans_fd.c:1147

[syzbot]

syzbot has bisected this issue to:

commit 54283409cd162fc60480df514924ed4cb313735e
Author: Andrei Otcheretianski <andrei.otc...@intel.com>
Date: Tue Jun 14 14:20:04 2022 +0000

wifi: mac80211: Consider MLO links in offchannel logic

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1046e22a080000
start commit: ca688bff68bc Add linux-next specific files for 20220808
git tree: linux-next
final oops: https://syzkaller.appspot.com/x/report.txt?x=1246e22a080000
console output: https://syzkaller.appspot.com/x/log.txt?x=1446e22a080000

kernel config: https://syzkaller.appspot.com/x/.config?x=4c20e006003cdecb
dashboard link: https://syzkaller.appspot.com/bug?extid=50f7e8d06c3768dd97f3

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11f6ea66080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1640de8e080000

Reported-by: syzbot+50f7e8...@syzkaller.appspotmail.com
Fixes: 54283409cd16 ("wifi: mac80211: Consider MLO links in offchannel logic")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Hillf Danton]

On Mon, 08 Aug 2022 07:14:23 -0700

>
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: ca688bff68bc Add linux-next specific files for 20220808
> git tree: linux-next
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=1408148e080000
> kernel config: https://syzkaller.appspot.com/x/.config?x=4c20e006003cdecb
> dashboard link: https://syzkaller.appspot.com/bug?extid=50f7e8d06c3768dd97f3
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11f6ea66080000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1640de8e080000

Put request outside of client->lock.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git ca688bff68bc

--- a/net/9p/trans_fd.c
+++ t/net/9p/trans_fd.c
@@ -204,6 +204,7 @@ static void p9_conn_cancel(struct p9_con
list_for_each_entry_safe(req, rtmp, &m->unsent_req_list, req_list) {
list_move(&req->req_list, &cancel_list);
}
+ spin_unlock(&m->client->lock);

list_for_each_entry_safe(req, rtmp, &cancel_list, req_list) {
p9_debug(P9_DEBUG_ERROR, "call back req %p\n", req);
@@ -212,7 +213,6 @@ static void p9_conn_cancel(struct p9_con
req->t_err = err;
p9_client_cb(m->client, req, REQ_STATUS_ERROR);
}
- spin_unlock(&m->client->lock);
}

static __poll_t
--

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+50f7e8...@syzkaller.appspotmail.com

Tested on:

commit: ca688bff Add linux-next specific files for 20220808
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
console output: https://syzkaller.appspot.com/x/log.txt?x=147d0be1080000

kernel config: https://syzkaller.appspot.com/x/.config?x=4c20e006003cdecb
dashboard link: https://syzkaller.appspot.com/bug?extid=50f7e8d06c3768dd97f3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

patch: https://syzkaller.appspot.com/x/patch.diff?x=142dbea9080000

Note: testing is done by a robot and is best-effort only.

[Dominique Martinet]

syzbot reported a double-lock here and we no longer need this
lock after requests have been moved off to local list:
just drop the lock earlier.

Reported-by: syzbot+50f7e8...@syzkaller.appspotmail.com
Signed-off-by: Dominique Martinet <asma...@codewreck.org>
Tested-by: Schspa Shi <sch...@gmail.com>
---
net/9p/trans_fd.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/9p/trans_fd.c b/net/9p/trans_fd.c
index e758978b44be..60fcc6b30b46 100644
--- a/net/9p/trans_fd.c
+++ b/net/9p/trans_fd.c
@@ -205,6 +205,8 @@ static void p9_conn_cancel(struct p9_conn *m, int err)

list_move(&req->req_list, &cancel_list);
}

+ spin_unlock(&m->client->lock);

+

list_for_each_entry_safe(req, rtmp, &cancel_list, req_list) {
p9_debug(P9_DEBUG_ERROR, "call back req %p\n", req);

list_del(&req->req_list);
@@ -212,7 +214,6 @@ static void p9_conn_cancel(struct p9_conn *m, int err)

req->t_err = err;
p9_client_cb(m->client, req, REQ_STATUS_ERROR);
}
- spin_unlock(&m->client->lock);
}

static __poll_t
--

2.35.1