[syzbot] [net?] possible deadlock in __unix_gc

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 4d2008430ce8 Merge tag 'docs-6.9-fixes2' of git://git.lwn...
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=14a15280980000
kernel config: https://syzkaller.appspot.com/x/.config?x=98d5a8e00ed1044a
dashboard link: https://syzkaller.appspot.com/bug?extid=fa379358c28cc87cc307
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16a8fb4f180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17ceeb73180000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/5670e5771b96/disk-4d200843.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/03314e6c8879/vmlinux-4d200843.xz
kernel image: https://storage.googleapis.com/syzbot-assets/41aca7a9505a/bzImage-4d200843.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+fa3793...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.9.0-rc5-syzkaller-00007-g4d2008430ce8 #0 Not tainted
------------------------------------------------------
kworker/u8:1/11 is trying to acquire lock:
ffff88807cea4e70 (&u->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807cea4e70 (&u->lock){+.+.}-{2:2}, at: __unix_gc+0x40e/0xf70 net/unix/garbage.c:302

but task is already holding lock:
ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: __unix_gc+0x117/0xf70 net/unix/garbage.c:261

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (unix_gc_lock){+.+.}-{2:2}:
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
unix_notinflight+0x13d/0x390 net/unix/garbage.c:140
unix_detach_fds net/unix/af_unix.c:1819 [inline]
unix_destruct_scm+0x221/0x350 net/unix/af_unix.c:1876
skb_release_head_state+0x100/0x250 net/core/skbuff.c:1188
skb_release_all net/core/skbuff.c:1200 [inline]
__kfree_skb net/core/skbuff.c:1216 [inline]
kfree_skb_reason+0x16d/0x3b0 net/core/skbuff.c:1252
kfree_skb include/linux/skbuff.h:1262 [inline]
manage_oob net/unix/af_unix.c:2672 [inline]
unix_stream_read_generic+0x1125/0x2700 net/unix/af_unix.c:2749
unix_stream_splice_read+0x239/0x320 net/unix/af_unix.c:2981
do_splice_read fs/splice.c:985 [inline]
splice_file_to_pipe+0x299/0x500 fs/splice.c:1295
do_splice+0xf2d/0x1880 fs/splice.c:1379
__do_splice fs/splice.c:1436 [inline]
__do_sys_splice fs/splice.c:1652 [inline]
__se_sys_splice+0x331/0x4a0 fs/splice.c:1634
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&u->lock){+.+.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__unix_gc+0x40e/0xf70 net/unix/garbage.c:302
process_one_work kernel/workqueue.c:3254 [inline]
process_scheduled_works+0xa10/0x17c0 kernel/workqueue.c:3335
worker_thread+0x86d/0xd70 kernel/workqueue.c:3416
kthread+0x2f0/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(unix_gc_lock);
lock(&u->lock);
lock(unix_gc_lock);
lock(&u->lock);

*** DEADLOCK ***

3 locks held by kworker/u8:1/11:
#0: ffff888015089148 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3229 [inline]
#0: ffff888015089148 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_scheduled_works+0x8e0/0x17c0 kernel/workqueue.c:3335
#1: ffffc90000107d00 (unix_gc_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3230 [inline]
#1: ffffc90000107d00 (unix_gc_work){+.+.}-{0:0}, at: process_scheduled_works+0x91b/0x17c0 kernel/workqueue.c:3335
#2: ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#2: ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: __unix_gc+0x117/0xf70 net/unix/garbage.c:261

stack backtrace:
CPU: 0 PID: 11 Comm: kworker/u8:1 Not tainted 6.9.0-rc5-syzkaller-00007-g4d2008430ce8 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events_unbound __unix_gc
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__unix_gc+0x40e/0xf70 net/unix/garbage.c:302
process_one_work kernel/workqueue.c:3254 [inline]
process_scheduled_works+0xa10/0x17c0 kernel/workqueue.c:3335
worker_thread+0x86d/0xd70 kernel/workqueue.c:3416
kthread+0x2f0/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Hillf Danton]

On Tue, 23 Apr 2024 09:09:22 -0700

> syzbot found the following issue on:
>
> HEAD commit: 4d2008430ce8 Merge tag 'docs-6.9-fixes2' of git://git.lwn...
> git tree: upstream

> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17ceeb73180000

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4d2008430ce8

--- x/net/unix/af_unix.c
+++ y/net/unix/af_unix.c
@@ -2646,7 +2646,7 @@ static int unix_stream_recv_urg(struct u
}

static struct sk_buff *manage_oob(struct sk_buff *skb, struct sock *sk,
- int flags, int copied)
+ int flags, int copied, struct sk_buff_head *qh)
{
struct unix_sock *u = unix_sk(sk);

@@ -2669,7 +2669,7 @@ static struct sk_buff *manage_oob(struct
skb_unlink(skb, &sk->sk_receive_queue);
WRITE_ONCE(u->oob_skb, NULL);
if (!WARN_ON_ONCE(skb_unref(skb)))
- kfree_skb(skb);
+ __skb_queue_tail(qh, skb);
skb = skb_peek(&sk->sk_receive_queue);
}
}
@@ -2703,6 +2703,9 @@ static int unix_stream_read_generic(stru
int skip;
size_t size = state->size;
unsigned int last_len;
+ struct sk_buff_head qh;
+
+ __skb_queue_head_init(&qh);

if (unlikely(sk->sk_state != TCP_ESTABLISHED)) {
err = -EINVAL;
@@ -2746,7 +2749,7 @@ redo:
again:
#if IS_ENABLED(CONFIG_AF_UNIX_OOB)
if (skb) {
- skb = manage_oob(skb, sk, flags, copied);
+ skb = manage_oob(skb, sk, flags, copied, &qh);
if (!skb && copied) {
unix_state_unlock(sk);
break;
@@ -2901,6 +2904,7 @@ unlock:
else
scm_destroy(&scm);
out:
+ __skb_queue_purge(&qh);
return copied ? : err;
}

--

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+fa3793...@syzkaller.appspotmail.com

Tested on:

commit: 4d200843 Merge tag 'docs-6.9-fixes2' of git://git.lwn...
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=179a2a80980000

kernel config: https://syzkaller.appspot.com/x/.config?x=98d5a8e00ed1044a
dashboard link: https://syzkaller.appspot.com/bug?extid=fa379358c28cc87cc307
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

patch: https://syzkaller.appspot.com/x/patch.diff?x=128da46b180000

Note: testing is done by a robot and is best-effort only.

[Kuniyuki Iwashima]

From: syzbot <syzbot+fa3793...@syzkaller.appspotmail.com>
Date: Tue, 23 Apr 2024 09:09:22 -0700

This is a normal socket calling recvmsg(),

> do_splice_read fs/splice.c:985 [inline]
> splice_file_to_pipe+0x299/0x500 fs/splice.c:1295
> do_splice+0xf2d/0x1880 fs/splice.c:1379
> __do_splice fs/splice.c:1436 [inline]
> __do_sys_splice fs/splice.c:1652 [inline]
> __se_sys_splice+0x331/0x4a0 fs/splice.c:1634
> do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
> entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> -> #0 (&u->lock){+.+.}-{2:2}:
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
> __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> spin_lock include/linux/spinlock.h:351 [inline]

and this is for TCP_LISTEN introduced by this patch.
https://lore.kernel.org/netdev/20240409201047...@rbox.co/

So, this is false positive.

> __unix_gc+0x40e/0xf70 net/unix/garbage.c:302
> process_one_work kernel/workqueue.c:3254 [inline]
> process_scheduled_works+0xa10/0x17c0 kernel/workqueue.c:3335
> worker_thread+0x86d/0xd70 kernel/workqueue.c:3416
> kthread+0x2f0/0x390 kernel/kthread.c:388
> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>
> other info that might help us debug this:
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(unix_gc_lock);
> lock(&u->lock);
> lock(unix_gc_lock);
> lock(&u->lock);
>
> *** DEADLOCK ***
>
> 3 locks held by kworker/u8:1/11:
> #0: ffff888015089148 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3229 [inline]
> #0: ffff888015089148 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_scheduled_works+0x8e0/0x17c0 kernel/workqueue.c:3335
> #1: ffffc90000107d00 (unix_gc_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3230 [inline]
> #1: ffffc90000107d00 (unix_gc_work){+.+.}-{0:0}, at: process_scheduled_works+0x91b/0x17c0 kernel/workqueue.c:3335
> #2: ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
> #2: ffffffff8f6ab638 (unix_gc_lock){+.+.}-{2:2}, at: __unix_gc+0x117/0xf70 net/unix/garbage.c:261

[...]

>
> If you want syzbot to run the reproducer, reply with:
> #syz test: git://repo/address.git branch-or-commit-hash
> If you attach or paste a git patch, syzbot will apply it before testing.

This suppressed the splat on my setup but just in case

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4d2008430ce87061c9cefd4f83daf2d5bb323a96

diff --git a/include/net/af_unix.h b/include/net/af_unix.h
index 627ea8e2d915..6dcbccfaac04 100644
--- a/include/net/af_unix.h
+++ b/include/net/af_unix.h
@@ -85,6 +85,9 @@ enum unix_socket_lock_class {
U_LOCK_NORMAL,
U_LOCK_SECOND, /* for double locking, see unix_state_double_lock(). */
U_LOCK_DIAG, /* used while dumping icons, see sk_diag_dump_icons(). */
+ U_LOCK_GC_LISTENER, /* used while determining gc candidates for listneing
+ * socket to remove a small race window.
+ */
};

static inline void unix_state_lock_nested(struct sock *sk,
diff --git a/net/unix/garbage.c b/net/unix/garbage.c
index 6433a414acf8..0104be9d4704 100644
--- a/net/unix/garbage.c
+++ b/net/unix/garbage.c
@@ -299,7 +299,7 @@ static void __unix_gc(struct work_struct *work)
__set_bit(UNIX_GC_MAYBE_CYCLE, &u->gc_flags);

if (sk->sk_state == TCP_LISTEN) {
- unix_state_lock(sk);
+ unix_state_lock_nested(sk, U_LOCK_GC_LISTENER);
unix_state_unlock(sk);
}
}

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+fa3793...@syzkaller.appspotmail.com

Tested on:

commit: 4d200843 Merge tag 'docs-6.9-fixes2' of git://git.lwn...
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1756ad73180000

kernel config: https://syzkaller.appspot.com/x/.config?x=98d5a8e00ed1044a
dashboard link: https://syzkaller.appspot.com/bug?extid=fa379358c28cc87cc307
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

patch: https://syzkaller.appspot.com/x/patch.diff?x=11beab4f180000

[syzbot]

syzbot has bisected this issue to:

commit 47d8ac011fe1c9251070e1bd64cb10b48193ec51
Author: Michal Luczaj <mh...@rbox.co>
Date: Tue Apr 9 20:09:39 2024 +0000

af_unix: Fix garbage collector racing against connect()

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=13f440d3180000
start commit: 4d2008430ce8 Merge tag 'docs-6.9-fixes2' of git://git.lwn...
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=100c40d3180000
console output: https://syzkaller.appspot.com/x/log.txt?x=17f440d3180000

kernel config: https://syzkaller.appspot.com/x/.config?x=98d5a8e00ed1044a
dashboard link: https://syzkaller.appspot.com/bug?extid=fa379358c28cc87cc307

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16a8fb4f180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17ceeb73180000

Reported-by: syzbot+fa3793...@syzkaller.appspotmail.com
Fixes: 47d8ac011fe1 ("af_unix: Fix garbage collector racing against connect()")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection