[syzbot] possible deadlock in ___neigh_create

19 views
Skip to first unread message

syzbot

unread,
Jan 31, 2022, 3:48:30 PM1/31/22
to dan...@iogearbox.net, da...@davemloft.net, dsa...@kernel.org, edum...@google.com, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, ro...@nvidia.com, syzkall...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: 6449520391df net: stmmac: properly handle with runtime pm ..
git tree: net
console output: https://syzkaller.appspot.com/x/log.txt?x=111187e0700000
kernel config: https://syzkaller.appspot.com/x/.config?x=a6620d0aab7dd315
dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+5239d0...@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
5.17.0-rc1-syzkaller-00210-g6449520391df #0 Not tainted
--------------------------------------------
kworker/0:16/14617 is trying to acquire lock:
ffffffff8d4dd370 (&tbl->lock){++-.}-{2:2}, at: ___neigh_create+0x9e1/0x2990 net/core/neighbour.c:652

but task is already holding lock:
ffffffff8d4dd370 (&tbl->lock){++-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&tbl->lock);
lock(&tbl->lock);

*** DEADLOCK ***

May be due to missing lock nesting notation

5 locks held by kworker/0:16/14617:
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:631 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:658 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: process_one_work+0x890/0x1650 kernel/workqueue.c:2278
#1: ffffc9000293fdb8 ((work_completion)(&(&tbl->managed_work)->work)){+.+.}-{0:0}, at: process_one_work+0x8c4/0x1650 kernel/workqueue.c:2282
#2: ffffffff8d4dd370 (&tbl->lock){++-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572
#3: ffffffff8bb83ae0 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:466 [inline]
#3: ffffffff8bb83ae0 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x84b/0x17f0 net/ipv6/ndisc.c:502
#4: ffffffff8bb83a80 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:95 [inline]
#4: ffffffff8bb83a80 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x2ad/0x14f0 net/ipv6/ip6_output.c:112

stack backtrace:
CPU: 0 PID: 14617 Comm: kworker/0:16 Not tainted 5.17.0-rc1-syzkaller-00210-g6449520391df #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_power_efficient neigh_managed_work
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
print_deadlock_bug kernel/locking/lockdep.c:2956 [inline]
check_deadlock kernel/locking/lockdep.c:2999 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire.cold+0x149/0x3ab kernel/locking/lockdep.c:5027
lock_acquire kernel/locking/lockdep.c:5639 [inline]
lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5604
__raw_write_lock_bh include/linux/rwlock_api_smp.h:202 [inline]
_raw_write_lock_bh+0x2f/0x40 kernel/locking/spinlock.c:334
___neigh_create+0x9e1/0x2990 net/core/neighbour.c:652
ip6_finish_output2+0x1070/0x14f0 net/ipv6/ip6_output.c:123
__ip6_finish_output net/ipv6/ip6_output.c:191 [inline]
__ip6_finish_output+0x61e/0xe90 net/ipv6/ip6_output.c:170
ip6_finish_output+0x32/0x200 net/ipv6/ip6_output.c:201
NF_HOOK_COND include/linux/netfilter.h:296 [inline]
ip6_output+0x1e4/0x530 net/ipv6/ip6_output.c:224
dst_output include/net/dst.h:451 [inline]
NF_HOOK include/linux/netfilter.h:307 [inline]
ndisc_send_skb+0xa99/0x17f0 net/ipv6/ndisc.c:508
ndisc_send_ns+0x3a9/0x840 net/ipv6/ndisc.c:650
ndisc_solicit+0x2cd/0x4f0 net/ipv6/ndisc.c:742
neigh_probe+0xc2/0x110 net/core/neighbour.c:1040
__neigh_event_send+0x37d/0x1570 net/core/neighbour.c:1201
neigh_event_send include/net/neighbour.h:470 [inline]
neigh_managed_work+0x162/0x250 net/core/neighbour.c:1574
process_one_work+0x9ac/0x1650 kernel/workqueue.c:2307
worker_thread+0x657/0x1110 kernel/workqueue.c:2454
kthread+0x2e9/0x3a0 kernel/kthread.c:377
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Eric Dumazet

unread,
Jan 31, 2022, 3:51:32 PM1/31/22
to syzbot, Daniel Borkmann, David Miller, David Ahern, Jakub Kicinski, LKML, netdev, Roopa Prabhu, syzkaller-bugs
Probably caused by
commit 7482e3841d520a368426ac196720601687e2dc47 net, neigh: Add
NTF_MANAGED flag for managed neighbor entries



+static void neigh_managed_work(struct work_struct *work)
+{
+ struct neigh_table *tbl = container_of(work, struct neigh_table,
+ managed_work.work);
+ struct neighbour *neigh;
+
+ write_lock_bh(&tbl->lock);
+ list_for_each_entry(neigh, &tbl->managed_list, managed_list)
+ neigh_event_send(neigh, NULL); // HERE

neigh_event_send() can need to lock tbl->lock, leading to a deadlock

+ queue_delayed_work(system_power_efficient_wq, &tbl->managed_work,
+ NEIGH_VAR(&tbl->parms, DELAY_PROBE_TIME));
+ write_unlock_bh(&tbl->lock);
+}
+

syzbot

unread,
Feb 2, 2022, 1:43:18 AM2/2/22
to dan...@iogearbox.net, da...@davemloft.net, dsa...@kernel.org, edum...@google.com, john.fa...@gmail.com, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, ro...@nvidia.com, syzkall...@googlegroups.com
syzbot has found a reproducer for the following issue on:

HEAD commit: e4d2763f9aaf Merge branch 'lan966x-ptp'
git tree: net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=15b941f0700000
kernel config: https://syzkaller.appspot.com/x/.config?x=e029d3b2ccd4c91a
dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16f9a76c700000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+5239d0...@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
5.17.0-rc1-syzkaller-00547-ge4d2763f9aaf #0 Not tainted
--------------------------------------------
kworker/1:5/3747 is trying to acquire lock:
ffffffff8d4ddef0 (&tbl->lock){+.-.}-{2:2}, at: ___neigh_create+0x9e1/0x2990 net/core/neighbour.c:652

but task is already holding lock:
ffffffff8d4ddef0 (&tbl->lock){+.-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&tbl->lock);
lock(&tbl->lock);

*** DEADLOCK ***

May be due to missing lock nesting notation

5 locks held by kworker/1:5/3747:
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:631 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:658 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: process_one_work+0x890/0x1650 kernel/workqueue.c:2278
#1: ffffc90002d1fdb8 ((work_completion)(&(&tbl->managed_work)->work)){+.+.}-{0:0}, at: process_one_work+0x8c4/0x1650 kernel/workqueue.c:2282
#2: ffffffff8d4ddef0 (&tbl->lock){+.-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572
#3: ffffffff8bb83ae0 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:466 [inline]
#3: ffffffff8bb83ae0 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x84b/0x17f0 net/ipv6/ndisc.c:502
#4: ffffffff8bb83a80 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:95 [inline]
#4: ffffffff8bb83a80 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x2ad/0x14f0 net/ipv6/ip6_output.c:112

stack backtrace:
CPU: 1 PID: 3747 Comm: kworker/1:5 Not tainted 5.17.0-rc1-syzkaller-00547-ge4d2763f9aaf #0

Daniel Borkmann

unread,
Feb 2, 2022, 4:01:10 AM2/2/22
to syzbot, da...@davemloft.net, dsa...@kernel.org, edum...@google.com, john.fa...@gmail.com, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, ro...@nvidia.com, syzkall...@googlegroups.com
On 2/2/22 7:43 AM, syzbot wrote:
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: e4d2763f9aaf Merge branch 'lan966x-ptp'
> git tree: net-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=15b941f0700000
> kernel config: https://syzkaller.appspot.com/x/.config?x=e029d3b2ccd4c91a
> dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16f9a76c700000
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+5239d0...@syzkaller.appspotmail.com

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/bpf.git pr/neigh-probe

syzbot

unread,
Feb 2, 2022, 5:29:08 AM2/2/22
to dan...@iogearbox.net, da...@davemloft.net, dsa...@kernel.org, edum...@google.com, john.fa...@gmail.com, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, ro...@nvidia.com, syzkall...@googlegroups.com
Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+5239d0...@syzkaller.appspotmail.com

Tested on:

commit: 4b48392c net, neigh: Do not trigger immediate probes o..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/bpf.git pr/neigh-probe
kernel config: https://syzkaller.appspot.com/x/.config?x=5044676c290190f2
dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: testing is done by a robot and is best-effort only.

syzbot

unread,
Feb 2, 2022, 6:51:28 AM2/2/22
to dan...@iogearbox.net, da...@davemloft.net, dsa...@kernel.org, edum...@google.com, john.fa...@gmail.com, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, ro...@nvidia.com, syzkall...@googlegroups.com
syzbot has found a reproducer for the following issue on:

HEAD commit: 9f7fb8de5d9b Merge tag 'spi-fix-v5.17-rc2' of git://git.ke..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=12b5fcf4700000
kernel config: https://syzkaller.appspot.com/x/.config?x=b4a89edfcc8f7c74
dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10dd62bfb00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=10581cbfb00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+5239d0...@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
5.17.0-rc2-syzkaller-00039-g9f7fb8de5d9b #0 Not tainted
--------------------------------------------
kworker/0:1/7 is trying to acquire lock:
ffffffff8d4dd130 (&tbl->lock){+.-.}-{2:2}, at: ___neigh_create+0x9e1/0x2990 net/core/neighbour.c:652

but task is already holding lock:
ffffffff8d4dd130 (&tbl->lock){+.-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&tbl->lock);
lock(&tbl->lock);

*** DEADLOCK ***

May be due to missing lock nesting notation

5 locks held by kworker/0:1/7:
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:631 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:658 [inline]
#0: ffff888010c65d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: process_one_work+0x890/0x1650 kernel/workqueue.c:2278
#1: ffffc90000cc7db8 ((work_completion)(&(&tbl->managed_work)->work)){+.+.}-{0:0}, at: process_one_work+0x8c4/0x1650 kernel/workqueue.c:2282
#2: ffffffff8d4dd130 (&tbl->lock){+.-.}-{2:2}, at: neigh_managed_work+0x35/0x250 net/core/neighbour.c:1572
#3: ffffffff8bb83c20 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:466 [inline]
#3: ffffffff8bb83c20 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x84b/0x17f0 net/ipv6/ndisc.c:502
#4: ffffffff8bb83bc0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:95 [inline]
#4: ffffffff8bb83bc0 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x2ad/0x14f0 net/ipv6/ip6_output.c:112

stack backtrace:
CPU: 0 PID: 7 Comm: kworker/0:1 Not tainted 5.17.0-rc2-syzkaller-00039-g9f7fb8de5d9b #0

syzbot

unread,
Feb 2, 2022, 7:11:15 AM2/2/22
to dan...@iogearbox.net, syzkall...@googlegroups.com
Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+5239d0...@syzkaller.appspotmail.com

Tested on:

commit: 4b48392c net, neigh: Do not trigger immediate probes o..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/bpf.git pr/neigh-probe
kernel config: https://syzkaller.appspot.com/x/.config?x=5044676c290190f2
dashboard link: https://syzkaller.appspot.com/bug?extid=5239d0e1778a500d477a
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Reply all
Reply to author
Forward
0 new messages