Hello,
syzbot has tested the proposed patch but the reproducer still triggered
crash:
possible deadlock in rtnl_lock
IPVS: stopping backup sync thread 4762 ...
IPVS: stopping backup sync thread 4767 ...
IPVS: stopping backup sync thread 4771 ...
======================================================
WARNING: possible circular locking dependency detected
4.16.0-rc6+ #76 Not tainted
------------------------------------------------------
syz-executor/4753 is trying to acquire lock:
(rtnl_mutex){+.+.}, at: [<00000000e9304478>] rtnl_lock+0x17/0x20
net/core/rtnetlink.c:74
but task is already holding lock:
(ipvs->sync_mutex){+.+.}, at: [<00000000b1eae02f>]
start_sync_thread+0x141/0x2b20 net/netfilter/ipvs/ip_vs_sync.c:1774
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (ipvs->sync_mutex){+.+.}:
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
start_sync_thread+0x141/0x2b20 net/netfilter/ipvs/ip_vs_sync.c:1774
do_ip_vs_set_ctl+0x111c/0x1c90 net/netfilter/ipvs/ip_vs_ctl.c:2387
nf_sockopt net/netfilter/nf_sockopt.c:106 [inline]
nf_setsockopt+0x67/0xc0 net/netfilter/nf_sockopt.c:115
ip_setsockopt+0x97/0xa0 net/ipv4/ip_sockglue.c:1253
udp_setsockopt+0x45/0x80 net/ipv4/udp.c:2400
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:3039
SYSC_setsockopt net/socket.c:1850 [inline]
SyS_setsockopt+0x189/0x360 net/socket.c:1829
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
-> #0 (rtnl_mutex){+.+.}:
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74
ip_mc_drop_socket+0x88/0x230 net/ipv4/igmp.c:2643
inet_release+0x4e/0x1c0 net/ipv4/af_inet.c:413
sock_release+0x8d/0x1e0 net/socket.c:594
start_sync_thread+0x1f6c/0x2b20 net/netfilter/ipvs/ip_vs_sync.c:1911
do_ip_vs_set_ctl+0x111c/0x1c90 net/netfilter/ipvs/ip_vs_ctl.c:2387
nf_sockopt net/netfilter/nf_sockopt.c:106 [inline]
nf_setsockopt+0x67/0xc0 net/netfilter/nf_sockopt.c:115
ip_setsockopt+0x97/0xa0 net/ipv4/ip_sockglue.c:1253
udp_setsockopt+0x45/0x80 net/ipv4/udp.c:2400
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:3039
SYSC_setsockopt net/socket.c:1850 [inline]
SyS_setsockopt+0x189/0x360 net/socket.c:1829
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(ipvs->sync_mutex);
lock(rtnl_mutex);
lock(ipvs->sync_mutex);
lock(rtnl_mutex);
*** DEADLOCK ***
1 lock held by syz-executor/4753:
#0: (ipvs->sync_mutex){+.+.}, at: [<00000000b1eae02f>]
start_sync_thread+0x141/0x2b20 net/netfilter/ipvs/ip_vs_sync.c:1774
stack backtrace:
CPU: 1 PID: 4753 Comm: syz-executor Not tainted 4.16.0-rc6+ #76
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x194/0x24d lib/dump_stack.c:53
print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
check_prev_add kernel/locking/lockdep.c:1863 [inline]
check_prevs_add kernel/locking/lockdep.c:1976 [inline]
validate_chain kernel/locking/lockdep.c:2417 [inline]
__lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74
ip_mc_drop_socket+0x88/0x230 net/ipv4/igmp.c:2643
inet_release+0x4e/0x1c0 net/ipv4/af_inet.c:413
sock_release+0x8d/0x1e0 net/socket.c:594
start_sync_thread+0x1f6c/0x2b20 net/netfilter/ipvs/ip_vs_sync.c:1911
do_ip_vs_set_ctl+0x111c/0x1c90 net/netfilter/ipvs/ip_vs_ctl.c:2387
nf_sockopt net/netfilter/nf_sockopt.c:106 [inline]
nf_setsockopt+0x67/0xc0 net/netfilter/nf_sockopt.c:115
ip_setsockopt+0x97/0xa0 net/ipv4/ip_sockglue.c:1253
udp_setsockopt+0x45/0x80 net/ipv4/udp.c:2400
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:3039
SYSC_setsockopt net/socket.c:1850 [inline]
SyS_setsockopt+0x189/0x360 net/socket.c:1829
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x454879
RSP: 002b:00007f738259ac68 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
RAX: ffffffffffffffda RBX: 00007f738259b6d4 RCX: 0000000000454879
RDX: 000000000000048b RSI: 0000000000000000 RDI: 0000000000000004
RBP: 000000000072bea0 R08: 0000000000000018 R09: 0000000000000000
R10: 00000000200000c0 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000000000000051a R14: 00000000006f9b10 R15: 0000000000000000
IPVS: stopping backup sync thread 4780 ...
IPVS: stopping backup sync thread 4785 ...
IPVS: stopping backup sync thread 4789 ...
IPVS: stopping backup sync thread 4793 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4797 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4800 ...
IPVS: stopping backup sync thread 4799 ...
IPVS: stopping backup sync thread 4806 ...
IPVS: stopping backup sync thread 4814 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4815 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4824 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4830 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4832 ...
IPVS: stopping backup sync thread 4833 ...
IPVS: stopping backup sync thread 4837 ...
IPVS: stopping backup sync thread 4840 ...
IPVS: stopping backup sync thread 4852 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4857 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4863 ...
IPVS: stopping backup sync thread 4862 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4871 ...
IPVS: stopping backup sync thread 4867 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4866 ...
IPVS: stopping backup sync thread 4872 ...
IPVS: stopping backup sync thread 4876 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4886 ...
IPVS: stopping backup sync thread 4893 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4896 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4900 ...
IPVS: stopping backup sync thread 4902 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4906 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4912 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4913 ...
IPVS: stopping backup sync thread 4917 ...
IPVS: stopping backup sync thread 4927 ...
IPVS: stopping backup sync thread 4934 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4939 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4940 ...
IPVS: stopping backup sync thread 4942 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4943 ...
IPVS: stopping backup sync thread 4949 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4956 ...
IPVS: stopping backup sync thread 4962 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4965 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4971 ...
IPVS: stopping backup sync thread 4973 ...
IPVS: stopping backup sync thread 4952 ...
IPVS: stopping backup sync thread 4980 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4983 ...
IPVS: stopping backup sync thread 4990 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4992 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 4997 ...
IPVS: stopping backup sync thread 4996 ...
IPVS: stopping backup sync thread 5003 ...
IPVS: stopping backup sync thread 4998 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5011 ...
IPVS: stopping backup sync thread 5013 ...
IPVS: stopping backup sync thread 5018 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5024 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5028 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5030 ...
IPVS: stopping backup sync thread 5033 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5036 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5039 ...
IPVS: stopping backup sync thread 5052 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5058 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5062 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5065 ...
IPVS: sync thread started: state = BACKUP, mcast_ifn = sit0, syncid = 0, id
= 0
IPVS: stopping backup sync thread 5061 ...
IPVS: stopping backup sync thread 5073 ...
Tested on net-next commit
5d22d47b9ed96eddb35821dc2cc4f629f45827f7 (Tue Mar 27 17:33:21 2018 +0000)
Merge branch 'sfc-filter-locking'
compiler: gcc (GCC) 7.1.1 20170620
Patch:
https://syzkaller.appspot.com/x/patch.diff?id=5868717594378240
Kernel config:
https://syzkaller.appspot.com/x/.config?id=4372867303600475372
Raw console output:
https://syzkaller.appspot.com/x/log.txt?id=5907429879447552