[syzbot] [netfilter?] possible deadlock in nf_tables_dumpreset_rules

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 54e82e93ca93 Merge tag 'core_urgent_for_v6.19_rc4' of git:..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=10b1ee22580000
kernel config: https://syzkaller.appspot.com/x/.config?x=8bfa57a8c0ab3aa8
dashboard link: https://syzkaller.appspot.com/bug?extid=ee287f5effa60050d9ac
compiler: gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: i386

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/d900f083ada3/non_bootable_disk-54e82e93.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/c7af41d4f0f4/vmlinux-54e82e93.xz
kernel image: https://storage.googleapis.com/syzbot-assets/02aa2250dd4f/bzImage-54e82e93.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+ee287f...@syzkaller.appspotmail.com

netlink: 48 bytes leftover after parsing attributes in process `syz.8.6539'.
======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Tainted: G L
------------------------------------------------------
syz.8.6539/2008 is trying to acquire lock:
ffff888052e32cd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_rules+0x6f/0xa0 net/netfilter/nf_tables_api.c:3913

but task is already holding lock:
ffff888025cb16f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}:
__mutex_lock_common kernel/locking/mutex.c:614 [inline]
__mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
__netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404
netlink_dump_start include/linux/netlink.h:341 [inline]
ip_set_dump+0x17f/0x210 net/netfilter/ipset/ip_set_core.c:1717
nfnetlink_rcv_msg+0x9fc/0x1200 net/netfilter/nfnetlink.c:302
netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
sock_sendmsg_nosec net/socket.c:727 [inline]
__sock_sendmsg net/socket.c:742 [inline]
____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
__sys_sendmsg+0x16d/0x220 net/socket.c:2678
do_syscall_32_irqs_on arch/x86/entry/syscall_32.c:83 [inline]
__do_fast_syscall_32+0xe8/0x680 arch/x86/entry/syscall_32.c:307
do_fast_syscall_32+0x32/0x80 arch/x86/entry/syscall_32.c:332
entry_SYSENTER_compat_after_hwframe+0x84/0x8e

-> #1 (nfnl_subsys_ipset){+.+.}-{4:4}:
__mutex_lock_common kernel/locking/mutex.c:614 [inline]
__mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
ip_set_nfnl_get_byindex+0x7c/0x290 net/netfilter/ipset/ip_set_core.c:909
set_target_v1_checkentry+0x1ac/0x570 net/netfilter/xt_set.c:313
xt_check_target+0x27c/0xa40 net/netfilter/x_tables.c:1038
nft_target_init+0x459/0x7d0 net/netfilter/nft_compat.c:267
nf_tables_newexpr net/netfilter/nf_tables_api.c:3550 [inline]
nf_tables_newrule+0xedd/0x2910 net/netfilter/nf_tables_api.c:4419
nfnetlink_rcv_batch+0x190d/0x2350 net/netfilter/nfnetlink.c:526
nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:649 [inline]
nfnetlink_rcv+0x3c1/0x430 net/netfilter/nfnetlink.c:667
netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
sock_sendmsg_nosec net/socket.c:727 [inline]
__sock_sendmsg net/socket.c:742 [inline]
____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
__sys_sendmsg+0x16d/0x220 net/socket.c:2678
do_syscall_32_irqs_on arch/x86/entry/syscall_32.c:83 [inline]
__do_fast_syscall_32+0xe8/0x680 arch/x86/entry/syscall_32.c:307
do_fast_syscall_32+0x32/0x80 arch/x86/entry/syscall_32.c:332
entry_SYSENTER_compat_after_hwframe+0x84/0x8e

-> #0 (&nft_net->commit_mutex){+.+.}-{4:4}:
check_prev_add kernel/locking/lockdep.c:3165 [inline]
check_prevs_add kernel/locking/lockdep.c:3284 [inline]
validate_chain kernel/locking/lockdep.c:3908 [inline]
__lock_acquire+0x1669/0x2890 kernel/locking/lockdep.c:5237
lock_acquire kernel/locking/lockdep.c:5868 [inline]
lock_acquire+0x179/0x330 kernel/locking/lockdep.c:5825
__mutex_lock_common kernel/locking/mutex.c:614 [inline]
__mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
nf_tables_dumpreset_rules+0x6f/0xa0 net/netfilter/nf_tables_api.c:3913
netlink_dump+0x539/0xd30 net/netlink/af_netlink.c:2325
__netlink_dump_start+0x6d6/0x990 net/netlink/af_netlink.c:2440
netlink_dump_start include/linux/netlink.h:341 [inline]
nft_netlink_dump_start_rcu+0x81/0x1f0 net/netfilter/nf_tables_api.c:1309
nf_tables_getrule_reset+0x56b/0x6b0 net/netfilter/nf_tables_api.c:4053
nfnetlink_rcv_msg+0x583/0x1200 net/netfilter/nfnetlink.c:290
netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
sock_sendmsg_nosec net/socket.c:727 [inline]
__sock_sendmsg net/socket.c:742 [inline]
____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
__sys_sendmsg+0x16d/0x220 net/socket.c:2678
do_syscall_32_irqs_on arch/x86/entry/syscall_32.c:83 [inline]
__do_fast_syscall_32+0xe8/0x680 arch/x86/entry/syscall_32.c:307
do_fast_syscall_32+0x32/0x80 arch/x86/entry/syscall_32.c:332
entry_SYSENTER_compat_after_hwframe+0x84/0x8e

other info that might help us debug this:

Chain exists of:
&nft_net->commit_mutex --> nfnl_subsys_ipset --> nlk_cb_mutex-NETFILTER

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(nlk_cb_mutex-NETFILTER);
lock(nfnl_subsys_ipset);
lock(nlk_cb_mutex-NETFILTER);
lock(&nft_net->commit_mutex);

*** DEADLOCK ***

1 lock held by syz.8.6539/2008:
#0: ffff888025cb16f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

stack backtrace:
CPU: 2 UID: 0 PID: 2008 Comm: syz.8.6539 Tainted: G L syzkaller #0 PREEMPT(full)
Tainted: [L]=SOFTLOCKUP
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94 [inline]
dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:120
print_circular_bug+0x275/0x340 kernel/locking/lockdep.c:2043
check_noncircular+0x146/0x160 kernel/locking/lockdep.c:2175
check_prev_add kernel/locking/lockdep.c:3165 [inline]
check_prevs_add kernel/locking/lockdep.c:3284 [inline]
validate_chain kernel/locking/lockdep.c:3908 [inline]
__lock_acquire+0x1669/0x2890 kernel/locking/lockdep.c:5237
lock_acquire kernel/locking/lockdep.c:5868 [inline]
lock_acquire+0x179/0x330 kernel/locking/lockdep.c:5825
__mutex_lock_common kernel/locking/mutex.c:614 [inline]
__mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
nf_tables_dumpreset_rules+0x6f/0xa0 net/netfilter/nf_tables_api.c:3913
netlink_dump+0x539/0xd30 net/netlink/af_netlink.c:2325
__netlink_dump_start+0x6d6/0x990 net/netlink/af_netlink.c:2440
netlink_dump_start include/linux/netlink.h:341 [inline]
nft_netlink_dump_start_rcu+0x81/0x1f0 net/netfilter/nf_tables_api.c:1309
nf_tables_getrule_reset+0x56b/0x6b0 net/netfilter/nf_tables_api.c:4053
nfnetlink_rcv_msg+0x583/0x1200 net/netfilter/nfnetlink.c:290
netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
sock_sendmsg_nosec net/socket.c:727 [inline]
__sock_sendmsg net/socket.c:742 [inline]
____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
__sys_sendmsg+0x16d/0x220 net/socket.c:2678
do_syscall_32_irqs_on arch/x86/entry/syscall_32.c:83 [inline]
__do_fast_syscall_32+0xe8/0x680 arch/x86/entry/syscall_32.c:307
do_fast_syscall_32+0x32/0x80 arch/x86/entry/syscall_32.c:332
entry_SYSENTER_compat_after_hwframe+0x84/0x8e
RIP: 0023:0xf702d579
Code: b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 00 00 00 00 00 00 00 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d b4 26 00 00 00 00 8d b4 26 00 00 00 00
RSP: 002b:00000000f4f9f55c EFLAGS: 00000296 ORIG_RAX: 0000000000000172
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000080000240
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000296 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
</TASK>
----------------
Code disassembly (best guess), 2 bytes skipped:
0: 10 06 adc %al,(%rsi)
2: 03 74 b4 01 add 0x1(%rsp,%rsi,4),%esi
6: 10 07 adc %al,(%rdi)
8: 03 74 b0 01 add 0x1(%rax,%rsi,4),%esi
c: 10 08 adc %cl,(%rax)
e: 03 74 d8 01 add 0x1(%rax,%rbx,8),%esi
1e: 00 51 52 add %dl,0x52(%rcx)
21: 55 push %rbp
22: 89 e5 mov %esp,%ebp
24: 0f 34 sysenter
26: cd 80 int $0x80
* 28: 5d pop %rbp <-- trapping instruction
29: 5a pop %rdx
2a: 59 pop %rcx
2b: c3 ret
2c: 90 nop
2d: 90 nop
2e: 90 nop
2f: 90 nop
30: 8d b4 26 00 00 00 00 lea 0x0(%rsi,%riz,1),%esi
37: 8d b4 26 00 00 00 00 lea 0x0(%rsi,%riz,1),%esi


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Florian Westphal]

#syz dup: possible deadlock in nf_tables_dumpreset_obj