[syzbot] WARNING in __nf_unregister_net_hook (4)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: cc0626c2 net: smsc911x: skip acpi_device_id table when !CO..
git tree: net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=110a3096d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=7eff0f22b8563a5f
dashboard link: https://syzkaller.appspot.com/bug?extid=154bd5be532a63aa778b

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+154bd5...@syzkaller.appspotmail.com

hook not found, pf 2 num 0
WARNING: CPU: 1 PID: 8144 at net/netfilter/core.c:480 __nf_unregister_net_hook+0x1eb/0x610 net/netfilter/core.c:480
Modules linked in:
CPU: 1 PID: 8144 Comm: syz-executor.0 Not tainted 5.12.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:__nf_unregister_net_hook+0x1eb/0x610 net/netfilter/core.c:480
Code: 0f b6 14 02 48 89 c8 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 11 04 00 00 8b 53 1c 89 ee 48 c7 c7 e0 26 6c 8a e8 72 df 87 01 <0f> 0b e9 e5 00 00 00 e8 09 1d 37 fa 44 8b 3c 24 4c 89 f8 48 c1 e0
RSP: 0018:ffffc9001534f418 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff88802f867a00 RCX: 0000000000000000
RDX: 0000000000040000 RSI: ffffffff815c5205 RDI: fffff52002a69e75
RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff815bdf9e R11: 0000000000000000 R12: ffff8880272c8f20
R13: 0000000000000000 R14: ffff88802fa34c00 R15: 0000000000000006
FS: 00007feaf7d10700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb651f70ca0 CR3: 0000000069f31000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
nf_unregister_net_hook+0xd5/0x110 net/netfilter/core.c:502
nf_tables_unregister_hook.part.0+0x131/0x200 net/netfilter/nf_tables_api.c:234
nf_tables_unregister_hook net/netfilter/nf_tables_api.c:8122 [inline]
nf_tables_commit+0x1d9b/0x4710 net/netfilter/nf_tables_api.c:8122
nfnetlink_rcv_batch+0x975/0x21b0 net/netfilter/nfnetlink.c:508
nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:580 [inline]
nfnetlink_rcv+0x3af/0x420 net/netfilter/nfnetlink.c:598
netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline]
netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1338
netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1927
sock_sendmsg_nosec net/socket.c:654 [inline]
sock_sendmsg+0xcf/0x120 net/socket.c:674
____sys_sendmsg+0x6e8/0x810 net/socket.c:2350
___sys_sendmsg+0xf3/0x170 net/socket.c:2404
__sys_sendmsg+0xe5/0x1b0 net/socket.c:2433
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x466459
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007feaf7d10188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000466459
RDX: 0000000000000000 RSI: 000000002000c2c0 RDI: 0000000000000003
RBP: 00000000004bf9fb R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
R13: 00007ffe0fcaf04f R14: 00007feaf7d10300 R15: 0000000000022000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Dmitry Vyukov]

On Sat, Apr 10, 2021 at 4:49 PM syzbot
<syzbot+154bd5...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: cc0626c2 net: smsc911x: skip acpi_device_id table when !CO..
> git tree: net-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=110a3096d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=7eff0f22b8563a5f
> dashboard link: https://syzkaller.appspot.com/bug?extid=154bd5be532a63aa778b
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+154bd5...@syzkaller.appspotmail.com

Is this also fixed by "netfilter: arptables: use pernet ops struct
during unregister"?
The warning is the same, but the stack is different...

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000008ce91e05bf9f62bc%40google.com.

[Florian Westphal]

Dmitry Vyukov <dvy...@google.com> wrote:
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+154bd5...@syzkaller.appspotmail.com
>
> Is this also fixed by "netfilter: arptables: use pernet ops struct
> during unregister"?
> The warning is the same, but the stack is different...

No, this is a different bug.

In both cases the caller attempts to unregister a hook that the core
can't find, but in this case the caller is nftables, not arptables.

[Pablo Neira Ayuso]

I see no reproducer for this bug. Maybe I broke the dormant flag handling?

Or maybe syzbot got here after the arptables bug has been hitted?

[Dmitry Vyukov]

syzbot always stops after the first bug to give you perfect "Not
tainted" oopses.

[Pablo Neira Ayuso]

Looking at the log file:

https://syzkaller.appspot.com/text?tag=CrashLog&x=110a3096d00000

This is mixing calls to nftables:

14:43:16 executing program 0:
r0 = socket$nl_netfilter(0x10, 0x3, 0xc)
sendmsg$NFT_BATCH(r0, &(0x7f000000c2c0)={0x0, 0x0, &(0x7f0000000000)={&(0x7f00000001c0)={{0x9}, [@NFT_MSG_NEWTABLE={0x28, 0x0, 0xa, 0x3, 0x0, 0x0, {0x2}, [@NFTA_TABLE_NAME={0x9, 0x1, 'syz0\x00'}, @NFTA_TABLE_FLAGS={0x8}]}], {0x14}}, 0x50}}, 0x0)

with arptables:

14:43:16 executing program 1:
r0 = socket$inet_udp(0x2, 0x2, 0x0)
setsockopt$ARPT_SO_SET_REPLACE(r0, 0x0, 0x60, &(0x7f0000000000)={'filter\x00', 0x4, 0x4, 0x3f8, 0x310, 0x200, 0x200, 0x310, 0x310, 0x310, 0x4, 0x0, {[{{@arp={@broadcast, @rand_addr, 0x87010000, 0x0, 0x0, 0x0, {@mac=@link_local}, {@mac}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 'bridge0\x00', 'erspan0\x00'}, 0xc0, 0x100}, @unspec=@RATEEST={0x40, 'RATEEST\x00', 0x0, {'syz1\x00', 0x0, 0x4}}}, {{@arp={@initdev={0xac, 0x1e, 0x0, 0x0}, @local, 0x0, 0x0, 0x0, 0x0, {@mac=@remote}, {}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 'veth0_to_bridge\x00', 'geneve1\x00'}, 0xc0, 0x100}, @unspec=@RATEEST={0x40, 'RATEEST\x00', 0x0, {'syz0\x00', 0x0, 0x2}}}, {{@arp={@local, @multicast1, 0x0, 0x0, 0x0, 0x0, {}, {@mac=@broadcast}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 'veth0_to_batadv\x00', 'veth0_to_hsr\x00'}, 0xc0, 0x110}, @mangle={0x50, 'mangle\x00', 0x0, {@mac=@remote, @mac=@local, @multicast2, @initdev={0xac, 0x1e, 0x0, 0x0}}}}], {{[], 0xc0, 0xe8}, {0x28}}}}, 0x448)

arptables was buggy at the time this bug has been reported.

Am I understanding correctly the syzbot log?

I wonder if the (buggy) arptables removed the incorrect hook from
nftables, then nftables crashed on the same location when removing the
hook. I don't see a clear sequence for this to happen though.

Would it be possible to make syzbot exercise the NFT_MSG_NEWTABLE
codepath (with NFTA_TABLE_FLAGS) to check if the problem still
persists?

Thanks.

[Dmitry Vyukov]

This happened only once so far 40 days ago. So if you consider it
possible that it actually happened due to the arptables issue, I would
mark it as invalid (with "#syz invalid") and move on. If it ever
happens again, syzbot will notify, but then we know it happened with
the aprtables issue fixed.

This bug does not have a reproducer, so it's not possible to test this
exact scenario. It's possible to replay the whole log, but somehow
syzkaller wasn't able to retrigger it by replaying the log. I don't
think it's worth our time at this point.

[Pablo Neira Ayuso]

Thanks.

I found the root cause, I was getting confused by the arptables
report. I'll post a patch.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 02d5e016800d Merge tag 'sound-5.15-rc4' of git://git.kerne..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=160132c0b00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9290a409049988d4
dashboard link: https://syzkaller.appspot.com/bug?extid=154bd5be532a63aa778b
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1400bf0f300000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=144eaf17300000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+154bd5...@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 1 PID: 2648 at net/netfilter/core.c:468 __nf_unregister_net_hook+0x4b1/0x600 net/netfilter/core.c:468
Modules linked in:
CPU: 0 PID: 2648 Comm: kworker/u4:6 Not tainted 5.15.0-rc3-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Workqueue: netns cleanup_net
RIP: 0010:__nf_unregister_net_hook+0x4b1/0x600 net/netfilter/core.c:468
Code: 00 00 00 e8 41 e9 16 fa 41 83 fc 05 74 5e e8 f6 e1 16 fa 44 89 e6 bf 05 00 00 00 e8 29 e9 16 fa e9 f5 fd ff ff e8 df e1 16 fa <0f> 0b 48 c7 c7 80 dd 17 8d e8 c1 a8 d7 01 e9 b1 fe ff ff 48 89 f7
RSP: 0018:ffffc9000b10f658 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff888070c20b98 RCX: 0000000000000000
RDX: ffff888024aa9c80 RSI: ffffffff875f1991 RDI: 0000000000000003
RBP: 0000000000000005 R08: 0000000000000000 R09: ffffc9000b10f597
R10: ffffffff875f159f R11: 000000000000000e R12: 0000000000000001
R13: ffff88801d2b43d8 R14: 0000000000000000 R15: dffffc0000000000
FS: 0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f2f45ae09b0 CR3: 000000000b68e000 CR4: 0000000000350ef0

Call Trace:
nf_unregister_net_hook+0xd5/0x110 net/netfilter/core.c:502

nft_netdev_unregister_hooks net/netfilter/nf_tables_api.c:230 [inline]
nf_tables_unregister_hook.part.0+0x1ab/0x200 net/netfilter/nf_tables_api.c:273
nf_tables_unregister_hook include/net/netfilter/nf_tables.h:1090 [inline]
__nft_release_basechain+0x138/0x640 net/netfilter/nf_tables_api.c:9524
nft_netdev_event net/netfilter/nft_chain_filter.c:351 [inline]
nf_tables_netdev_event+0x521/0x8a0 net/netfilter/nft_chain_filter.c:382
notifier_call_chain+0xb5/0x200 kernel/notifier.c:83
call_netdevice_notifiers_info+0xb5/0x130 net/core/dev.c:1996
call_netdevice_notifiers_extack net/core/dev.c:2008 [inline]
call_netdevice_notifiers net/core/dev.c:2022 [inline]
unregister_netdevice_many+0x951/0x1790 net/core/dev.c:11043
ieee80211_remove_interfaces+0x394/0x820 net/mac80211/iface.c:2140
ieee80211_unregister_hw+0x47/0x1f0 net/mac80211/main.c:1391
mac80211_hwsim_del_radio drivers/net/wireless/mac80211_hwsim.c:3457 [inline]
hwsim_exit_net+0x50e/0xca0 drivers/net/wireless/mac80211_hwsim.c:4217
ops_exit_list+0xb0/0x160 net/core/net_namespace.c:168
cleanup_net+0x4ea/0xb00 net/core/net_namespace.c:591
process_one_work+0x9bf/0x16b0 kernel/workqueue.c:2297
worker_thread+0x658/0x11f0 kernel/workqueue.c:2444
kthread+0x3e5/0x4d0 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+154bd5...@syzkaller.appspotmail.com

Tested on:

commit: 12c925fb netfilter: nftables: skip netdev notifier if ..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/fwestphal/nf.git skip_notifier_01

kernel config: https://syzkaller.appspot.com/x/.config?x=9290a409049988d4
dashboard link: https://syzkaller.appspot.com/bug?extid=154bd5be532a63aa778b
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: testing is done by a robot and is best-effort only.

[Florian Westphal]

syzbot reported following (harmless) WARN:

WARNING: CPU: 1 PID: 2648 at net/netfilter/core.c:468

nft_netdev_unregister_hooks net/netfilter/nf_tables_api.c:230 [inline]

nf_tables_unregister_hook include/net/netfilter/nf_tables.h:1090 [inline]
__nft_release_basechain+0x138/0x640 net/netfilter/nf_tables_api.c:9524
nft_netdev_event net/netfilter/nft_chain_filter.c:351 [inline]
nf_tables_netdev_event+0x521/0x8a0 net/netfilter/nft_chain_filter.c:382

reproducer:
unshare -n bash -c 'ip link add br0 type bridge; nft add table netdev t ; \
nft add chain netdev t ingress \{ type filter hook ingress device "br0" \
priority 0\; policy drop\; \}'

Problem is that when netns device exit hooks create the UNREGISTER
event, the .pre_exit hook for nf_tables core has already removed the
base hook. Notifier attempts to do this again.

The need to do base hook unregister unconditionally was needed in the past,
because notifier was last stage where reg->dev dereference was safe.

Now that nf_tables does the hook removal in .pre_exit, this isn't
needed anymore.

Reported-and-tested-by: syzbot+154bd5...@syzkaller.appspotmail.com
Fixes: 767d1216bff825 ("netfilter: nftables: fix possible UAF over chains from packet path in netns")
Signed-off-by: Florian Westphal <f...@strlen.de>
---
net/netfilter/nft_chain_filter.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/net/netfilter/nft_chain_filter.c b/net/netfilter/nft_chain_filter.c
index 5b02408a920b..3ced0eb6b7c3 100644
--- a/net/netfilter/nft_chain_filter.c
+++ b/net/netfilter/nft_chain_filter.c
@@ -342,12 +342,6 @@ static void nft_netdev_event(unsigned long event, struct net_device *dev,
return;
}

- /* UNREGISTER events are also happening on netns exit.
- *
- * Although nf_tables core releases all tables/chains, only this event
- * handler provides guarantee that hook->ops.dev is still accessible,
- * so we cannot skip exiting net namespaces.
- */
__nft_release_basechain(ctx);
}

@@ -366,6 +360,9 @@ static int nf_tables_netdev_event(struct notifier_block *this,
event != NETDEV_CHANGENAME)
return NOTIFY_DONE;

+ if (!check_net(ctx.net))
+ return NOTIFY_DONE;
+
nft_net = nft_pernet(ctx.net);
mutex_lock(&nft_net->commit_mutex);
list_for_each_entry(table, &nft_net->tables, list) {
--
2.32.0

[Pablo Neira Ayuso]

On Wed, Oct 06, 2021 at 04:20:34PM +0200, Florian Westphal wrote:
> syzbot reported following (harmless) WARN:
>
> WARNING: CPU: 1 PID: 2648 at net/netfilter/core.c:468
> nft_netdev_unregister_hooks net/netfilter/nf_tables_api.c:230 [inline]
> nf_tables_unregister_hook include/net/netfilter/nf_tables.h:1090 [inline]
> __nft_release_basechain+0x138/0x640 net/netfilter/nf_tables_api.c:9524
> nft_netdev_event net/netfilter/nft_chain_filter.c:351 [inline]
> nf_tables_netdev_event+0x521/0x8a0 net/netfilter/nft_chain_filter.c:382
>
> reproducer:
> unshare -n bash -c 'ip link add br0 type bridge; nft add table netdev t ; \
> nft add chain netdev t ingress \{ type filter hook ingress device "br0" \
> priority 0\; policy drop\; \}'
>
> Problem is that when netns device exit hooks create the UNREGISTER
> event, the .pre_exit hook for nf_tables core has already removed the
> base hook. Notifier attempts to do this again.
>
> The need to do base hook unregister unconditionally was needed in the past,
> because notifier was last stage where reg->dev dereference was safe.
>
> Now that nf_tables does the hook removal in .pre_exit, this isn't
> needed anymore.

Applied, thanks.