[syzbot] [netfilter?] WARNING in nf_conntrack_cleanup_net_list

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 5ce74bc1b7cb Add linux-next specific files for 20251211
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=11a241b4580000
kernel config: https://syzkaller.appspot.com/x/.config?x=a94030c847137a18
dashboard link: https://syzkaller.appspot.com/bug?extid=4393c47753b7808dac7d
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/e1660176a50f/disk-5ce74bc1.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/616ede012bbd/vmlinux-5ce74bc1.xz
kernel image: https://storage.googleapis.com/syzbot-assets/7c0c5bc40fd8/bzImage-5ce74bc1.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+4393c4...@syzkaller.appspotmail.com

------------[ cut here ]------------
conntrack cleanup blocked for 60s
WARNING: net/netfilter/nf_conntrack_core.c:2512 at nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511, CPU#1: kworker/u8:0/12
Modules linked in:
CPU: 1 UID: 0 PID: 12 Comm: kworker/u8:0 Tainted: G L syzkaller #0 PREEMPT(full)
Tainted: [L]=SOFTLOCKUP
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/25/2025
Workqueue: netns cleanup_net
RIP: 0010:nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511
Code: 08 48 89 df e8 fd 78 a3 f8 4c 8b 3b 49 39 df 74 69 e8 20 16 3d f8 45 31 e4 e9 8e fe ff ff e8 13 16 3d f8 48 8d 3d ec 2f 0c 06 <67> 48 0f b9 3a eb c0 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c cd fe ff
RSP: 0018:ffffc90000117870 EFLAGS: 00010293
RAX: ffffffff8984a13d RBX: ffffc90000117a00 RCX: ffff88801ce8db80
RDX: 0000000000000000 RSI: ffffffffffffffcb RDI: ffffffff8f90d130
RBP: 0000000000000001 R08: ffff88807ce6b003 R09: 1ffff1100f9cd600
R10: dffffc0000000000 R11: ffffed100f9cd601 R12: 0000000000000001
R13: dffffc0000000000 R14: 00000001000068e3 R15: 0000000100006918
FS: 0000000000000000(0000) GS:ffff888125f32000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000020000 CR3: 00000000306ac000 CR4: 00000000003526f0
Call Trace:
<TASK>
ops_exit_list net/core/net_namespace.c:205 [inline]
ops_undo_list+0x525/0x990 net/core/net_namespace.c:252
cleanup_net+0x4d8/0x7a0 net/core/net_namespace.c:696
process_one_work+0x93a/0x15a0 kernel/workqueue.c:3279
process_scheduled_works kernel/workqueue.c:3362 [inline]
worker_thread+0x9b0/0xee0 kernel/workqueue.c:3443
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
</TASK>
----------------
Code disassembly (best guess):
0: 08 48 89 or %cl,-0x77(%rax)
3: df e8 fucomip %st(0),%st
5: fd std
6: 78 a3 js 0xffffffab
8: f8 clc
9: 4c 8b 3b mov (%rbx),%r15
c: 49 39 df cmp %rbx,%r15
f: 74 69 je 0x7a
11: e8 20 16 3d f8 call 0xf83d1636
16: 45 31 e4 xor %r12d,%r12d
19: e9 8e fe ff ff jmp 0xfffffeac
1e: e8 13 16 3d f8 call 0xf83d1636
23: 48 8d 3d ec 2f 0c 06 lea 0x60c2fec(%rip),%rdi # 0x60c3016
* 2a: 67 48 0f b9 3a ud1 (%edx),%rdi <-- trapping instruction
2f: eb c0 jmp 0xfffffff1
31: 89 e9 mov %ebp,%ecx
33: 80 e1 07 and $0x7,%cl
36: 80 c1 03 add $0x3,%cl
39: 38 c1 cmp %al,%cl
3b: 0f .byte 0xf
3c: 8c cd mov %cs,%ebp
3e: fe (bad)
3f: ff .byte 0xff


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Jakub Kicinski]

On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
> ------------[ cut here ]------------
> conntrack cleanup blocked for 60s
> WARNING: net/netfilter/nf_conntrack_core.c:2512 at

Yes, I was about to comment on the patch which added the warning..

There is still a leak somewhere. Running ip_defrag.sh and then load /
unload ipvlan repros this (modprobe ipvlan is a quick check if the
cleanup thread is wedged, if it is modprobe will hang, if it isn't
run ip_defrag.sh, again etc).

I looked around last night but couldn't find an skb stuck anywhere.
The nf_conntrack_net->count was == 1

[Florian Westphal]

Its caused skb skb fraglist skbs that still hold nf_conn references
on the softnet data defer lists.

setting net.core.skb_defer_max=0 makes the hang disappear for me.

[Eric Dumazet]

What kind of packets ? TCP ones ?

[Florian Westphal]

Eric Dumazet <edum...@google.com> wrote:
> > > I looked around last night but couldn't find an skb stuck anywhere.
> > > The nf_conntrack_net->count was == 1
> >
> > Its caused skb skb fraglist skbs that still hold nf_conn references
> > on the softnet data defer lists.
> >
> > setting net.core.skb_defer_max=0 makes the hang disappear for me.
>
> What kind of packets ? TCP ones ?

UDP, but I can't say yet if thats an udp specific issue or not.
(the packets are generated via ip_defrag.c).

[Eric Dumazet]

skb_release_head_state() does not follow the fraglist. Oh well.

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index a00808f7be6a1b86c595183f8b131996e3d0afcc..f597769d8c206dc063b53938a18edbe9620101d9
100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)

DEBUG_NET_WARN_ON_ONCE(!in_softirq());

- if (skb->alloc_cpu != smp_processor_id() && !skb_shared(skb)) {
+ if (skb->alloc_cpu != smp_processor_id() &&
+ !skb_shared(skb) &&
+ !skb_has_frag_list(skb)) {
skb_release_head_state(skb);
return skb_attempt_defer_free(skb);
}

[Florian Westphal]

Eric Dumazet <edum...@google.com> wrote:
> > UDP, but I can't say yet if thats an udp specific issue or not.
> > (the packets are generated via ip_defrag.c).
>
> skb_release_head_state() does not follow the fraglist. Oh well.
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index a00808f7be6a1b86c595183f8b131996e3d0afcc..f597769d8c206dc063b53938a18edbe9620101d9
> 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)
>
> DEBUG_NET_WARN_ON_ONCE(!in_softirq());
>
> - if (skb->alloc_cpu != smp_processor_id() && !skb_shared(skb)) {
> + if (skb->alloc_cpu != smp_processor_id() &&
> + !skb_shared(skb) &&
> + !skb_has_frag_list(skb)) {
> skb_release_head_state(skb);
> return skb_attempt_defer_free(skb);

There is also:
skb_attempt_defer_free -> skb_attempt_defer_free

Alternatively we could export skb_defer_free_flush or
kick_defer_list_purge() and call that from nf_conntrack
net exit path.

I will investigate more closely on monday, I still don't
understand why fragments are conntracked in the first place.