panic: pool_cache_item_magic_check: mbufpl cpu free list modified: item addr ADDR+16 0x0!=ADDR

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 4e9c41985603 Regenerate root CA list using updated format-..
git tree: https://github.com/openbsd/src.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=174381db400000
kernel config: https://syzkaller.appspot.com/x/.config?x=f2ee3db928411249
dashboard link: https://syzkaller.appspot.com/bug?extid=6237a20c91fa048719ea
compiler:

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+6237a2...@syzkaller.appspotmail.com

ppmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wiring for
pmap 0xffffff007f1233c0 pmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wiring for pmap 0xfpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpmap_unwire: wirpmap_unwire:
wirpmap_unwire: wirpmap_unwire: wirpanic: pool_cache_item_magic_check:
mbufpl cpu free list modified: item addr 0xffffff0006000100+16
0x0!=0xff9fb1b997772a78
Stopped at db_enter+0xa: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*507572 80458 0 0 0x4000000 0K syz-executor0
228533 22515 0 0x2 0x480 1 syz-fuzzer
db_enter() at db_enter+0xa sys/arch/amd64/amd64/db_interface.c:399
panic() at panic+0x147 sys/kern/subr_prf.c:208
pool_cache_get(2) at pool_cache_get+0x2bf pool_cache_item_magic_check
sys/kern/subr_pool.c:1789 [inline]
pool_cache_get(2) at pool_cache_get+0x2bf sys/kern/subr_pool.c:1892
pool_get(1,2) at pool_get+0x60 sys/kern/subr_pool.c:577
m_get(10000,ff5aff91) at m_get+0x2f sys/kern/uipc_mbuf.c:237
switchwrite(ffffff00614b8920,ffffff00614b8920,ffff8000211694a8) at
switchwrite+0x1d3 sys/net/switchctl.c:251
spec_write(ffffffff81e226e0) at spec_write+0xa8 sys/kern/spec_vnops.c:310
VOP_WRITE(1,ffffff00614b8920,1,ffffff0067c5de98) at VOP_WRITE+0x65
sys/kern/vfs_vops.c:268
vn_write(ffffff0067c5de98,ffff8000211694a8,ffffff91) at vn_write+0x161
sys/kern/vfs_vnops.c:397
dofilewritev(ffff8000211695d0,1,ffff8000211695e8,ffff8000210a2978,0) at
dofilewritev+0x13e sys/kern/sys_generic.c:364
sys_pwritev(10c0,ffff8000210a2978,0) at sys_pwritev+0xbf
sys/kern/vfs_syscalls.c:3141
syscall(0) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:99 [inline]
syscall(0) at syscall+0x489 sys/arch/amd64/amd64/trap.c:583
Xsyscall(6,0,ffffffffffffffb8,0,4,2a8332f010) at Xsyscall+0x128
end of kernel
end trace frame: 0x2d60e31c70, count: 2
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> set $lines = 0
ddb{0}> show panic
pool_cache_item_magic_check: mbufpl cpu free list modified: item addr
0xffffff0006000100+16 0x0!=0xff9fb1b997772a78
ddb{0}> trace
db_enter() at db_enter+0xa sys/arch/amd64/amd64/db_interface.c:399
panic() at panic+0x147 sys/kern/subr_prf.c:208
pool_cache_get(2) at pool_cache_get+0x2bf pool_cache_item_magic_check
sys/kern/subr_pool.c:1789 [inline]
pool_cache_get(2) at pool_cache_get+0x2bf sys/kern/subr_pool.c:1892
pool_get(1,2) at pool_get+0x60 sys/kern/subr_pool.c:577
m_get(10000,ff5aff91) at m_get+0x2f sys/kern/uipc_mbuf.c:237
switchwrite(ffffff00614b8920,ffffff00614b8920,ffff8000211694a8) at
switchwrite+0x1d3 sys/net/switchctl.c:251
spec_write(ffffffff81e226e0) at spec_write+0xa8 sys/kern/spec_vnops.c:310
VOP_WRITE(1,ffffff00614b8920,1,ffffff0067c5de98) at VOP_WRITE+0x65
sys/kern/vfs_vops.c:268
vn_write(ffffff0067c5de98,ffff8000211694a8,ffffff91) at vn_write+0x161
sys/kern/vfs_vnops.c:397
dofilewritev(ffff8000211695d0,1,ffff8000211695e8,ffff8000210a2978,0) at
dofilewritev+0x13e sys/kern/sys_generic.c:364
sys_pwritev(10c0,ffff8000210a2978,0) at sys_pwritev+0xbf
sys/kern/vfs_syscalls.c:3141
syscall(0) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:99 [inline]
syscall(0) at syscall+0x489 sys/arch/amd64/amd64/trap.c:583
Xsyscall(6,0,ffffffffffffffb8,0,4,2a8332f010) at Xsyscall+0x128
end of kernel
end trace frame: 0x2d60e31c70, count: -13
ddb{0}> show registers
rdi 0xffffffff81e35298 kprintf_mutex
rsi 0x5
rbp 0xffff800021169100
rbx 0xffff8000211691a0
rdx 0xffff800000ad6000
rcx 0x3ffff acpi_pdirpa+0x2be67
rax 0xffff80000005cd40
r8 0xffff8000211690d0
r9 0x8080808080808080
r10 0
r11 0xffffffff81714120 x86_bus_space_io_read_1
r12 0x3000000008
r13 0xffff800021169110
r14 0x100
r15 0xffffffff81bf574a cmd0646_9_tim_udma+0x1eb03
rip 0xffffffff8181218a db_enter+0xa
cs 0x8
rflags 0x246
rsp 0xffff800021169100
ss 0x10
db_enter+0xa: popq %rbp
ddb{0}> show proc
PROC (syz-executor0) pid=507572 stat=onproc
flags process=0 proc=4000000<THREAD>
pri=86, usrpri=86, nice=20
forw=0xffffffffffffffff, list=0xffff8000210a2018,0xffffffff81eb2728
process=0xffff8000210b7028 user=0xffff800021164000,
vmspace=0xffffff007f124d68
estcpu=36, cpticks=3, pctcpu=0.0
user=0, sys=3, intr=0
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
80458 417016 39639 0 2 0 syz-executor0
*80458 507572 39639 0 7 0x4000000 syz-executor0
2824 213630 25968 0 2 0x480 syz-executor1
2824 38602 25968 0 3 0x4000080 kqread syz-executor1
2824 180336 25968 0 3 0x4000080 fsleep syz-executor1
2824 266226 25968 0 3 0x4000080 fsleep syz-executor1
4533 464977 0 0 3 0x14200 bored sosplice
25968 513744 22515 0 2 0x482 syz-executor1
39639 488126 22515 0 2 0x482 syz-executor0
22515 228533 93864 0 7 0x482 syz-fuzzer
22515 172034 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 46679 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 451356 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 240447 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 77219 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 133623 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 500315 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 312653 93864 0 3 0x4000082 kqread syz-fuzzer
22515 110408 93864 0 3 0x4000082 thrsleep syz-fuzzer
22515 344455 93864 0 3 0x4000082 thrsleep syz-fuzzer
93864 327913 8073 0 3 0x10008a pause ksh
8073 31789 36518 0 3 0x92 select sshd
36601 509184 1 0 3 0x100083 ttyin getty
36518 298578 1 0 3 0x80 select sshd
32987 1752 14192 73 3 0x100090 kqread syslogd
14192 426988 1 0 3 0x100082 netio syslogd
56563 11914 1 77 3 0x100090 poll dhclient
62161 322898 1 0 3 0x80 poll dhclient
24236 190739 0 0 2 0x14200 zerothread
56520 479596 0 0 3 0x14200 aiodoned aiodoned
97627 18707 0 0 3 0x14200 syncer update
77538 136624 0 0 3 0x14200 cleaner cleaner
66376 104008 0 0 3 0x14200 reaper reaper
75761 216769 0 0 3 0x14200 pgdaemon pagedaemon
61930 398973 0 0 3 0x14200 bored crynlk
87817 65055 0 0 3 0x14200 bored crypto
88620 161788 0 0 3 0x40014200 acpi0 acpi0
75403 65081 0 0 3 0x40014200 idle1
7277 253276 0 0 3 0x14200 bored softnet
94435 414699 0 0 3 0x14200 bored systqmp
98910 457847 0 0 3 0x14200 bored systq
86877 425275 0 0 3 0x40014200 bored softclock
27446 452864 0 0 3 0x40014200 idle0
1 326219 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.

[Anton Lindqvist]

#syz fix: Fix mbuf releated crashes in switch(4). They have been found by syzkaller as pool corruption panic. It is unclear which bug caused what, but it should be better now. - Check M_PKTHDR with assertion before accessing m_pkthdr. - Do not access oh_length without m_pullup(). - After checking if there is space at the end of the mbuf, don't overwrite the data at the beginning. Append the new content. - Do not set m_len and m_pkthdr.len when it is unclear whether the ofp_error header fits at all. Use m_makespace() to adjust the mbuf. Reported-by: syzbot+6efc0a...@syzkaller.appspotmail.com test akoshibe@; OK claudio@

Let's see if this fix is enough to close all switch related panics.

[Anton Lindqvist]

[Anton Lindqvist]