Request assistance diagnosing netmap kernel panic
John E. Barfield
Greetings HardenedBSD Team,
I have an issue similar to the following issue from 2019:
https://groups.google.com/a/hardenedbsd.org/g/users/c/Vj5Hr98wzDQ/m/8vuudpNZBgAJ
While running Suricata 6.0 release in “netmap IPS mode” on HardenedBSD 12.2-RELEASE with an Intel X520 10Gbe NIC using iflib IX driver; attempts to kill the suricata process will cause kernel panics. I have the panic text output (pasted below) and several vmcore.X files available for upload upon request. In my suricata configuration I am simply copying packets from the ix0 interface to the hoststack ix0^ in order to perform routing after the IPS inspection occurs and from the hoststack ix0^ back to interface ix0 to send return response (also being inspected as well) traffic. Upon startup Suricata will load and I can ping the interface IP address on the internet and use this interface short term to access the internet. After a while usually about an hour the internet stops working entirely. I notice that I can kill the connection quickly by searching for and installing packages using the pkg subsystem for some reason.
When I restart suricata with “service suricata restart” the kernel panics during the stopping phase. I can reproduce this at will. Any attempt to restart or kill the suricata process panics the kernel while in netmap mode. With an assertion error similar to the aforementioned thread. I compiled a CURRENT kernel and world and just received more errors but the behavior is still reproducible at will. The server in question is a cisco UCS c200 m2 that we’re using for testing. Its an older Intel Hex Core Xeon chipset. I have disabled all offloading in rc.conf with ifconfig commands. I do have several ACPI errors that I’ve been ignoring from the advice of another thread. I hope that they are not somehow related and can post those upon request as well.
Any help or attempt to point us in the right direction is appreciated.
John Barfield
Chief Innovator
O: 214-506-8354
rc.conf … ix0
“inet X.X.X.X -tso4 -tso6 -lro -vlanhwtso -rxcsum -txcsum promisc mtu 1500”
CPU:
Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (2400.13-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> |
Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI> Autoboot in 0 seconds, hit [Enter] to boot or any other key to stop
AMD Features=0x2c000800<SYSCALL,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 103079215104 (98304 MB)
avail memory = 100170776576 (95530 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <Cisco0 CiscoUCS>
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s)
Panic error:
“panic: Assertion !init || nm_i == 0 failed at /usr/src/sys/net/iflib.c:912” (Full text panic at bottom)
Netmap Errors:
“netmap_transmit ix0 full hwcur 1265 hwtail 1266 qlen 2046”
Ix0 & ix1 both report the following:
“debugnet_any_ifnet_update: Bad dn_init result from ix0 (ifp 0xfffff80106bd6800), ignoring.”
Relevant Suricata.yml
netmap:
- interface: ix0
copy-iface: ix0^
- interface: ix0^
copy-iface: ix0
- interface: default
disable-promisc: no
copy-mode: ips
threads: auto
checksum-checks: yes
“Netmap Lock Order Reversal From /var/log/messages”
Dec 3 09:48:11 core-ips kernel: [80] lock order reversal:
Dec 3 09:48:11 core-ips kernel: [80] 1st 0xffffffff81fa6198 ifnet_sx (ifnet_sx, sx) @ /usr/src/sys/dev/netmap/netmap_freebsd.c:151
Dec 3 09:48:11 core-ips kernel: [80] 2nd 0xfffff80106bd7d78 iflib ctx lock (iflib ctx lock, sx) @ /usr/src/sys/net/iflib.c:798
Dec 3 09:48:11 core-ips kernel: [80] lock order iflib ctx lock -> ifnet_sx established at:
Dec 3 09:48:11 core-ips kernel: [80] #0 0xffffffff80c6eecd at witness_checkorder+0x46d
Dec 3 09:48:11 core-ips kernel: [80] #1 0xffffffff80c0a1c7 at _sx_xlock+0x67
Dec 3 09:48:11 core-ips kernel: [80] #2 0xffffffff80d1612f at if_addgroup+0x5f
Dec 3 09:48:11 core-ips kernel: [80] #3 0xffffffff80d1511a at if_attach_internal+0x9a
Dec 3 09:48:11 core-ips kernel: [80] #4 0xffffffff80d1f209 at ether_ifattach+0x29
Dec 3 09:48:11 core-ips kernel: [80] #5 0xffffffff80d2e883 at iflib_device_register+0x10c3
Dec 3 09:48:11 core-ips kernel: [80] #6 0xffffffff80d32c4b at iflib_device_attach+0xbb
Dec 3 09:48:11 core-ips kernel: [80] #7 0xffffffff80c3a812 at device_attach+0x3d2
Dec 3 09:48:11 core-ips kernel: [80] #8 0xffffffff80c3a3b0 at device_probe_and_attach+0x70
Dec 3 09:48:11 core-ips kernel: [80] #9 0xffffffff80c3ba28 at bus_generic_attach+0x18
Dec 3 09:48:11 core-ips kernel: [80] #10 0xffffffff80859911 at pci_attach+0xe1
Dec 3 09:48:11 core-ips kernel: [80] #11 0xffffffff80fabfb7 at acpi_pci_attach+0x17
Dec 3 09:48:11 core-ips kernel: [80] #12 0xffffffff80c3a812 at device_attach+0x3d2
Dec 3 09:48:11 core-ips kernel: [80] #13 0xffffffff80c3a3b0 at device_probe_and_attach+0x70
Dec 3 09:48:11 core-ips kernel: [80] #14 0xffffffff80c3ba28 at bus_generic_attach+0x18
Dec 3 09:48:11 core-ips kernel: [80] #15 0xffffffff80fafe3e at acpi_pcib_pci_attach+0x9e
Dec 3 09:48:11 core-ips kernel: [80] #16 0xffffffff80c3a812 at device_attach+0x3d2
Dec 3 09:48:11 core-ips kernel: [80] #17 0xffffffff80c3a3b0 at device_probe_and_attach+0x70
Dec 3 09:48:11 core-ips kernel: [80] lock order ifnet_sx -> iflib ctx lock attempted at:
Dec 3 09:48:11 core-ips kernel: [80] #0 0xffffffff80c6f82c at witness_checkorder+0xdcc
Dec 3 09:48:11 core-ips kernel: [80] #1 0xffffffff80c0a1c7 at _sx_xlock+0x67
Dec 3 09:48:11 core-ips kernel: [80] #2 0xffffffff80d37a3d at iflib_netmap_register+0x3d
Dec 3 09:48:11 core-ips kernel: [80] #3 0xffffffff807b4ff6 at netmap_hw_reg+0x36
Dec 3 09:48:11 core-ips kernel: [80] #4 0xffffffff807b2414 at netmap_do_regif+0x514
Dec 3 09:48:11 core-ips kernel: [80] #5 0xffffffff807b375f at netmap_ioctl+0x131f
Dec 3 09:48:11 core-ips kernel: [80] #6 0xffffffff807bd318 at netmap_ioctl_legacy+0x4b8
Dec 3 09:48:11 core-ips kernel: [80] #7 0xffffffff807b27dc at netmap_ioctl+0x39c
Dec 3 09:48:11 core-ips kernel: [80] #8 0xffffffff807b9bcb at freebsd_netmap_ioctl+0x8b
Dec 3 09:48:11 core-ips kernel: [80] #9 0xffffffff80aa070b at devfs_ioctl+0xbb
Dec 3 09:48:11 core-ips kernel: [80] #10 0xffffffff80cf5d11 at vn_ioctl+0x131
Dec 3 09:48:11 core-ips kernel: [80] #11 0xffffffff80aa0d84 at devfs_ioctl_f+0x24
Dec 3 09:48:11 core-ips kernel: [80] #12 0xffffffff80c74c1b at kern_ioctl+0x28b
Dec 3 09:48:11 core-ips kernel: [80] #13 0xffffffff80c748da at sys_ioctl+0x12a
Dec 3 09:48:11 core-ips kernel: [80] #14 0xffffffff812349aa at amd64_syscall+0x13a
Dec 3 09:48:11 core-ips kernel: [80] #15 0xffffffff8120800e at fast_syscall_common+0xf8
Kernel Panic:
Ddb text
panic: Assertion !init || nm_i == 0 failed at /usr/src/sys/net/iflib.c:912
cpuid = 11
time = 1606670627
__HardenedBSD_version = 1300061 __FreeBSD_version = 1300130
version = FreeBSD 13.0-CURRENT-HBSD #0 : Sat Nov 28 00:00:08 CST 2020
root@core-ips:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01967e8790
vpanic() at vpanic+0x186/frame 0xfffffe01967e87e0
panic() at panic+0x43/frame 0xfffffe01967e8840
netmap_fl_refill() at netmap_fl_refill+0x443/frame 0xfffffe01967e8920
iflib_init_locked() at iflib_init_locked+0x32a/frame 0xfffffe01967e8970
iflib_netmap_register() at iflib_netmap_register+0xfd/frame 0xfffffe01967e89b0
netmap_hw_reg() at netmap_hw_reg+0x36/frame 0xfffffe01967e89e0
netmap_do_unregif() at netmap_do_unregif+0x15a/frame 0xfffffe01967e8a10
netmap_dtor() at netmap_dtor+0x4b/frame 0xfffffe01967e8a30
netmap_dev_pager_dtor() at netmap_dev_pager_dtor+0x71/frame 0xfffffe01967e8a70
dev_pager_dealloc() at dev_pager_dealloc+0x3c/frame 0xfffffe01967e8aa0
vm_object_terminate() at vm_object_terminate+0x1f5/frame 0xfffffe01967e8ad0
vm_object_deallocate() at vm_object_deallocate+0x28f/frame 0xfffffe01967e8b00
vm_map_process_deferred() at vm_map_process_deferred+0xb0/frame 0xfffffe01967e8b20
vm_map_remove() at vm_map_remove+0xc9/frame 0xfffffe01967e8b50
vmspace_exit() at vmspace_exit+0xa9/frame 0xfffffe01967e8b80
exit1() at exit1+0x572/frame 0xfffffe01967e8bf0
sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe01967e8c00
amd64_syscall() at amd64_syscall+0x13a/frame 0xfffffe01967e8d30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01967e8d30
--- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x1caa332a54a, rsp = 0x788bdc3f6158, rbp = 0x788bdc3f6170 ---
Stacktrace:
Tracing pid 30 tid 100738 td 0xfffffe0197ae8300
kdb_enter() at kdb_enter+0x37/frame 0xfffffe01988e64f0
vpanic() at vpanic+0x1b7/frame 0xfffffe01988e6540
panic() at panic+0x43/frame 0xfffffe01988e65a0
netmap_fl_refill() at netmap_fl_refill+0x443/frame 0xfffffe01988e6680
iflib_init_locked() at iflib_init_locked+0x32a/frame 0xfffffe01988e66d0
iflib_netmap_register() at iflib_netmap_register+0xfd/frame 0xfffffe01988e6710
netmap_hw_reg() at netmap_hw_reg+0x36/frame 0xfffffe01988e6740
netmap_do_unregif() at netmap_do_unregif+0x15a/frame 0xfffffe01988e6770
netmap_dtor() at netmap_dtor+0x4b/frame 0xfffffe01988e6790
netmap_dev_pager_dtor() at netmap_dev_pager_dtor+0x71/frame 0xfffffe01988e67d0
dev_pager_dealloc() at dev_pager_dealloc+0x3c/frame 0xfffffe01988e6800
vm_object_terminate() at vm_object_terminate+0x1f5/frame 0xfffffe01988e6830
vm_object_deallocate() at vm_object_deallocate+0x28f/frame 0xfffffe01988e6860
vm_map_process_deferred() at vm_map_process_deferred+0xb0/frame 0xfffffe01988e6880
vm_map_remove() at vm_map_remove+0xc9/frame 0xfffffe01988e68b0
vmspace_exit() at vmspace_exit+0xa9/frame 0xfffffe01988e68e0
exit1() at exit1+0x572/frame 0xfffffe01988e6950
sigexit() at sigexit+0x15c/frame 0xfffffe01988e6c20
postsig() at postsig+0x2d9/frame 0xfffffe01988e6cf0
ast() at ast+0x46b/frame 0xfffffe01988e6d30
doreti_ast() at doreti_ast+0x1f/frame 0x71b5fa055ea0
####
db> acttrace
Tracing command suricata pid 30 tid 100738 td 0xfffffe0197ae8300 (CPU 2)
kdb_enter() at kdb_enter+0x37/frame 0xfffffe01988e64f0
vpanic() at vpanic+0x1b7/frame 0xfffffe01988e6540
panic() at panic+0x43/frame 0xfffffe01988e65a0
netmap_fl_refill() at netmap_fl_refill+0x443/frame 0xfffffe01988e6680
iflib_init_locked() at iflib_init_locked+0x32a/frame 0xfffffe01988e66d0
iflib_netmap_register() at iflib_netmap_register+0xfd/frame 0xfffffe01988e6710
netmap_hw_reg() at netmap_hw_reg+0x36/frame 0xfffffe01988e6740
netmap_do_unregif() at netmap_do_unregif+0x15a/frame 0xfffffe01988e6770
netmap_dtor() at netmap_dtor+0x4b/frame 0xfffffe01988e6790
netmap_dev_pager_dtor() at netmap_dev_pager_dtor+0x71/frame 0xfffffe01988e67d0
dev_pager_dealloc() at dev_pager_dealloc+0x3c/frame 0xfffffe01988e6800
vm_object_terminate() at vm_object_terminate+0x1f5/frame 0xfffffe01988e6830
vm_object_deallocate() at vm_object_deallocate+0x28f/frame 0xfffffe01988e6860
vm_map_process_deferred() at vm_map_process_deferred+0xb0/frame 0xfffffe01988e6880
vm_map_remove() at vm_map_remove+0xc9/frame 0xfffffe01988e68b0
vmspace_exit() at vmspace_exit+0xa9/frame 0xfffffe01988e68e0
exit1() at exit1+0x572/frame 0xfffffe01988e6950
sigexit() at sigexit+0x15c/frame 0xfffffe01988e6c20
postsig() at postsig+0x2d9/frame 0xfffffe01988e6cf0
ast() at ast+0x46b/frame 0xfffffe01988e6d30
doreti_ast() at doreti_ast+0x1f/frame 0x71b5fa055ea0
Tracing command idle pid 11 tid 100003 td 0xfffffe0114509300 (CPU 0)
cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff820026a0
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff820026b0
trap() at trap+0x56/frame 0xffffffff820027c0
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff820027c0
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3b5b90, rbp = 0xfffffe009f3b5b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3b5b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3b5be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3b5c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3b5c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3b5cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3b5d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3b5d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100004 td 0xfffffe0114508c00 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a4896e00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a4896e10
trap() at trap+0x56/frame 0xfffffe00a4896f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a4896f20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3bab90, rbp = 0xfffffe009f3bab90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3bab90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3babe0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3bac00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3bac20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3bacf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3bad30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3bad30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100006 td 0xfffffe0114507e00 (CPU 3)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48aee00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48aee10
trap() at trap+0x56/frame 0xfffffe00a48aef20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48aef20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3c4b90, rbp = 0xfffffe009f3c4b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3c4b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3c4be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3c4c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3c4c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3c4cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3c4d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3c4d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100007 td 0xfffffe0114507700 (CPU 4)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48bae00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48bae10
trap() at trap+0x56/frame 0xfffffe00a48baf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48baf20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3c9b90, rbp = 0xfffffe009f3c9b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3c9b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3c9be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3c9c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3c9c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3c9cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3c9d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3c9d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100008 td 0xfffffe0114507000 (CPU 5)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48c6e00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48c6e10
trap() at trap+0x56/frame 0xfffffe00a48c6f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48c6f20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3ceb90, rbp = 0xfffffe009f3ceb90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3ceb90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3cebe0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3cec00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3cec20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3cecf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3ced30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3ced30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100009 td 0xfffffe011450a800 (CPU 6)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48d2e00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48d2e10
trap() at trap+0x56/frame 0xfffffe00a48d2f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48d2f20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3d3b90, rbp = 0xfffffe009f3d3b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3d3b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3d3be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3d3c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3d3c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3d3cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3d3d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3d3d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100010 td 0xfffffe0114510100 (CPU 7)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48dee00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48dee10
trap() at trap+0x56/frame 0xfffffe00a48def20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48def20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3d8b90, rbp = 0xfffffe009f3d8b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3d8b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3d8be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3d8c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3d8c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3d8cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3d8d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3d8d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100011 td 0xfffffe011450fa00 (CPU 8)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48eae00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48eae10
trap() at trap+0x56/frame 0xfffffe00a48eaf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48eaf20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3ddb90, rbp = 0xfffffe009f3ddb90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3ddb90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3ddbe0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3ddc00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3ddc20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3ddcf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3ddd30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3ddd30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100012 td 0xfffffe011450f300 (CPU 9)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a48f6e00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a48f6e10
trap() at trap+0x56/frame 0xfffffe00a48f6f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a48f6f20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3e2b90, rbp = 0xfffffe009f3e2b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3e2b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3e2be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3e2c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3e2c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3e2cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3e2d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3e2d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100013 td 0xfffffe011450ec00 (CPU 10)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a4902e00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a4902e10
trap() at trap+0x56/frame 0xfffffe00a4902f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a4902f20
--- trap 0x13, rip = 0xffffffff811f00c0, rsp = 0xfffffe009f3e7b90, rbp = 0xfffffe009f3e7b90 ---
acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x70/frame 0xfffffe009f3e7b90
acpi_cpu_idle() at acpi_cpu_idle+0x1ca/frame 0xfffffe009f3e7be0
cpu_idle_acpi() at cpu_idle_acpi+0x41/frame 0xfffffe009f3e7c00
cpu_idle() at cpu_idle+0xa5/frame 0xfffffe009f3e7c20
sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe009f3e7cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3e7d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3e7d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100015 td 0xfffffe011450de00 (CPU 11)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00a490ee00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00a490ee10
trap() at trap+0x56/frame 0xfffffe00a490ef20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00a490ef20
--- trap 0x13, rip = 0xffffffff80c4d7b0, rsp = 0xfffffe009f3f1b00, rbp = 0xfffffe009f3f1b10 ---
lock_delay() at lock_delay+0x40/frame 0xfffffe009f3f1b10
_sx_xlock_hard() at _sx_xlock_hard+0x1a9/frame 0xfffffe009f3f1bb0
_sx_xlock() at _sx_xlock+0xba/frame 0xfffffe009f3f1bf0
_task_fn_admin() at _task_fn_admin+0xd1/frame 0xfffffe009f3f1c40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0xad/frame 0xfffffe009f3f1cc0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x97/frame 0xfffffe009f3f1cf0
fork_exit() at fork_exit+0x85/frame 0xfffffe009f3f1d30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f3f1d30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---