FreeBSD-HEAD gets stuck on vnode operations

[Roger Pau Monné]

Hello,

I've set up a FreeBSD-HEAD VM on Xen, and compiled the XENHVM kernel,
last commit in the repository is:

Date: Tue, 7 May 2013 12:39:14 +0000
Subject: [PATCH] By request, add an arrow from NetBSD-0.8 to FreeBSD-1.0.

While here, add a few more NetBSD versions to the tree itself.

Submitted by: Alan Barrett <a...@cequrux.com>
Submitted by: Thomas Klausner <w...@netbsd.org>

And I've started stressing the VM using the following tight loop:

while [ 1 ]; do make -j12 buildkernel; done

From time to time, I see the VM getting stuck and when breaking into
KDB this is the output of ps:

pid ppid pgrp uid state wmesg wchan cmd
32343 670 670 25 L *Name Cac 0xfffffe0017183780 sendmail
32342 32256 5273 0 L+ *Name Cac 0xfffffe0017183780 sh
32341 32335 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32340 32284 5273 0 D+ ufs 0xfffffe000dc76068 ctfconvert
32339 32293 5273 0 L+ *Name Cac 0xfffffe0017183780 ctfconvert
32337 32332 5273 0 L+ *vnode_fr 0xfffffe00872c23c0 cc
32335 32334 5273 0 S+ wait 0xfffffe010537d000 cc
32334 30655 5273 0 S+ wait 0xfffffe010ac974b8 sh
32333 32329 5273 0 L+ *vnode in 0xfffffe0017183180 cc
32332 32331 5273 0 S+ wait 0xfffffe00416a14b8 cc
32331 30655 5273 0 S+ wait 0xfffffe00b9846970 sh
32329 32328 5273 0 S+ wait 0xfffffe0049ace000 cc
32328 30655 5273 0 S+ wait 0xfffffe0041670000 sh
32324 32320 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32320 32318 5273 0 S+ wait 0xfffffe00416184b8 cc
32318 30655 5273 0 S+ wait 0xfffffe00879fd970 sh
32314 32313 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32313 32311 5273 0 S+ wait 0xfffffe00b9846000 cc
32312 32309 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32311 30655 5273 0 S+ wait 0xfffffe00415844b8 sh
32310 32305 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32309 32307 5273 0 S+ wait 0xfffffe00499624b8 cc
32307 30655 5273 0 S+ wait 0xfffffe010537c970 sh
32305 32304 5273 0 S+ wait 0xfffffe0041669000 cc
32304 30655 5273 0 S+ wait 0xfffffe00173d9970 sh
32303 32298 5273 0 L+ *Name Cac 0xfffffe0017183780 cc
32298 32295 5273 0 S+ wait 0xfffffe0049924000 cc
32295 30655 5273 0 S+ wait 0xfffffe0041631970 sh
32293 30655 5273 0 S+ wait 0xfffffe000d15b000 sh
32284 30655 5273 0 S+ wait 0xfffffe00416684b8 sh
32256 31391 5273 0 S+ select 0xfffffe000d965840 make
32022 30655 5273 0 L+ *Name Cac 0xfffffe0017183780 sh
31391 31386 5273 0 S+ wait 0xfffffe0041680970 sh
31386 30664 5273 0 S+ select 0xfffffe0017942cc0 make
30664 30663 5273 0 S+ wait 0xfffffe004169f4b8 sh
30663 30662 5273 0 S+ select 0xfffffe000d1c0d40 make
30662 30655 5273 0 S+ wait 0xfffffe00b9ddb4b8 sh
30655 5287 5273 0 S+ select 0xfffffe000d1c0040 make
5287 5280 5273 0 S+ wait 0xfffffe004148f970 sh
5280 5278 5273 0 S+ select 0xfffffe000d964540 make
5278 5273 5273 0 S+ wait 0xfffffe000de3a970 sh
5273 736 5273 0 S+ select 0xfffffe000bc81740 make
91658 735 91658 0 Ss+ ttyin 0xfffffe000d6caca8 bash
736 735 736 0 Ss+ wait 0xfffffe000d5084b8 bash
735 734 735 0 Ss select 0xfffffe000bc817c0 screen
734 730 734 0 S+ pause 0xfffffe000d6330a8 screen
730 729 730 0 S+ wait 0xfffffe000d58b000 bash
729 728 729 1001 S+ wait 0xfffffe000d7ab970 su
728 727 728 1001 Ss+ wait 0xfffffe000de3a4b8 sh
727 724 724 1001 S select 0xfffffe000d98f340 sshd
724 664 724 0 Ss select 0xfffffe000d4180c0 sshd
722 721 722 0 L+ *Name Cac 0xfffffe0017183780 bash
721 1 721 0 Ss+ wait 0xfffffe000d7ab000 login
720 1 720 0 Ss+ ttyin 0xfffffe000d0470a8 getty
719 1 719 0 Ss+ ttyin 0xfffffe000d0474a8 getty
718 1 718 0 Ss+ ttyin 0xfffffe000bd914a8 getty
717 1 717 0 Ss+ ttyin 0xfffffe000bd918a8 getty
716 1 716 0 Ss+ ttyin 0xfffffe000bd91ca8 getty
715 1 715 0 Ss+ ttyin 0xfffffe000d0440a8 getty
714 1 714 0 Ss+ ttyin 0xfffffe000d0444a8 getty
713 1 713 0 Ss+ ttyin 0xfffffe000d0448a8 getty
674 1 674 0 Ls *Name Cac 0xfffffe0017183780 cron
670 1 670 25 Ss pause 0xfffffe000d5090a8 sendmail
667 1 667 0 Ls *Name Cac 0xfffffe0017183780 sendmail
664 1 664 0 Ss select 0xfffffe000d964840 sshd
566 1 566 0 Ss select 0xfffffe000d98f4c0 syslogd
472 1 472 0 Ss select 0xfffffe000bc81840 devd
471 1 471 65 Ls *Name Cac 0xfffffe0017183780 dhclient
431 1 431 0 Ss select 0xfffffe000d98fc40 dhclient
18 0 0 0 LL *Name Cac 0xfffffe0017183780 [softdepflush]
17 0 0 0 RL CPU 1 [syncer]
9 0 0 0 DL vlruwt 0xfffffe000d15c000 [vnlru]
8 0 0 0 DL psleep 0xffffffff815305ac [bufdaemon]
7 0 0 0 DL pgzero 0xffffffff81538c94 [pagezero]
6 0 0 0 DL psleep 0xffffffff81537ed0 [vmdaemon]
5 0 0 0 DL psleep 0xffffffff815a4084 [pagedaemon]
4 0 0 0 DL ccb_scan 0xffffffff81300f30 [xpt_thrd]
3 0 0 0 DL waiting_ 0xffffffff81598160 [sctp_iterator]
2 0 0 0 DL balloon 0xffffffff8079b9f0 [balloon]
16 0 0 0 SL xbread 0xfffffe00feffc000 [xenstore_rcv]
15 0 0 0 SL waitev 0xffffffff81538e98 [xenwatch]
14 0 0 0 DL - 0xffffffff8131d008 [yarrow]
13 0 0 0 DL (threaded) [geom]
100023 D - 0xffffffff8158ca50 [g_down]
100022 D - 0xffffffff8158ca48 [g_up]
100021 D - 0xffffffff8158ca40 [g_event]
12 0 0 0 WL (threaded) [intr]
100044 I [irq7: ppc0]
100043 I [swi0: uart]
100042 I [irq12: psm0]
100041 I [irq1: atkbd0]
100038 I [irq28: xenpci0]
100037 I [irq15: ata1]
100036 I [irq14: ata0]
100032 I [swi2: cambio]
100030 I [swi5: fast taskq]
100028 I [swi6: Giant taskq]
100027 I [swi6: task queue]
100020 I [swi3: vm]
100019 I [swi1: netisr 0]
100018 I [swi4: clock]
100017 I [swi4: clock]
100016 I [swi4: clock]
100015 I [swi4: clock]
100014 I [swi4: clock]
100013 I [swi4: clock]
100012 I [swi4: clock]
100011 I [swi4: clock]
11 0 0 0 RL (threaded) [idle]
100010 Run CPU 7 [idle: cpu7]
100009 Run CPU 6 [idle: cpu6]
100008 Run CPU 5 [idle: cpu5]
100007 Run CPU 4 [idle: cpu4]
100006 Run CPU 3 [idle: cpu3]
100005 Run CPU 2 [idle: cpu2]
100004 CanRun [idle: cpu1]
100003 Run CPU 0 [idle: cpu0]
1 0 1 0 SLs wait 0xfffffe000bc86970 [init]
10 0 0 0 DL audit_wo 0xffffffff8159b860 [audit]
0 0 0 0 DLs (threaded) [kernel]
100046 D - 0xffffffff81343440 [deadlkres]
100045 D - 0xfffffe000bcbd600 [mca taskq]
100035 D - 0xfffffe000bcbd700 [acpi_task_2]
100034 D - 0xfffffe000bcbd700 [acpi_task_1]
100033 D - 0xfffffe000bcbd700 [acpi_task_0]
100031 D - 0xfffffe000bcbd900 [kqueue taskq]
100029 D - 0xfffffe000bcbdb00 [thread taskq]
100026 D - 0xfffffe000bcbde00 [ffs_trim taskq]
100024 D - 0xfffffe000bcbe000 [firmware taskq]
100000 D sched 0xffffffff8158cb48 [swapper]
db> tr 17
Tracing pid 17 tid 100055 td 0xfffffe000d14d000
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff8000232cf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff8000232d10
trap() at trap+0x42/frame 0xffffff8000232f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff8000232f20
--- trap 0x13, rip = 0xffffffff808b4bd6, rsp = 0xffffff8000232fe0, rbp = 0xffffff8120c569a0 ---
_mtx_trylock_flags_() at _mtx_trylock_flags_+0x106/frame 0xffffff8120c569a0
mnt_vnode_next_active() at mnt_vnode_next_active+0x141/frame 0xffffff8120c569e0
ffs_sync() at ffs_sync+0xb5/frame 0xffffff8120c56ab0
sync_fsync() at sync_fsync+0x12d/frame 0xffffff8120c56ae0
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf0/frame 0xffffff8120c56b10
sched_sync() at sched_sync+0x34c/frame 0xffffff8120c56bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c56bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c56bf0
--- trap 0, rip = 0, rsp = 0xffffff8120c56cb0, rbp = 0 ---

The VM can be stuck in this state for quite some time, it generally
varies between a couple of minutes (5-10min) to an hour or two, after
this the VM recovers itself and resumes normal operation. I still have
to test this on a bare metal FreeBSD install, but I would like to ask
if someone has seen a similar behaviour, or if someone is suspicious of
a change that could cause this.

Thanks, Roger.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Roger Pau Monné]

A little bit more of debug info:

db> show allpcpu
Current CPU: 2

cpuid = 0
dynamic pcpu = 0x5d6f00
curthread = 0xfffffe000bc88490: pid 11 "idle: cpu0"
curpcb = 0xffffff800026dcc0
fpcurthread = none
idlethread = 0xfffffe000bc88490: tid 100003 "idle: cpu0"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a4210
commontssp = 0xffffffff815a4210
rsp0 = 0xffffff800026dcc0
gs32p = 0xffffffff815a5c88
ldt = 0xffffffff815a5cc8
tss = 0xffffffff815a5cb8
spin locks held:

cpuid = 1
dynamic pcpu = 0xffffff807ef33f00
curthread = 0xfffffe000d14d000: pid 17 "syncer"
curpcb = 0xffffff8120c56cc0
fpcurthread = none
idlethread = 0xfffffe000bc88000: tid 100004 "idle: cpu1"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a4278
commontssp = 0xffffffff815a4278
rsp0 = 0xffffff8120c56cc0
gs32p = 0xffffffff815a5cf0
ldt = 0xffffffff815a5d30
tss = 0xffffffff815a5d20
spin locks held:

cpuid = 2
dynamic pcpu = 0xffffff807ef3af00
curthread = 0xfffffe000bc94920: pid 11 "idle: cpu2"
curpcb = 0xffffff8000277cc0
fpcurthread = none
idlethread = 0xfffffe000bc94920: tid 100005 "idle: cpu2"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a42e0
commontssp = 0xffffffff815a42e0
rsp0 = 0xffffff8000277cc0
gs32p = 0xffffffff815a5d58
ldt = 0xffffffff815a5d98
tss = 0xffffffff815a5d88
spin locks held:

cpuid = 3
dynamic pcpu = 0xffffff807ef41f00
curthread = 0xfffffe000bc94490: pid 11 "idle: cpu3"
curpcb = 0xffffff800027ccc0
fpcurthread = none
idlethread = 0xfffffe000bc94490: tid 100006 "idle: cpu3"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a4348
commontssp = 0xffffffff815a4348
rsp0 = 0xffffff800027ccc0
gs32p = 0xffffffff815a5dc0
ldt = 0xffffffff815a5e00
tss = 0xffffffff815a5df0
spin locks held:

cpuid = 4
dynamic pcpu = 0xffffff807ef48f00
curthread = 0xfffffe000bc94000: pid 11 "idle: cpu4"
curpcb = 0xffffff8000281cc0
fpcurthread = none
idlethread = 0xfffffe000bc94000: tid 100007 "idle: cpu4"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a43b0
commontssp = 0xffffffff815a43b0
rsp0 = 0xffffff8000281cc0
gs32p = 0xffffffff815a5e28
ldt = 0xffffffff815a5e68
tss = 0xffffffff815a5e58
spin locks held:

cpuid = 5
dynamic pcpu = 0xffffff807ef4ff00
curthread = 0xfffffe000bc89920: pid 11 "idle: cpu5"
curpcb = 0xffffff8000286cc0
fpcurthread = none
idlethread = 0xfffffe000bc89920: tid 100008 "idle: cpu5"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a4418
commontssp = 0xffffffff815a4418
rsp0 = 0xffffff8000286cc0
gs32p = 0xffffffff815a5e90
ldt = 0xffffffff815a5ed0
tss = 0xffffffff815a5ec0
spin locks held:

cpuid = 6
dynamic pcpu = 0xffffff807ef56f00
curthread = 0xfffffe000bc89490: pid 11 "idle: cpu6"
curpcb = 0xffffff800028bcc0
fpcurthread = none
idlethread = 0xfffffe000bc89490: tid 100009 "idle: cpu6"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a4480
commontssp = 0xffffffff815a4480
rsp0 = 0xffffff800028bcc0
gs32p = 0xffffffff815a5ef8
ldt = 0xffffffff815a5f38
tss = 0xffffffff815a5f28
spin locks held:

cpuid = 7
dynamic pcpu = 0xffffff807ef5df00
curthread = 0xfffffe000bc96920: pid 11 "idle: cpu7"
curpcb = 0xffffff8000290cc0
fpcurthread = none
idlethread = 0xfffffe000bc96920: tid 100010 "idle: cpu7"
curpmap = 0xffffffff8158d5b0
tssp = 0xffffffff815a44e8
commontssp = 0xffffffff815a44e8
rsp0 = 0xffffff8000290cc0
gs32p = 0xffffffff815a5f60
ldt = 0xffffffff815a5fa0
tss = 0xffffffff815a5f90
spin locks held:


db> show alllocks
Process 32343 (sendmail) thread 0xfffffe004157a000 (101356)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32342 (sh) thread 0xfffffe000d7ad490 (100171)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05b5f0) locked @ /usr/home/royger/freebsd/sys/kern/vfs_subr.c:2168
Process 32339 (ctfconvert) thread 0xfffffe000d652490 (100121)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000dc76068) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32337 (cc) thread 0xfffffe0041e75490 (101363)
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xfffffe00d78d5270) locked @ /usr/home/royger/freebsd/sys/kern/vfs_subr.c:2520
Process 32333 (cc) thread 0xfffffe00173a8490 (100328)
shared rw Name Cache (Name Cache) r = 0 (0xffffffff815306e8) locked @ /usr/home/royger/freebsd/sys/kern/vfs_cache.c:488
shared lockmgr ufs (ufs) r = 0 (0xfffffe000dc76068) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32314 (cc) thread 0xfffffe00413fd000 (100215)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32312 (cc) thread 0xfffffe008710a920 (100149)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32310 (cc) thread 0xfffffe000d68e490 (100118)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32303 (cc) thread 0xfffffe000d614490 (100099)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 32022 (sh) thread 0xfffffe0041562000 (100259)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 722 (bash) thread 0xfffffe000d664000 (100090)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d05bd50) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 674 (cron) thread 0xfffffe000d6de920 (100137)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000dc2bb78) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 667 (sendmail) thread 0xfffffe000d60a490 (100103)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000dc17068) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 471 (dhclient) thread 0xfffffe000d02f920 (100060)
shared lockmgr ufs (ufs) r = 0 (0xfffffe000d6ba9a0) locked @ /usr/home/royger/freebsd/sys/kern/vfs_lookup.c:517
Process 18 (softdepflush) thread 0xfffffe000d14c920 (100056)
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe00170475f0) locked @ /usr/home/royger/freebsd/sys/kern/vfs_subr.c:2168
Process 17 (syncer) thread 0xfffffe000d14d000 (100055)
exclusive sleep mutex vnode_free_list (vnode_free_list) r = 0 (0xffffffff81530918) locked @ /usr/home/royger/freebsd/sys/kern/vfs_subr.c:4753
exclusive lockmgr syncer (syncer) r = 0 (0xfffffe000d6dc418) locked @ /usr/home/royger/freebsd/sys/kern/vfs_subr.c:1787

db> alltrace

Tracing command sendmail pid 32343 tid 101356 td 0xfffffe004157a000
sched_switch() at sched_switch+0x482/frame 0xffffff81211c44d0
mi_switch() at mi_switch+0x179/frame 0xffffff81211c4510
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff81211c4560
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff81211c4600
cache_lookup() at cache_lookup+0x78/frame 0xffffff81211c4680
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff81211c46d0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff81211c4700
lookup() at lookup+0x590/frame 0xffffff81211c4790
namei() at namei+0x464/frame 0xffffff81211c4840
vn_open_cred() at vn_open_cred+0x26d/frame 0xffffff81211c4990
kern_openat() at kern_openat+0x1fc/frame 0xffffff81211c4ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff81211c4bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff81211c4bf0
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x801449bea, rsp = 0x7fffffffc688, rbp = 0x7fffffffc6a0 ---

Tracing command sh pid 32342 tid 100171 td 0xfffffe000d7ad490
sched_switch() at sched_switch+0x482/frame 0xffffff8120e9a5b0
mi_switch() at mi_switch+0x179/frame 0xffffff8120e9a5f0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120e9a640
__rw_wlock_hard() at __rw_wlock_hard+0x25d/frame 0xffffff8120e9a6d0
_rw_wlock_cookie() at _rw_wlock_cookie+0xb4/frame 0xffffff8120e9a710
cache_lookup() at cache_lookup+0xc0/frame 0xffffff8120e9a790
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120e9a7e0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120e9a810
lookup() at lookup+0x590/frame 0xffffff8120e9a8a0
namei() at namei+0x464/frame 0xffffff8120e9a950
kern_readlinkat() at kern_readlinkat+0xb7/frame 0xffffff8120e9aac0
sys_readlink() at sys_readlink+0x26/frame 0xffffff8120e9aae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e9abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e9abf0
--- syscall (58, FreeBSD ELF64, sys_readlink), rip = 0x800d59fba, rsp = 0x7fffffffbe88, rbp = 0x7fffffffc320 ---

Tracing command cc pid 32341 tid 100211 td 0xfffffe00416d9000
sched_switch() at sched_switch+0x482/frame 0xffffff8120f628f0
mi_switch() at mi_switch+0x179/frame 0xffffff8120f62930
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120f62980
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120f62a20
vn_fullpath1() at vn_fullpath1+0x84/frame 0xffffff8120f62a80
kern___getcwd() at kern___getcwd+0xd6/frame 0xffffff8120f62ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120f62bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120f62bf0
--- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x156d22a, rsp = 0x7fffffffa938, rbp = 0x7fffffffaa20 ---

Tracing command ctfconvert pid 32340 tid 100180 td 0xfffffe0087f21000
sched_switch() at sched_switch+0x482/frame 0xffffff8120ec7470
mi_switch() at mi_switch+0x179/frame 0xffffff8120ec74b0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120ec74f0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff8120ec7520
sleeplk() at sleeplk+0x11a/frame 0xffffff8120ec7570
__lockmgr_args() at __lockmgr_args+0x92c/frame 0xffffff8120ec76a0
ffs_lock() at ffs_lock+0x84/frame 0xffffff8120ec76f0
VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf9/frame 0xffffff8120ec7720
_vn_lock() at _vn_lock+0xab/frame 0xffffff8120ec7790
lookup() at lookup+0x496/frame 0xffffff8120ec7820
namei() at namei+0x464/frame 0xffffff8120ec78d0
kern_renameat() at kern_renameat+0xa5/frame 0xffffff8120ec7ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120ec7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120ec7bf0
--- syscall (128, FreeBSD ELF64, sys_rename), rip = 0x80130109a, rsp = 0x7fffffffd268, rbp = 0x7fffffffd2c0 ---

Tracing command ctfconvert pid 32339 tid 100121 td 0xfffffe000d652490
sched_switch() at sched_switch+0x482/frame 0xffffff8120da04d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120da0510
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120da0560
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120da0600
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120da0680
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120da06d0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120da0700
lookup() at lookup+0x590/frame 0xffffff8120da0790
namei() at namei+0x464/frame 0xffffff8120da0840
vn_open_cred() at vn_open_cred+0x26d/frame 0xffffff8120da0990
kern_openat() at kern_openat+0x1fc/frame 0xffffff8120da0ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120da0bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120da0bf0
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x8013b7bea, rsp = 0x7fffffffcea8, rbp = 0x7fffffffcfa0 ---

Tracing command cc pid 32337 tid 101363 td 0xfffffe0041e75490
sched_switch() at sched_switch+0x482/frame 0xffffff81211e75b0
mi_switch() at mi_switch+0x179/frame 0xffffff81211e75f0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff81211e7640
__mtx_lock_sleep() at __mtx_lock_sleep+0x255/frame 0xffffff81211e76c0
__mtx_lock_flags() at __mtx_lock_flags+0xda/frame 0xffffff81211e7700
vdropl() at vdropl+0x255/frame 0xffffff81211e7730
vputx() at vputx+0x27c/frame 0xffffff81211e7790
namei() at namei+0x3dd/frame 0xffffff81211e7840
vn_open_cred() at vn_open_cred+0x26d/frame 0xffffff81211e7990
kern_openat() at kern_openat+0x1fc/frame 0xffffff81211e7ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff81211e7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff81211e7bf0
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x159216a, rsp = 0x7fffffff8198, rbp = 0x7fffffff81c0 ---

Tracing command cc pid 32335 tid 101391 td 0xfffffe00492c3000
sched_switch() at sched_switch+0x482/frame 0xffffff8121273690
mi_switch() at mi_switch+0x179/frame 0xffffff81212736d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121273710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121273780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81212737b0
_sleep() at _sleep+0x349/frame 0xffffff8121273850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81212738f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121273ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121273bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121273bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbca8, rbp = 0x7fffffffbd70 ---

Tracing command sh pid 32334 tid 101291 td 0xfffffe00178e7920
sched_switch() at sched_switch+0x482/frame 0xffffff8121250690
mi_switch() at mi_switch+0x179/frame 0xffffff81212506d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121250710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121250780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81212507b0
_sleep() at _sleep+0x349/frame 0xffffff8121250850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81212508f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121250ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121250bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121250bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32333 tid 100328 td 0xfffffe00173a8490
sched_switch() at sched_switch+0x482/frame 0xffffff812111f4b0
mi_switch() at mi_switch+0x179/frame 0xffffff812111f4f0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff812111f540
__mtx_lock_sleep() at __mtx_lock_sleep+0x255/frame 0xffffff812111f5c0
__mtx_lock_flags() at __mtx_lock_flags+0xda/frame 0xffffff812111f600
cache_lookup() at cache_lookup+0x561/frame 0xffffff812111f680
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff812111f6d0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff812111f700
lookup() at lookup+0x590/frame 0xffffff812111f790
namei() at namei+0x464/frame 0xffffff812111f840
vn_open_cred() at vn_open_cred+0x26d/frame 0xffffff812111f990
kern_openat() at kern_openat+0x1fc/frame 0xffffff812111fae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff812111fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff812111fbf0
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x159216a, rsp = 0x7fffffff8608, rbp = 0x7fffffff8630 ---

Tracing command cc pid 32332 tid 100244 td 0xfffffe00870fa000
sched_switch() at sched_switch+0x482/frame 0xffffff8121007690
mi_switch() at mi_switch+0x179/frame 0xffffff81210076d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121007710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121007780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81210077b0
_sleep() at _sleep+0x349/frame 0xffffff8121007850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81210078f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121007ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121007bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121007bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbca8, rbp = 0x7fffffffbd70 ---

Tracing command sh pid 32331 tid 101338 td 0xfffffe004132e920
sched_switch() at sched_switch+0x482/frame 0xffffff812116a690
mi_switch() at mi_switch+0x179/frame 0xffffff812116a6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff812116a710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff812116a780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff812116a7b0
_sleep() at _sleep+0x349/frame 0xffffff812116a850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff812116a8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff812116aae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff812116abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff812116abf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32329 tid 101380 td 0xfffffe004901d490
sched_switch() at sched_switch+0x482/frame 0xffffff812123c690
mi_switch() at mi_switch+0x179/frame 0xffffff812123c6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff812123c710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff812123c780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff812123c7b0
_sleep() at _sleep+0x349/frame 0xffffff812123c850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff812123c8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff812123cae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff812123cbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff812123cbf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbca8, rbp = 0x7fffffffbd70 ---

Tracing command sh pid 32328 tid 100231 td 0xfffffe004157b000
sched_switch() at sched_switch+0x482/frame 0xffffff8120fc6690
mi_switch() at mi_switch+0x179/frame 0xffffff8120fc66d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120fc6710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120fc6780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120fc67b0
_sleep() at _sleep+0x349/frame 0xffffff8120fc6850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120fc68f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120fc6ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120fc6bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120fc6bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32324 tid 100248 td 0xfffffe00414b3920
sched_switch() at sched_switch+0x482/frame 0xffffff812101b8f0
mi_switch() at mi_switch+0x179/frame 0xffffff812101b930
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff812101b980
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff812101ba20
vn_fullpath1() at vn_fullpath1+0x84/frame 0xffffff812101ba80
kern___getcwd() at kern___getcwd+0xd6/frame 0xffffff812101bae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff812101bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff812101bbf0
--- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x156d22a, rsp = 0x7fffffffa9d8, rbp = 0x7fffffffaac0 ---

Tracing command cc pid 32320 tid 100216 td 0xfffffe0041347920
sched_switch() at sched_switch+0x482/frame 0xffffff8120f7b690
mi_switch() at mi_switch+0x179/frame 0xffffff8120f7b6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120f7b710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120f7b780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120f7b7b0
_sleep() at _sleep+0x349/frame 0xffffff8120f7b850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120f7b8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120f7bae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120f7bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120f7bbf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbca8, rbp = 0x7fffffffbd70 ---

Tracing command sh pid 32318 tid 100157 td 0xfffffe0087a34490
sched_switch() at sched_switch+0x482/frame 0xffffff8120e54690
mi_switch() at mi_switch+0x179/frame 0xffffff8120e546d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120e54710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120e54780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120e547b0
_sleep() at _sleep+0x349/frame 0xffffff8120e54850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120e548f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120e54ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e54bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e54bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32314 tid 100215 td 0xfffffe00413fd000
sched_switch() at sched_switch+0x482/frame 0xffffff8120f76560
mi_switch() at mi_switch+0x179/frame 0xffffff8120f765a0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120f765f0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120f76690
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120f76710
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120f76760
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120f76790
lookup() at lookup+0x590/frame 0xffffff8120f76820
namei() at namei+0x464/frame 0xffffff8120f768d0
kern_renameat() at kern_renameat+0xa5/frame 0xffffff8120f76ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120f76bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120f76bf0
--- syscall (128, FreeBSD ELF64, sys_rename), rip = 0x156bbfa, rsp = 0x7fffffffb2b8, rbp = 0x7fffffffb420 ---

Tracing command cc pid 32313 tid 101340 td 0xfffffe00b9a42920
sched_switch() at sched_switch+0x482/frame 0xffffff8121174690
mi_switch() at mi_switch+0x179/frame 0xffffff81211746d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121174710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121174780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81211747b0
_sleep() at _sleep+0x349/frame 0xffffff8121174850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81211748f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121174ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121174bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121174bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbcb8, rbp = 0x7fffffffbd80 ---

Tracing command cc pid 32312 tid 100149 td 0xfffffe008710a920
sched_switch() at sched_switch+0x482/frame 0xffffff8120e2c560
mi_switch() at mi_switch+0x179/frame 0xffffff8120e2c5a0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120e2c5f0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120e2c690
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120e2c710
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120e2c760
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120e2c790
lookup() at lookup+0x590/frame 0xffffff8120e2c820
namei() at namei+0x464/frame 0xffffff8120e2c8d0
kern_renameat() at kern_renameat+0xa5/frame 0xffffff8120e2cae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e2cbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e2cbf0
--- syscall (128, FreeBSD ELF64, sys_rename), rip = 0x156bbfa, rsp = 0x7fffffffb2b8, rbp = 0x7fffffffb420 ---

Tracing command sh pid 32311 tid 100260 td 0xfffffe0041683920
sched_switch() at sched_switch+0x482/frame 0xffffff8121057690
mi_switch() at mi_switch+0x179/frame 0xffffff81210576d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121057710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121057780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81210577b0
_sleep() at _sleep+0x349/frame 0xffffff8121057850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81210578f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121057ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121057bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121057bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32310 tid 100118 td 0xfffffe000d68e490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d91560
mi_switch() at mi_switch+0x179/frame 0xffffff8120d915a0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120d915f0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120d91690
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120d91710
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120d91760
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120d91790
lookup() at lookup+0x590/frame 0xffffff8120d91820
namei() at namei+0x464/frame 0xffffff8120d918d0
kern_renameat() at kern_renameat+0xa5/frame 0xffffff8120d91ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d91bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d91bf0
--- syscall (128, FreeBSD ELF64, sys_rename), rip = 0x156bbfa, rsp = 0x7fffffffb2a8, rbp = 0x7fffffffb410 ---

Tracing command cc pid 32309 tid 101376 td 0xfffffe0041577920
sched_switch() at sched_switch+0x482/frame 0xffffff8121228690
mi_switch() at mi_switch+0x179/frame 0xffffff81212286d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121228710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121228780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81212287b0
_sleep() at _sleep+0x349/frame 0xffffff8121228850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81212288f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121228ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121228bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121228bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbcb8, rbp = 0x7fffffffbd80 ---

Tracing command sh pid 32307 tid 101392 td 0xfffffe0017308000
sched_switch() at sched_switch+0x482/frame 0xffffff8121278690
mi_switch() at mi_switch+0x179/frame 0xffffff81212786d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121278710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121278780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81212787b0
_sleep() at _sleep+0x349/frame 0xffffff8121278850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81212788f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121278ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121278bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121278bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32305 tid 100217 td 0xfffffe004157d000
sched_switch() at sched_switch+0x482/frame 0xffffff8120f80690
mi_switch() at mi_switch+0x179/frame 0xffffff8120f806d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120f80710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120f80780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120f807b0
_sleep() at _sleep+0x349/frame 0xffffff8120f80850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120f808f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120f80ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120f80bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120f80bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbcb8, rbp = 0x7fffffffbd80 ---

Tracing command sh pid 32304 tid 100330 td 0xfffffe0017711920
sched_switch() at sched_switch+0x482/frame 0xffffff8121129690
mi_switch() at mi_switch+0x179/frame 0xffffff81211296d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121129710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121129780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81211297b0
_sleep() at _sleep+0x349/frame 0xffffff8121129850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81211298f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121129ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121129bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121129bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command cc pid 32303 tid 100099 td 0xfffffe000d614490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d32560
mi_switch() at mi_switch+0x179/frame 0xffffff8120d325a0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120d325f0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120d32690
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120d32710
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120d32760
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120d32790
lookup() at lookup+0x590/frame 0xffffff8120d32820
namei() at namei+0x464/frame 0xffffff8120d328d0
kern_renameat() at kern_renameat+0xa5/frame 0xffffff8120d32ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d32bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d32bf0
--- syscall (128, FreeBSD ELF64, sys_rename), rip = 0x156bbfa, rsp = 0x7fffffffb2b8, rbp = 0x7fffffffb420 ---

Tracing command cc pid 32298 tid 100431 td 0xfffffe00492e2000
sched_switch() at sched_switch+0x482/frame 0xffffff81211f1690
mi_switch() at mi_switch+0x179/frame 0xffffff81211f16d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff81211f1710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff81211f1780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81211f17b0
_sleep() at _sleep+0x349/frame 0xffffff81211f1850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81211f18f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff81211f1ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff81211f1bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff81211f1bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x156d28a, rsp = 0x7fffffffbcb8, rbp = 0x7fffffffbd80 ---

Tracing command sh pid 32295 tid 100234 td 0xfffffe000d65a490
sched_switch() at sched_switch+0x482/frame 0xffffff8120fd5690
mi_switch() at mi_switch+0x179/frame 0xffffff8120fd56d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120fd5710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120fd5780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120fd57b0
_sleep() at _sleep+0x349/frame 0xffffff8120fd5850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120fd58f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120fd5ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120fd5bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120fd5bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command sh pid 32293 tid 100058 td 0xfffffe000d14c490
sched_switch() at sched_switch+0x482/frame 0xffffff8120c65690
mi_switch() at mi_switch+0x179/frame 0xffffff8120c656d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c65710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120c65780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120c657b0
_sleep() at _sleep+0x349/frame 0xffffff8120c65850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120c658f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120c65ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c65bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c65bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command sh pid 32284 tid 100226 td 0xfffffe004157c490
sched_switch() at sched_switch+0x482/frame 0xffffff8120fad690
mi_switch() at mi_switch+0x179/frame 0xffffff8120fad6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120fad710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120fad780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120fad7b0
_sleep() at _sleep+0x349/frame 0xffffff8120fad850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120fad8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120fadae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120fadbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120fadbf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command make pid 32256 tid 100154 td 0xfffffe000d50a920
sched_switch() at sched_switch+0x482/frame 0xffffff8120e456c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120e45700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120e45740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120e457b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120e457e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120e45860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120e458a0
kern_select() at kern_select+0x892/frame 0xffffff8120e45aa0
sys_select() at sys_select+0x54/frame 0xffffff8120e45ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e45bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e45bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffbf88, rbp = 0x7fffffffc030 ---

Tracing command sh pid 32022 tid 100259 td 0xfffffe0041562000
sched_switch() at sched_switch+0x482/frame 0xffffff8121052550
mi_switch() at mi_switch+0x179/frame 0xffffff8121052590
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff81210525e0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8121052680
cache_lookup() at cache_lookup+0x78/frame 0xffffff8121052700
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8121052750
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8121052780
lookup() at lookup+0x590/frame 0xffffff8121052810
namei() at namei+0x464/frame 0xffffff81210528c0
kern_statat_vnhook() at kern_statat_vnhook+0x99/frame 0xffffff8121052a40
sys_stat() at sys_stat+0x2d/frame 0xffffff8121052ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121052bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121052bf0
--- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800dc7f0a, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd020 ---

Tracing command sh pid 31391 tid 100228 td 0xfffffe00416d4490
sched_switch() at sched_switch+0x482/frame 0xffffff8120fb7690
mi_switch() at mi_switch+0x179/frame 0xffffff8120fb76d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120fb7710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120fb7780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120fb77b0
_sleep() at _sleep+0x349/frame 0xffffff8120fb7850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120fb78f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120fb7ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120fb7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120fb7bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcd68, rbp = 0x7fffffffce10 ---

Tracing command make pid 31386 tid 100336 td 0xfffffe0017941490
sched_switch() at sched_switch+0x482/frame 0xffffff81211476c0
mi_switch() at mi_switch+0x179/frame 0xffffff8121147700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121147740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff81211477b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff81211477e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8121147860
seltdwait() at seltdwait+0x8b/frame 0xffffff81211478a0
kern_select() at kern_select+0x892/frame 0xffffff8121147aa0
sys_select() at sys_select+0x54/frame 0xffffff8121147ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121147bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121147bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffbfb8, rbp = 0x7fffffffc060 ---

Tracing command sh pid 30664 tid 100255 td 0xfffffe0041622000
sched_switch() at sched_switch+0x482/frame 0xffffff812103e690
mi_switch() at mi_switch+0x179/frame 0xffffff812103e6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff812103e710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff812103e780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff812103e7b0
_sleep() at _sleep+0x349/frame 0xffffff812103e850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff812103e8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff812103eae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff812103ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff812103ebf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcd98, rbp = 0x7fffffffce40 ---

Tracing command make pid 30663 tid 100153 td 0xfffffe000d50e000
sched_switch() at sched_switch+0x482/frame 0xffffff8120e406c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120e40700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120e40740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120e407b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120e407e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120e40860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120e408a0
kern_select() at kern_select+0x892/frame 0xffffff8120e40aa0
sys_select() at sys_select+0x54/frame 0xffffff8120e40ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e40bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e40bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffbfe8, rbp = 0x7fffffffc090 ---

Tracing command sh pid 30662 tid 101302 td 0xfffffe0017184000
sched_switch() at sched_switch+0x482/frame 0xffffff81211fb690
mi_switch() at mi_switch+0x179/frame 0xffffff81211fb6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff81211fb710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff81211fb780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81211fb7b0
_sleep() at _sleep+0x349/frame 0xffffff81211fb850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81211fb8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff81211fbae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff81211fbbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff81211fbbf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffcf48, rbp = 0x7fffffffcff0 ---

Tracing command make pid 30655 tid 100236 td 0xfffffe00414c8490
sched_switch() at sched_switch+0x482/frame 0xffffff8120fdf6c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120fdf700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120fdf740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120fdf7b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120fdf7e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120fdf860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120fdf8a0
kern_select() at kern_select+0x892/frame 0xffffff8120fdfaa0
sys_select() at sys_select+0x54/frame 0xffffff8120fdfae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120fdfbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120fdfbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffc0b8, rbp = 0x7fffffffc160 ---

Tracing command sh pid 5287 tid 100249 td 0xfffffe004160a000
sched_switch() at sched_switch+0x482/frame 0xffffff8121020690
mi_switch() at mi_switch+0x179/frame 0xffffff81210206d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8121020710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8121020780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff81210207b0
_sleep() at _sleep+0x349/frame 0xffffff8121020850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff81210208f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8121020ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121020bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121020bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffd378, rbp = 0x7fffffffd420 ---

Tracing command make pid 5280 tid 100085 td 0xfffffe000d79f490
sched_switch() at sched_switch+0x482/frame 0xffffff8120cec6c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120cec700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120cec740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120cec7b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120cec7e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120cec860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120cec8a0
kern_select() at kern_select+0x892/frame 0xffffff8120cecaa0
sys_select() at sys_select+0x54/frame 0xffffff8120cecae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120cecbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120cecbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffc4f8, rbp = 0x7fffffffc5a0 ---

Tracing command sh pid 5278 tid 100138 td 0xfffffe000d608000
sched_switch() at sched_switch+0x482/frame 0xffffff8120df5690
mi_switch() at mi_switch+0x179/frame 0xffffff8120df56d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120df5710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120df5780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120df57b0
_sleep() at _sleep+0x349/frame 0xffffff8120df5850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120df58f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120df5ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120df5bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120df5bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffd3d8, rbp = 0x7fffffffd480 ---

Tracing command make pid 5273 tid 100142 td 0xfffffe00870d7000
sched_switch() at sched_switch+0x482/frame 0xffffff8120e096c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120e09700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120e09740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120e097b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120e097e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120e09860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120e098a0
kern_select() at kern_select+0x892/frame 0xffffff8120e09aa0
sys_select() at sys_select+0x54/frame 0xffffff8120e09ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120e09bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120e09bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44dc9a, rsp = 0x7fffffffc5b8, rbp = 0x7fffffffc660 ---

Tracing command bash pid 91658 tid 100191 td 0xfffffe00871db000
sched_switch() at sched_switch+0x482/frame 0xffffff8120efe760
mi_switch() at mi_switch+0x179/frame 0xffffff8120efe7a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120efe7e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120efe850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120efe880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120efe8f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120efe920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120efe980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120efe9b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120efea10
dofileread() at dofileread+0x7b/frame 0xffffff8120efea50
kern_readv() at kern_readv+0x48/frame 0xffffff8120efea90
sys_read() at sys_read+0x63/frame 0xffffff8120efeae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120efebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120efebf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80112a2fa, rsp = 0x7fffffffc458, rbp = 0x7fffffffc480 ---

Tracing command bash pid 736 tid 100071 td 0xfffffe000d50f490
sched_switch() at sched_switch+0x482/frame 0xffffff8120ca6690
mi_switch() at mi_switch+0x179/frame 0xffffff8120ca66d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120ca6710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120ca6780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120ca67b0
_sleep() at _sleep+0x349/frame 0xffffff8120ca6850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120ca68f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120ca6ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120ca6bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120ca6bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010846da, rsp = 0x7fffffffcc48, rbp = 0x7fffffffcc90 ---

Tracing command screen pid 735 tid 100104 td 0xfffffe000d64c000
sched_switch() at sched_switch+0x482/frame 0xffffff8120d4b6d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120d4b710
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d4b750
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d4b7c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d4b7f0
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d4b860
seltdwait() at seltdwait+0x78/frame 0xffffff8120d4b8a0
kern_select() at kern_select+0x892/frame 0xffffff8120d4baa0
sys_select() at sys_select+0x54/frame 0xffffff8120d4bae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d4bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d4bbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80142327a, rsp = 0x7fffffffd778, rbp = 0x7fffffffd8d0 ---

Tracing command screen pid 734 tid 100091 td 0xfffffe000d64c490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d0a890
mi_switch() at mi_switch+0x179/frame 0xffffff8120d0a8d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d0a910
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d0a980
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d0a9b0
_sleep() at _sleep+0x349/frame 0xffffff8120d0aa50
kern_sigsuspend() at kern_sigsuspend+0xcd/frame 0xffffff8120d0aab0
sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xffffff8120d0aae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d0abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d0abf0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80136431a, rsp = 0x7fffffffd8a8, rbp = 0x7fffffffd8c0 ---

Tracing command bash pid 730 tid 100095 td 0xfffffe000d67f920
sched_switch() at sched_switch+0x482/frame 0xffffff8120d1e690
mi_switch() at mi_switch+0x179/frame 0xffffff8120d1e6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d1e710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d1e780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d1e7b0
_sleep() at _sleep+0x349/frame 0xffffff8120d1e850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120d1e8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120d1eae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d1ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d1ebf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010846da, rsp = 0x7fffffffd578, rbp = 0x7fffffffd5c0 ---

Tracing command su pid 729 tid 100112 td 0xfffffe000d6fb920
sched_switch() at sched_switch+0x482/frame 0xffffff8120d73690
mi_switch() at mi_switch+0x179/frame 0xffffff8120d736d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d73710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d73780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d737b0
_sleep() at _sleep+0x349/frame 0xffffff8120d73850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120d738f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120d73ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d73bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d73bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ed46da, rsp = 0x7fffffffd618, rbp = 0x7fffffffdb80 ---

Tracing command sh pid 728 tid 100139 td 0xfffffe000d603920
sched_switch() at sched_switch+0x482/frame 0xffffff8120dfa690
mi_switch() at mi_switch+0x179/frame 0xffffff8120dfa6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120dfa710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120dfa780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120dfa7b0
_sleep() at _sleep+0x349/frame 0xffffff8120dfa850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120dfa8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120dfaae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120dfabf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120dfabf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d326da, rsp = 0x7fffffffd838, rbp = 0x7fffffffd8e0 ---

Tracing command sshd pid 727 tid 100108 td 0xfffffe000d655490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d5f6d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120d5f710
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d5f750
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d5f7c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d5f7f0
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d5f860
seltdwait() at seltdwait+0x78/frame 0xffffff8120d5f8a0
kern_select() at kern_select+0x892/frame 0xffffff8120d5faa0
sys_select() at sys_select+0x54/frame 0xffffff8120d5fae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d5fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d5fbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80257327a, rsp = 0x7fffffffd168, rbp = 0x7fffffffd1d0 ---

Tracing command sshd pid 724 tid 100107 td 0xfffffe000d67f490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d5a770
mi_switch() at mi_switch+0x179/frame 0xffffff8120d5a7b0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d5a7f0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d5a860
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d5a890
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d5a900
seltdwait() at seltdwait+0x78/frame 0xffffff8120d5a940
sys_poll() at sys_poll+0x38c/frame 0xffffff8120d5aae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d5abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d5abf0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80250bd1a, rsp = 0x7fffffffd1f8, rbp = 0x7fffffffd250 ---

Tracing command bash pid 722 tid 100090 td 0xfffffe000d664000
sched_switch() at sched_switch+0x482/frame 0xffffff8120d05550
mi_switch() at mi_switch+0x179/frame 0xffffff8120d05590
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120d055e0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120d05680
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120d05700
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120d05750
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120d05780
lookup() at lookup+0x590/frame 0xffffff8120d05810
namei() at namei+0x464/frame 0xffffff8120d058c0
kern_statat_vnhook() at kern_statat_vnhook+0x99/frame 0xffffff8120d05a40
sys_stat() at sys_stat+0x2d/frame 0xffffff8120d05ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d05bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d05bf0
--- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x801119f0a, rsp = 0x7fffffffb898, rbp = 0x7fffffffc9e0 ---

Tracing command login pid 721 tid 100123 td 0xfffffe000d6fb000
sched_switch() at sched_switch+0x482/frame 0xffffff8120daa690
mi_switch() at mi_switch+0x179/frame 0xffffff8120daa6d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120daa710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120daa780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120daa7b0
_sleep() at _sleep+0x349/frame 0xffffff8120daa850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff8120daa8f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8120daaae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120daabf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120daabf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ed66da, rsp = 0x7fffffffdc08, rbp = 0x7fffffffdd40 ---

Tracing command getty pid 720 tid 100113 td 0xfffffe000d60a000
sched_switch() at sched_switch+0x482/frame 0xffffff8120d78760
mi_switch() at mi_switch+0x179/frame 0xffffff8120d787a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d787e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d78850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d78880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d788f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120d78920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120d78980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120d789b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120d78a10
dofileread() at dofileread+0x7b/frame 0xffffff8120d78a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120d78a90
sys_read() at sys_read+0x63/frame 0xffffff8120d78ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d78bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d78bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 719 tid 100057 td 0xfffffe000d510920
sched_switch() at sched_switch+0x482/frame 0xffffff8120c60760
mi_switch() at mi_switch+0x179/frame 0xffffff8120c607a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c607e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120c60850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120c60880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120c608f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120c60920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120c60980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120c609b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120c60a10
dofileread() at dofileread+0x7b/frame 0xffffff8120c60a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120c60a90
sys_read() at sys_read+0x63/frame 0xffffff8120c60ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c60bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c60bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 718 tid 100128 td 0xfffffe000d608490
sched_switch() at sched_switch+0x482/frame 0xffffff8120dc3760
mi_switch() at mi_switch+0x179/frame 0xffffff8120dc37a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120dc37e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120dc3850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120dc3880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120dc38f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120dc3920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120dc3980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120dc39b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120dc3a10
dofileread() at dofileread+0x7b/frame 0xffffff8120dc3a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120dc3a90
sys_read() at sys_read+0x63/frame 0xffffff8120dc3ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120dc3bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120dc3bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 717 tid 100115 td 0xfffffe000d609490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d82760
mi_switch() at mi_switch+0x179/frame 0xffffff8120d827a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d827e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d82850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d82880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d828f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120d82920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120d82980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120d829b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120d82a10
dofileread() at dofileread+0x7b/frame 0xffffff8120d82a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120d82a90
sys_read() at sys_read+0x63/frame 0xffffff8120d82ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d82bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d82bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 716 tid 100127 td 0xfffffe000d608920
sched_switch() at sched_switch+0x482/frame 0xffffff8120dbe760
mi_switch() at mi_switch+0x179/frame 0xffffff8120dbe7a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120dbe7e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120dbe850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120dbe880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120dbe8f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120dbe920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120dbe980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120dbe9b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120dbea10
dofileread() at dofileread+0x7b/frame 0xffffff8120dbea50
kern_readv() at kern_readv+0x48/frame 0xffffff8120dbea90
sys_read() at sys_read+0x63/frame 0xffffff8120dbeae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120dbebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120dbebf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 715 tid 100116 td 0xfffffe000d609000
sched_switch() at sched_switch+0x482/frame 0xffffff8120d87760
mi_switch() at mi_switch+0x179/frame 0xffffff8120d877a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d877e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d87850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d87880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d878f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120d87920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120d87980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120d879b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120d87a10
dofileread() at dofileread+0x7b/frame 0xffffff8120d87a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120d87a90
sys_read() at sys_read+0x63/frame 0xffffff8120d87ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d87bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d87bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 714 tid 100062 td 0xfffffe000d60b000
sched_switch() at sched_switch+0x482/frame 0xffffff8120c79760
mi_switch() at mi_switch+0x179/frame 0xffffff8120c797a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c797e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120c79850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120c79880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120c798f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120c79920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120c79980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120c799b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120c79a10
dofileread() at dofileread+0x7b/frame 0xffffff8120c79a50
kern_readv() at kern_readv+0x48/frame 0xffffff8120c79a90
sys_read() at sys_read+0x63/frame 0xffffff8120c79ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c79bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c79bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command getty pid 713 tid 100114 td 0xfffffe000d609920
sched_switch() at sched_switch+0x482/frame 0xffffff8120d7d760
mi_switch() at mi_switch+0x179/frame 0xffffff8120d7d7a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d7d7e0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d7d850
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d7d880
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d7d8f0
tty_wait() at tty_wait+0x42/frame 0xffffff8120d7d920
ttydisc_read() at ttydisc_read+0x353/frame 0xffffff8120d7d980
ttydev_read() at ttydev_read+0x6d/frame 0xffffff8120d7d9b0
devfs_read_f() at devfs_read_f+0x81/frame 0xffffff8120d7da10
dofileread() at dofileread+0x7b/frame 0xffffff8120d7da50
kern_readv() at kern_readv+0x48/frame 0xffffff8120d7da90
sys_read() at sys_read+0x63/frame 0xffffff8120d7dae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d7dbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d7dbf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b572fa, rsp = 0x7fffffffdc98, rbp = 0x7fffffffdcd0 ---

Tracing command cron pid 674 tid 100137 td 0xfffffe000d6de920
sched_switch() at sched_switch+0x482/frame 0xffffff8120df0550
mi_switch() at mi_switch+0x179/frame 0xffffff8120df0590
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120df05e0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120df0680
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120df0700
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120df0750
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120df0780
lookup() at lookup+0x590/frame 0xffffff8120df0810
namei() at namei+0x464/frame 0xffffff8120df08c0
kern_statat_vnhook() at kern_statat_vnhook+0x99/frame 0xffffff8120df0a40
sys_stat() at sys_stat+0x2d/frame 0xffffff8120df0ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120df0bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120df0bf0
--- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800d55f0a, rsp = 0x7fffffffd7d8, rbp = 0x7fffffffdb20 ---

Tracing command sendmail pid 670 tid 100066 td 0xfffffe000d510000
sched_switch() at sched_switch+0x482/frame 0xffffff8120c8d890
mi_switch() at mi_switch+0x179/frame 0xffffff8120c8d8d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c8d910
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120c8d980
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120c8d9b0
_sleep() at _sleep+0x349/frame 0xffffff8120c8da50
kern_sigsuspend() at kern_sigsuspend+0xcd/frame 0xffffff8120c8dab0
sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xffffff8120c8dae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c8dbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c8dbf0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80139231a, rsp = 0x7fffffffc818, rbp = 0x7fffffffc830 ---

Tracing command sendmail pid 667 tid 100103 td 0xfffffe000d60a490
sched_switch() at sched_switch+0x482/frame 0xffffff8120d46280
mi_switch() at mi_switch+0x179/frame 0xffffff8120d462c0
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120d46310
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120d463b0
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120d46430
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120d46480
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120d464b0
lookup() at lookup+0x590/frame 0xffffff8120d46540
namei() at namei+0x464/frame 0xffffff8120d465f0
kern_statfs() at kern_statfs+0x96/frame 0xffffff8120d468e0
sys_statfs() at sys_statfs+0x2c/frame 0xffffff8120d46ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d46bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d46bf0
--- syscall (396, FreeBSD ELF64, sys_statfs), rip = 0x801391eda, rsp = 0x7fffffffba98, rbp = 0x7fffffffbc90 ---

Tracing command sshd pid 664 tid 100133 td 0xfffffe000d72e490
sched_switch() at sched_switch+0x482/frame 0xffffff8120ddc6d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120ddc710
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120ddc750
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120ddc7c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120ddc7f0
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120ddc860
seltdwait() at seltdwait+0x78/frame 0xffffff8120ddc8a0
kern_select() at kern_select+0x892/frame 0xffffff8120ddcaa0
sys_select() at sys_select+0x54/frame 0xffffff8120ddcae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120ddcbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120ddcbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80257327a, rsp = 0x7fffffffd288, rbp = 0x7fffffffdd10 ---

Tracing command syslogd pid 566 tid 100092 td 0xfffffe000d79e920
sched_switch() at sched_switch+0x482/frame 0xffffff8120d0f6d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120d0f710
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d0f750
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d0f7c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d0f7f0
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d0f860
seltdwait() at seltdwait+0x78/frame 0xffffff8120d0f8a0
kern_select() at kern_select+0x892/frame 0xffffff8120d0faa0
sys_select() at sys_select+0x54/frame 0xffffff8120d0fae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d0fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d0fbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b5a27a, rsp = 0x7fffffffcce8, rbp = 0x7fffffffdd00 ---

Tracing command devd pid 472 tid 100064 td 0xfffffe000d616490
sched_switch() at sched_switch+0x482/frame 0xffffff8120c836c0
mi_switch() at mi_switch+0x179/frame 0xffffff8120c83700
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c83740
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120c837b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8120c837e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1fe/frame 0xffffff8120c83860
seltdwait() at seltdwait+0x8b/frame 0xffffff8120c838a0
kern_select() at kern_select+0x892/frame 0xffffff8120c83aa0
sys_select() at sys_select+0x54/frame 0xffffff8120c83ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c83bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c83bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x44c5ba, rsp = 0x7fffffffbba8, rbp = 0x7fffffffdd20 ---

Tracing command dhclient pid 471 tid 100060 td 0xfffffe000d02f920
sched_switch() at sched_switch+0x482/frame 0xffffff8120c6f430
mi_switch() at mi_switch+0x179/frame 0xffffff8120c6f470
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120c6f4c0
__rw_rlock() at __rw_rlock+0xc6/frame 0xffffff8120c6f560
cache_lookup() at cache_lookup+0x78/frame 0xffffff8120c6f5e0
vfs_cache_lookup() at vfs_cache_lookup+0xa1/frame 0xffffff8120c6f630
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xea/frame 0xffffff8120c6f660
lookup() at lookup+0x590/frame 0xffffff8120c6f6f0
namei() at namei+0x464/frame 0xffffff8120c6f7a0
unp_connectat() at unp_connectat+0x1ba/frame 0xffffff8120c6fa10
uipc_connect() at uipc_connect+0x4c/frame 0xffffff8120c6fa40
kern_connectat() at kern_connectat+0x10f/frame 0xffffff8120c6faa0
sys_connect() at sys_connect+0x9b/frame 0xffffff8120c6fae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120c6fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120c6fbf0
--- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x800b4648a, rsp = 0x7fffffffcc88, rbp = 0x7fffffffcd10 ---

Tracing command dhclient pid 431 tid 100101 td 0xfffffe000d63d920
sched_switch() at sched_switch+0x482/frame 0xffffff8120d3c770
mi_switch() at mi_switch+0x179/frame 0xffffff8120d3c7b0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120d3c7f0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8120d3c860
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff8120d3c890
_cv_wait_sig() at _cv_wait_sig+0x1e2/frame 0xffffff8120d3c900
seltdwait() at seltdwait+0x78/frame 0xffffff8120d3c940
sys_poll() at sys_poll+0x38c/frame 0xffffff8120d3cae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120d3cbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120d3cbf0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800afdd1a, rsp = 0x7fffffffdc88, rbp = 0x7fffffffdcb0 ---

Tracing command softdepflush pid 18 tid 100056 td 0xfffffe000d14c920
sched_switch() at sched_switch+0x482/frame 0xffffff8120c5b6d0
mi_switch() at mi_switch+0x179/frame 0xffffff8120c5b710
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120c5b760
__rw_wlock_hard() at __rw_wlock_hard+0x25d/frame 0xffffff8120c5b7f0
_rw_wlock_cookie() at _rw_wlock_cookie+0xb4/frame 0xffffff8120c5b830
cache_purge() at cache_purge+0x42/frame 0xffffff8120c5b850
vgonel() at vgonel+0x5d9/frame 0xffffff8120c5b8f0
vrecycle() at vrecycle+0x3e/frame 0xffffff8120c5b920
ufs_inactive() at ufs_inactive+0x31d/frame 0xffffff8120c5b960
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf0/frame 0xffffff8120c5b990
vinactive() at vinactive+0xc6/frame 0xffffff8120c5b9e0
vputx() at vputx+0x24e/frame 0xffffff8120c5ba40
handle_workitem_remove() at handle_workitem_remove+0x326/frame 0xffffff8120c5bab0
process_worklist_item() at process_worklist_item+0x28a/frame 0xffffff8120c5bb30
softdep_process_worklist() at softdep_process_worklist+0x79/frame 0xffffff8120c5bb70
softdep_flush() at softdep_flush+0x10a/frame 0xffffff8120c5bbb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c5bbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c5bbf0
--- trap 0, rip = 0, rsp = 0xffffff8120c5bcb0, rbp = 0 ---

Tracing command syncer pid 17 tid 100055 td 0xfffffe000d14d000

cpustop_handler() at cpustop_handler+0x28/frame 0xffffff8000232cf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff8000232d10
trap() at trap+0x42/frame 0xffffff8000232f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff8000232f20
--- trap 0x13, rip = 0xffffffff808b4bd6, rsp = 0xffffff8000232fe0, rbp = 0xffffff8120c569a0 ---
_mtx_trylock_flags_() at _mtx_trylock_flags_+0x106/frame 0xffffff8120c569a0
mnt_vnode_next_active() at mnt_vnode_next_active+0x141/frame 0xffffff8120c569e0
ffs_sync() at ffs_sync+0xb5/frame 0xffffff8120c56ab0
sync_fsync() at sync_fsync+0x12d/frame 0xffffff8120c56ae0
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf0/frame 0xffffff8120c56b10
sched_sync() at sched_sync+0x34c/frame 0xffffff8120c56bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c56bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c56bf0
--- trap 0, rip = 0, rsp = 0xffffff8120c56cb0, rbp = 0 ---

Tracing command vnlru pid 9 tid 100054 td 0xfffffe000d14d490
sched_switch() at sched_switch+0x482/frame 0xffffff8120c519e0
mi_switch() at mi_switch+0x179/frame 0xffffff8120c51a20
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c51a60
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff8120c51a90
_sleep() at _sleep+0x336/frame 0xffffff8120c51b30
vnlru_proc() at vnlru_proc+0x48/frame 0xffffff8120c51bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c51bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c51bf0
--- trap 0, rip = 0, rsp = 0xffffff8120c51cb0, rbp = 0 ---

Tracing command bufdaemon pid 8 tid 100053 td 0xfffffe000d14d920
sched_switch() at sched_switch+0x482/frame 0xffffff8120c4ca40
mi_switch() at mi_switch+0x179/frame 0xffffff8120c4ca80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c4cac0
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff8120c4caf0
_sleep() at _sleep+0x336/frame 0xffffff8120c4cb90
buf_daemon() at buf_daemon+0x68/frame 0xffffff8120c4cbb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c4cbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c4cbf0
--- trap 0, rip = 0, rsp = 0xffffff8120c4ccb0, rbp = 0 ---

Tracing command pagezero pid 7 tid 100052 td 0xfffffe000d150000
sched_switch() at sched_switch+0x482/frame 0xffffff8120c47a30
mi_switch() at mi_switch+0x179/frame 0xffffff8120c47a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c47ab0
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff8120c47ae0
_sleep() at _sleep+0x336/frame 0xffffff8120c47b80
vm_pagezero() at vm_pagezero+0x88/frame 0xffffff8120c47bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c47bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c47bf0
--- trap 0, rip = 0, rsp = 0xffffff8120c47cb0, rbp = 0 ---

Tracing command vmdaemon pid 6 tid 100051 td 0xfffffe000d150490
sched_switch() at sched_switch+0x482/frame 0xffffff8120c429f0
mi_switch() at mi_switch+0x179/frame 0xffffff8120c42a30
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c42a70
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff8120c42aa0
_sleep() at _sleep+0x369/frame 0xffffff8120c42b40
vm_daemon() at vm_daemon+0x7e/frame 0xffffff8120c42bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c42bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c42bf0
--- trap 0, rip = 0, rsp = 0xffffff8120c42cb0, rbp = 0 ---

Tracing command pagedaemon pid 5 tid 100050 td 0xfffffe000d150920
sched_switch() at sched_switch+0x482/frame 0xffffff8120c3d960
mi_switch() at mi_switch+0x179/frame 0xffffff8120c3d9a0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8120c3d9e0
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff8120c3da10
_sleep() at _sleep+0x336/frame 0xffffff8120c3dab0
vm_pageout() at vm_pageout+0x24b/frame 0xffffff8120c3dbb0
fork_exit() at fork_exit+0x84/frame 0xffffff8120c3dbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8120c3dbf0
--- trap 0, rip = 0, rsp = 0xffffff8120c3dcb0, rbp = 0 ---

Tracing command xpt_thrd pid 4 tid 100049 td 0xfffffe000d02b000
sched_switch() at sched_switch+0x482/frame 0xffffff80003c0a30
mi_switch() at mi_switch+0x179/frame 0xffffff80003c0a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80003c0ab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80003c0ae0
_sleep() at _sleep+0x369/frame 0xffffff80003c0b80
xpt_scanner_thread() at xpt_scanner_thread+0xdc/frame 0xffffff80003c0bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80003c0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003c0bf0
--- trap 0, rip = 0, rsp = 0xffffff80003c0cb0, rbp = 0 ---

Tracing command sctp_iterator pid 3 tid 100048 td 0xfffffe000d02b490
sched_switch() at sched_switch+0x482/frame 0xffffff80003bba40
mi_switch() at mi_switch+0x179/frame 0xffffff80003bba80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80003bbac0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80003bbaf0
_sleep() at _sleep+0x369/frame 0xffffff80003bbb90
sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xffffff80003bbbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80003bbbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003bbbf0
--- trap 0, rip = 0, rsp = 0xffffff80003bbcb0, rbp = 0 ---

Tracing command balloon pid 2 tid 100047 td 0xfffffe000d02b920
sched_switch() at sched_switch+0x482/frame 0xffffff80003b59e0
mi_switch() at mi_switch+0x179/frame 0xffffff80003b5a20
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80003b5a60
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80003b5a90
_sleep() at _sleep+0x369/frame 0xffffff80003b5b30
balloon_process() at balloon_process+0x62/frame 0xffffff80003b5bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80003b5bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003b5bf0
--- trap 0, rip = 0, rsp = 0xffffff80003b5cb0, rbp = 0 ---

Tracing command xenstore_rcv pid 16 tid 100040 td 0xfffffe000d02f000
sched_switch() at sched_switch+0x482/frame 0xffffff800038e940
mi_switch() at mi_switch+0x179/frame 0xffffff800038e980
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff800038e9c0
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff800038ea30
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff800038ea60
_sleep() at _sleep+0x349/frame 0xffffff800038eb00
xs_read_store() at xs_read_store+0x169/frame 0xffffff800038eb80
xs_rcv_thread() at xs_rcv_thread+0x57/frame 0xffffff800038ebb0
fork_exit() at fork_exit+0x84/frame 0xffffff800038ebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800038ebf0
--- trap 0, rip = 0, rsp = 0xffffff800038ecb0, rbp = 0 ---

Tracing command xenwatch pid 15 tid 100039 td 0xfffffe000be75000
sched_switch() at sched_switch+0x482/frame 0xffffff80003899d0
mi_switch() at mi_switch+0x179/frame 0xffffff8000389a10
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8000389a50
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8000389ac0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xffffff8000389af0
_sleep() at _sleep+0x324/frame 0xffffff8000389b90
xenwatch_thread() at xenwatch_thread+0xa4/frame 0xffffff8000389bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000389bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000389bf0
--- trap 0, rip = 0, rsp = 0xffffff8000389cb0, rbp = 0 ---

Tracing command yarrow pid 14 tid 100025 td 0xfffffe000bcb9000
sched_switch() at sched_switch+0x482/frame 0xffffff80002dba30
mi_switch() at mi_switch+0x179/frame 0xffffff80002dba70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002dbab0
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff80002dbae0
msleep_spin_sbt() at msleep_spin_sbt+0x1f0/frame 0xffffff80002dbb70
random_kthread() at random_kthread+0x68/frame 0xffffff80002dbbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002dbbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002dbbf0
--- trap 0, rip = 0, rsp = 0xffffff80002dbcb0, rbp = 0 ---

Tracing command geom pid 13 tid 100023 td 0xfffffe000bcb9920
sched_switch() at sched_switch+0x482/frame 0xffffff80002d1a00
mi_switch() at mi_switch+0x179/frame 0xffffff80002d1a40
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002d1a80
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002d1ab0
_sleep() at _sleep+0x369/frame 0xffffff80002d1b50
g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xffffff80002d1ba0
g_down_procbody() at g_down_procbody+0x7d/frame 0xffffff80002d1bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002d1bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d1bf0
--- trap 0, rip = 0, rsp = 0xffffff80002d1cb0, rbp = 0 ---

Tracing command geom pid 13 tid 100022 td 0xfffffe000bcab000
sched_switch() at sched_switch+0x482/frame 0xffffff80002cca20
mi_switch() at mi_switch+0x179/frame 0xffffff80002cca60
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002ccaa0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002ccad0
_sleep() at _sleep+0x369/frame 0xffffff80002ccb70
g_io_schedule_up() at g_io_schedule_up+0xbc/frame 0xffffff80002ccba0
g_up_procbody() at g_up_procbody+0x7d/frame 0xffffff80002ccbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002ccbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002ccbf0
--- trap 0, rip = 0, rsp = 0xffffff80002cccb0, rbp = 0 ---

Tracing command geom pid 13 tid 100021 td 0xfffffe000bcab490
sched_switch() at sched_switch+0x482/frame 0xffffff80002c7a20
mi_switch() at mi_switch+0x179/frame 0xffffff80002c7a60
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002c7aa0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002c7ad0
_sleep() at _sleep+0x369/frame 0xffffff80002c7b70
g_run_events() at g_run_events+0x464/frame 0xffffff80002c7bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002c7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002c7bf0
--- trap 0, rip = 0, rsp = 0xffffff80002c7cb0, rbp = 0 ---

Tracing command intr pid 12 tid 100044 td 0xfffffe000d02c920
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100043 td 0xfffffe000d02e000
sched_switch() at sched_switch+0x482/frame 0xffffff800039db30
mi_switch() at mi_switch+0x179/frame 0xffffff800039db70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff800039dbb0
fork_exit() at fork_exit+0x84/frame 0xffffff800039dbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800039dbf0
--- trap 0, rip = 0, rsp = 0xffffff800039dcb0, rbp = 0 ---

Tracing command intr pid 12 tid 100042 td 0xfffffe000d02e490
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100041 td 0xfffffe000d02e920
sched_switch() at sched_switch+0x482/frame 0xffffff8000393b30
mi_switch() at mi_switch+0x179/frame 0xffffff8000393b70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff8000393bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000393bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000393bf0
--- trap 0, rip = 0, rsp = 0xffffff8000393cb0, rbp = 0 ---

Tracing command intr pid 12 tid 100038 td 0xfffffe000be75490
sched_switch() at sched_switch+0x482/frame 0xffffff8000364b30
mi_switch() at mi_switch+0x179/frame 0xffffff8000364b70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff8000364bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000364bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000364bf0
--- trap 0, rip = 0, rsp = 0xffffff8000364cb0, rbp = 0 ---

Tracing command intr pid 12 tid 100037 td 0xfffffe000be75920
sched_switch() at sched_switch+0x482/frame 0xffffff800035eb30
mi_switch() at mi_switch+0x179/frame 0xffffff800035eb70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff800035ebb0
fork_exit() at fork_exit+0x84/frame 0xffffff800035ebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800035ebf0
--- trap 0, rip = 0, rsp = 0xffffff800035ecb0, rbp = 0 ---

Tracing command intr pid 12 tid 100036 td 0xfffffe000be76000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100032 td 0xfffffe000be77490
sched_switch() at sched_switch+0x482/frame 0xffffff80002ffb30
mi_switch() at mi_switch+0x179/frame 0xffffff80002ffb70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff80002ffbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002ffbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002ffbf0
--- trap 0, rip = 0, rsp = 0xffffff80002ffcb0, rbp = 0 ---

Tracing command intr pid 12 tid 100030 td 0xfffffe000bcaf490
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100028 td 0xfffffe000bcb8000
sched_switch() at sched_switch+0x482/frame 0xffffff80002ebb30
mi_switch() at mi_switch+0x179/frame 0xffffff80002ebb70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff80002ebbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002ebbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002ebbf0
--- trap 0, rip = 0, rsp = 0xffffff80002ebcb0, rbp = 0 ---

Tracing command intr pid 12 tid 100027 td 0xfffffe000bcb8490
sched_switch() at sched_switch+0x482/frame 0xffffff80002e6b30
mi_switch() at mi_switch+0x179/frame 0xffffff80002e6b70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff80002e6bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002e6bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002e6bf0
--- trap 0, rip = 0, rsp = 0xffffff80002e6cb0, rbp = 0 ---

Tracing command intr pid 12 tid 100020 td 0xfffffe000bcab920
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100019 td 0xfffffe000bcad000
sched_switch() at sched_switch+0x482/frame 0xffffff80002bdb30
mi_switch() at mi_switch+0x179/frame 0xffffff80002bdb70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff80002bdbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002bdbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bdbf0
--- trap 0, rip = 0, rsp = 0xffffff80002bdcb0, rbp = 0 ---

Tracing command intr pid 12 tid 100018 td 0xfffffe000bcad490
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100017 td 0xfffffe000bcad920
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100016 td 0xfffffe000bcaf000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100015 td 0xfffffe000bc95000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100014 td 0xfffffe000bc95490
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100013 td 0xfffffe000bc95920
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100012 td 0xfffffe000bc96000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100011 td 0xfffffe000bc96490
sched_switch() at sched_switch+0x482/frame 0xffffff8000295b30
mi_switch() at mi_switch+0x179/frame 0xffffff8000295b70
ithread_loop() at ithread_loop+0xf0/frame 0xffffff8000295bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000295bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000295bf0
--- trap 0, rip = 0, rsp = 0xffffff8000295cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100010 td 0xfffffe000bc96920
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff800025ccf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff800025cd10
trap() at trap+0x42/frame 0xffffff800025cf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800025cf20
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffff800025cfe0, rbp = 0xffffff8000290a70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000290a70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff8000290ac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff8000290ae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff8000290b00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff8000290bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000290bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000290bf0
--- trap 0, rip = 0, rsp = 0xffffff8000290cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100009 td 0xfffffe000bc89490
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff8000255cf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff8000255d10
trap() at trap+0x42/frame 0xffffff8000255f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff8000255f20
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffff8000255fe0, rbp = 0xffffff800028ba70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800028ba70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff800028bac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff800028bae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff800028bb00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff800028bbb0
fork_exit() at fork_exit+0x84/frame 0xffffff800028bbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800028bbf0
--- trap 0, rip = 0, rsp = 0xffffff800028bcb0, rbp = 0 ---

Tracing command idle pid 11 tid 100008 td 0xfffffe000bc89920
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff800024ecf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff800024ed10
trap() at trap+0x42/frame 0xffffff800024ef20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800024ef20
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffff800024efe0, rbp = 0xffffff8000286a70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000286a70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff8000286ac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff8000286ae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff8000286b00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff8000286bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000286bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000286bf0
--- trap 0, rip = 0, rsp = 0xffffff8000286cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100007 td 0xfffffe000bc94000
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff8000247cf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff8000247d10
trap() at trap+0x42/frame 0xffffff8000247f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff8000247f20
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffff8000247fe0, rbp = 0xffffff8000281a70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000281a70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff8000281ac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff8000281ae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff8000281b00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff8000281bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000281bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000281bf0
--- trap 0, rip = 0, rsp = 0xffffff8000281cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100006 td 0xfffffe000bc94490
cpustop_handler() at cpustop_handler+0x28/frame 0xffffff8000240cf0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffff8000240d10
trap() at trap+0x42/frame 0xffffff8000240f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff8000240f20
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffff8000240fe0, rbp = 0xffffff800027ca70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800027ca70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff800027cac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff800027cae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff800027cb00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff800027cbb0
fork_exit() at fork_exit+0x84/frame 0xffffff800027cbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800027cbf0
--- trap 0, rip = 0, rsp = 0xffffff800027ccb0, rbp = 0 ---

Tracing command idle pid 11 tid 100005 td 0xfffffe000bc94920
kdb_alt_break_internal() at kdb_alt_break_internal+0x17f/frame 0xffffff8000277870
kdb_alt_break() at kdb_alt_break+0xb/frame 0xffffff8000277880
uart_intr_rxready() at uart_intr_rxready+0x96/frame 0xffffff80002778b0
uart_intr() at uart_intr+0x111/frame 0xffffff80002778f0
intr_event_handle() at intr_event_handle+0x8c/frame 0xffffff8000277940
intr_execute_handlers() at intr_execute_handlers+0x48/frame 0xffffff8000277970
lapic_handle_intr() at lapic_handle_intr+0x73/frame 0xffffff80002779a0
Xapic_isr1() at Xapic_isr1+0xa4/frame 0xffffff80002779a0
--- interrupt, rip = 0xffffffff80c46446, rsp = 0xffffff8000277a60, rbp = 0xffffff8000277a70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000277a70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff8000277ac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff8000277ae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff8000277b00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff8000277bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000277bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000277bf0
--- trap 0, rip = 0, rsp = 0xffffff8000277cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100004 td 0xfffffe000bc88000
sched_switch() at sched_switch+0x482/frame 0xffffff8000272ac0
mi_switch() at mi_switch+0x179/frame 0xffffff8000272b00
sched_idletd() at sched_idletd+0x102/frame 0xffffff8000272bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000272bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000272bf0
--- trap 0, rip = 0, rsp = 0xffffff8000272cb0, rbp = 0 ---

Tracing command idle pid 11 tid 100003 td 0xfffffe000bc88490
cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff8153ccb0
ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff8153ccd0
trap() at trap+0x42/frame 0xffffffff8153cee0
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff8153cee0
--- trap 0x13, rip = 0xffffffff80c46446, rsp = 0xffffffff8153cfa0, rbp = 0xffffff800026da70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800026da70
acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xffffff800026dac0
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xffffff800026dae0
cpu_idle() at cpu_idle+0x93/frame 0xffffff800026db00
sched_idletd() at sched_idletd+0x1e8/frame 0xffffff800026dbb0
fork_exit() at fork_exit+0x84/frame 0xffffff800026dbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026dbf0
--- trap 0, rip = 0, rsp = 0xffffff800026dcb0, rbp = 0 ---

Tracing command init pid 1 tid 100002 td 0xfffffe000bc88920
sched_switch() at sched_switch+0x482/frame 0xffffff8000268690
mi_switch() at mi_switch+0x179/frame 0xffffff80002686d0
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8000268710
sleepq_catch_signals() at sleepq_catch_signals+0x30d/frame 0xffffff8000268780
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xffffff80002687b0
_sleep() at _sleep+0x349/frame 0xffffff8000268850
kern_wait6() at kern_wait6+0x3e7/frame 0xffffff80002688f0
sys_wait4() at sys_wait4+0x76/frame 0xffffff8000268ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8000268bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8000268bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41251a, rsp = 0x7fffffffd758, rbp = 0x7fffffffd7c0 ---

Tracing command audit pid 10 tid 100001 td 0xfffffe000bc89000
sched_switch() at sched_switch+0x482/frame 0xffffff8000263a30
mi_switch() at mi_switch+0x179/frame 0xffffff8000263a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8000263ab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff8000263ae0
_cv_wait() at _cv_wait+0x1df/frame 0xffffff8000263b50
audit_worker() at audit_worker+0xa3/frame 0xffffff8000263bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000263bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000263bf0
--- trap 0, rip = 0, rsp = 0xffffff8000263cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100046 td 0xfffffe000d02c000
sched_switch() at sched_switch+0x482/frame 0xffffff80003b09c0
mi_switch() at mi_switch+0x179/frame 0xffffff80003b0a00
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80003b0a40
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffff80003b0a70
_sleep() at _sleep+0x336/frame 0xffffff80003b0b10
pause_sbt() at pause_sbt+0x70/frame 0xffffff80003b0b60
deadlkres() at deadlkres+0x66/frame 0xffffff80003b0bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80003b0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003b0bf0
--- trap 0, rip = 0, rsp = 0xffffff80003b0cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100045 td 0xfffffe000d02c490
sched_switch() at sched_switch+0x482/frame 0xffffff80003aba40
mi_switch() at mi_switch+0x179/frame 0xffffff80003aba80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80003abac0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80003abaf0
msleep_spin_sbt() at msleep_spin_sbt+0x200/frame 0xffffff80003abb80
taskqueue_thread_loop() at taskqueue_thread_loop+0x8d/frame 0xffffff80003abbb0
fork_exit() at fork_exit+0x84/frame 0xffffff80003abbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003abbf0
--- trap 0, rip = 0, rsp = 0xffffff80003abcb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100035 td 0xfffffe000be76490
sched_switch() at sched_switch+0x482/frame 0xffffff800030ea40
mi_switch() at mi_switch+0x179/frame 0xffffff800030ea80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff800030eac0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff800030eaf0
msleep_spin_sbt() at msleep_spin_sbt+0x200/frame 0xffffff800030eb80
taskqueue_thread_loop() at taskqueue_thread_loop+0x8d/frame 0xffffff800030ebb0
fork_exit() at fork_exit+0x84/frame 0xffffff800030ebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800030ebf0
--- trap 0, rip = 0, rsp = 0xffffff800030ecb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100034 td 0xfffffe000be76920
sched_switch() at sched_switch+0x482/frame 0xffffff8000309a40
mi_switch() at mi_switch+0x179/frame 0xffffff8000309a80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8000309ac0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff8000309af0
msleep_spin_sbt() at msleep_spin_sbt+0x200/frame 0xffffff8000309b80
taskqueue_thread_loop() at taskqueue_thread_loop+0x8d/frame 0xffffff8000309bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000309bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000309bf0
--- trap 0, rip = 0, rsp = 0xffffff8000309cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100033 td 0xfffffe000be77000
sched_switch() at sched_switch+0x482/frame 0xffffff8000304a40
mi_switch() at mi_switch+0x179/frame 0xffffff8000304a80
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff8000304ac0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff8000304af0
msleep_spin_sbt() at msleep_spin_sbt+0x200/frame 0xffffff8000304b80
taskqueue_thread_loop() at taskqueue_thread_loop+0x8d/frame 0xffffff8000304bb0
fork_exit() at fork_exit+0x84/frame 0xffffff8000304bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000304bf0
--- trap 0, rip = 0, rsp = 0xffffff8000304cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100031 td 0xfffffe000be77920
sched_switch() at sched_switch+0x482/frame 0xffffff80002faa30
mi_switch() at mi_switch+0x179/frame 0xffffff80002faa70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002faab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002faae0
_sleep() at _sleep+0x369/frame 0xffffff80002fab80
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xffffff80002fabb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002fabf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002fabf0
--- trap 0, rip = 0, rsp = 0xffffff80002facb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100029 td 0xfffffe000bcaf920
sched_switch() at sched_switch+0x482/frame 0xffffff80002f0a30
mi_switch() at mi_switch+0x179/frame 0xffffff80002f0a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002f0ab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002f0ae0
_sleep() at _sleep+0x369/frame 0xffffff80002f0b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xffffff80002f0bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002f0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002f0bf0
--- trap 0, rip = 0, rsp = 0xffffff80002f0cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100026 td 0xfffffe000bcb8920
sched_switch() at sched_switch+0x482/frame 0xffffff80002e1a30
mi_switch() at mi_switch+0x179/frame 0xffffff80002e1a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002e1ab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002e1ae0
_sleep() at _sleep+0x369/frame 0xffffff80002e1b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xffffff80002e1bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002e1bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002e1bf0
--- trap 0, rip = 0, rsp = 0xffffff80002e1cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100024 td 0xfffffe000bcb9490
sched_switch() at sched_switch+0x482/frame 0xffffff80002d6a30
mi_switch() at mi_switch+0x179/frame 0xffffff80002d6a70
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffff80002d6ab0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff80002d6ae0
_sleep() at _sleep+0x369/frame 0xffffff80002d6b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xffffff80002d6bb0
fork_exit() at fork_exit+0x84/frame 0xffffff80002d6bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d6bf0
--- trap 0, rip = 0, rsp = 0xffffff80002d6cb0, rbp = 0 ---

Tracing command kernel pid 0 tid 100000 td 0xffffffff8158d010
sched_switch() at sched_switch+0x482/frame 0xffffffff818d5af0
mi_switch() at mi_switch+0x179/frame 0xffffffff818d5b30
sleepq_switch() at sleepq_switch+0x18a/frame 0xffffffff818d5b70
sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffffff818d5ba0
_sleep() at _sleep+0x336/frame 0xffffffff818d5c40
scheduler() at scheduler+0x261/frame 0xffffffff818d5c90
mi_startup() at mi_startup+0x118/frame 0xffffffff818d5cb0
btext() at btext+0x2c

[Roger Pau Monné]

On 13/05/13 13:18, Roger Pau Monn� wrote:
> The VM can be stuck in this state for quite some time, it generally

I would like to explain this a little bit more, the syncer process
doesn't get blocked on the _mtx_trylock_flags_ call, it just continues
looping forever in what seems to be an endless loop around
mnt_vnode_next_active/ffs_sync. Also while in this state there is no
noticeable disk activity, so I'm unsure of what is happening.

> varies between a couple of minutes (5-10min) to an hour or two, after
> this the VM recovers itself and resumes normal operation. I still have
> to test this on a bare metal FreeBSD install, but I would like to ask
> if someone has seen a similar behaviour, or if someone is suspicious of
> a change that could cause this.

[Roger Pau Monné]

On 13/05/13 17:00, Konstantin Belousov wrote:
> On Mon, May 13, 2013 at 04:33:04PM +0200, Roger Pau Monn? wrote:
>> On 13/05/13 13:18, Roger Pau Monn? wrote:

Thanks for taking a look,

>> I would like to explain this a little bit more, the syncer process
>> doesn't get blocked on the _mtx_trylock_flags_ call, it just continues
>> looping forever in what seems to be an endless loop around
>> mnt_vnode_next_active/ffs_sync. Also while in this state there is no
>> noticeable disk activity, so I'm unsure of what is happening.

> How many CPUs does your VM have ?

7 vCPUs, but I've also seen this issue with 4 and 16 vCPUs.

>
> The loop you describing means that other thread owns the vnode
> interlock. Can you track what this thread does ? E.g. look at the
> vp->v_interlock.mtx_lock, which is basically a pointer to the struct
> thread owning the mutex, clear low bits as needed. Then you can
> inspect the thread and get a backtrace.

There are no other threads running, only syncer is running on CPU 1 (see
ps in previous email). All other CPUs are idle, and as seen from the ps
quite a lot of threads are blocked in vnode related operations, either
"*Name Cac", "*vnode_fr" or "*vnode in". I've also attached the output
of alllocks in the previous email.

>
> Does the loop you described stuck on the same vnode during the whole
> lock-step time, or is the progress made, possibly slowly ?

I'm not sure how to measure "progress", but indeed the syncer process is
not locked, it is iterating over mnt_vnode_next_active.

>
> I suppose that your HEAD is recent.

Last commit in my local repository is:

Date: Tue, 7 May 2013 12:39:14 +0000
Subject: [PATCH] By request, add an arrow from NetBSD-0.8 to FreeBSD-1.0.

While here, add a few more NetBSD versions to the tree itself.

Submitted by: Alan Barrett <a...@cequrux.com>
Submitted by: Thomas Klausner <w...@netbsd.org>

[Roger Pau Monné]

On 14/05/13 18:31, Konstantin Belousov wrote:

> On Tue, May 14, 2013 at 06:08:45PM +0200, Roger Pau Monn? wrote:
>> On 13/05/13 17:00, Konstantin Belousov wrote:
>>> On Mon, May 13, 2013 at 04:33:04PM +0200, Roger Pau Monn? wrote:
>>>> On 13/05/13 13:18, Roger Pau Monn? wrote:
>>
>> Thanks for taking a look,
>>
>>>> I would like to explain this a little bit more, the syncer process
>>>> doesn't get blocked on the _mtx_trylock_flags_ call, it just continues
>>>> looping forever in what seems to be an endless loop around
>>>> mnt_vnode_next_active/ffs_sync. Also while in this state there is no
>>>> noticeable disk activity, so I'm unsure of what is happening.
>>> How many CPUs does your VM have ?
>>
>> 7 vCPUs, but I've also seen this issue with 4 and 16 vCPUs.
>>
>>>
>>> The loop you describing means that other thread owns the vnode
>>> interlock. Can you track what this thread does ? E.g. look at the
>>> vp->v_interlock.mtx_lock, which is basically a pointer to the struct
>>> thread owning the mutex, clear low bits as needed. Then you can
>>> inspect the thread and get a backtrace.
>>
>> There are no other threads running, only syncer is running on CPU 1 (see
>> ps in previous email). All other CPUs are idle, and as seen from the ps
>> quite a lot of threads are blocked in vnode related operations, either
>> "*Name Cac", "*vnode_fr" or "*vnode in". I've also attached the output
>> of alllocks in the previous email.

> This is not useful. You need to look at the mutex which fails the
> trylock operation in the mnt_vnode_next_active(), see who owns it,
> and then 'unwind' the locking dependencies from there.

Sorry, now I get it, let's see if I can find the locked vnodes and the
thread that owns them...

[John Baldwin]

You can use 'show lock <address of vp->v_interlock>' to find an owning
thread and then use 'show sleepchain <thread>'. If you are using kgdb on the
live system (probably easier) then you can grab my scripts at
www.freebsd.org/~jhb/gdb/ (do 'cd /path/to/scripts; source gdb6'). You can
then find the offending thread and do 'mtx_owner &vp->v_interlock' and then
'sleepchain <tid>'

--
John Baldwin

[Roger Pau Monné]

Hello,

Sorry for the delay in debugging this, but the fact that the issue
itself is hard to reproduce, and that I didn't have much time prevented
me from digging into this. It seems like the locked node is always the
same, or at least I haven't been able to see this loop with more than
one node locked (so far):

db> show lock 0xfffffe0030cdf448
class: sleep mutex
name: vnode interlock
flags: {DEF}
state: {OWNED}
owner: 0xfffffe008d1e9000 (tid 101020, pid 66630, "cc")
db> show sleepchain 0xfffffe008d1e9000
thread 101020 (pid 66630, cc) inhibited
db> tr 66630
Tracing pid 66630 tid 101020 td 0xfffffe008d1e9000
sched_switch() at sched_switch+0x482/frame 0xffffff8120ff3630
mi_switch() at mi_switch+0x179/frame 0xffffff8120ff3670
turnstile_wait() at turnstile_wait+0x3ac/frame 0xffffff8120ff36c0
__mtx_lock_sleep() at __mtx_lock_sleep+0x255/frame 0xffffff8120ff3740
__mtx_lock_flags() at __mtx_lock_flags+0xda/frame 0xffffff8120ff3780
vdropl() at vdropl+0x255/frame 0xffffff8120ff37b0
vputx() at vputx+0x27c/frame 0xffffff8120ff3810
namei() at namei+0x3dd/frame 0xffffff8120ff38c0
kern_statat_vnhook() at kern_statat_vnhook+0x99/frame 0xffffff8120ff3a40
sys_stat() at sys_stat+0x2d/frame 0xffffff8120ff3ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8120ff3bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8120ff3bf0
--- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x18280ea, rsp = 0x7fffffffa148, rbp = 0x7fffffffa170 ---
db> ps

pid ppid pgrp uid state wmesg wchan cmd

66630 66591 66588 0 L+ *vnode_fr 0xfffffe000bc9fa80 cc
66591 66590 66588 0 S+ wait 0xfffffe0030dbd4b8 cc
66590 66588 66588 0 S+ wait 0xfffffe008de20000 sh
66588 66572 66588 0 S+ wait 0xfffffe008d8a8970 sh
66572 49649 49649 0 S+ select 0xfffffe003029dac0 make
49649 49631 49649 0 S+ wait 0xfffffe000d801970 sh
49631 49629 49629 0 S+ select 0xfffffe00302b3cc0 make
49629 49614 49629 0 S+ wait 0xfffffe008dbfa970 sh
49614 45214 45214 0 S+ select 0xfffffe00303e3640 make
45214 45207 45214 0 S+ wait 0xfffffe008dbfa000 sh
45207 45205 45200 0 S+ select 0xfffffe000d66d3c0 make
45205 45200 45200 0 S+ wait 0xfffffe008da1b4b8 sh
45200 757 45200 0 S+ select 0xfffffe000d6093c0 make
30611 737 30611 0 S+ ttyin 0xfffffe000bd70ca8 bash
2325 756 2325 0 Ss+ ttyin 0xfffffe008dd6eca8 bash
757 756 757 0 Ss+ wait 0xfffffe000d7fd4b8 bash
756 755 756 0 Ss select 0xfffffe000d783a40 screen
755 744 755 0 S+ pause 0xfffffe000d9560a8 screen
744 743 744 0 S+ wait 0xfffffe000d4d6970 bash
743 742 743 1001 S+ wait 0xfffffe000d4d5000 su
742 741 742 1001 Ss+ wait 0xfffffe000d6674b8 sh
741 738 738 1001 S select 0xfffffe000d60a8c0 sshd
738 680 738 0 Ss select 0xfffffe000d60a940 sshd
737 1 737 0 Ss+ wait 0xfffffe000d605970 login
736 1 736 0 Ss+ ttyin 0xfffffe000bd718a8 getty
735 1 735 0 Ss+ ttyin 0xfffffe000bd71ca8 getty
734 1 734 0 Ss+ ttyin 0xfffffe000bfcc0a8 getty
733 1 733 0 Ss+ ttyin 0xfffffe000bfcc4a8 getty
732 1 732 0 Ss+ ttyin 0xfffffe000bfcc8a8 getty
731 1 731 0 Ss+ ttyin 0xfffffe000bd700a8 getty
730 1 730 0 Ss+ ttyin 0xfffffe000bd704a8 getty
729 1 729 0 Ss+ ttyin 0xfffffe000bd708a8 getty
690 1 690 0 Ss nanslp 0xffffffff81343734 cron
686 1 686 25 Ss pause 0xfffffe000d77a560 sendmail
683 1 683 0 Ss select 0xfffffe000d7834c0 sendmail
680 1 680 0 Ss select 0xfffffe000d4a5e40 sshd
580 1 580 0 Ss select 0xfffffe000d66d040 syslogd
486 1 486 0 Ss select 0xfffffe000d783540 devd
485 1 485 65 Ss select 0xfffffe000d962040 dhclient
445 1 445 0 Ss select 0xfffffe000d66d1c0 dhclient
18 0 0 0 DL sdflush 0xffffffff8153675c [softdepflush]
17 0 0 0 DL vlruwt 0xfffffe000d4d5970 [vnlru]
9 0 0 0 RL CPU 2 [syncer]
8 0 0 0 DL psleep 0xffffffff8152f72c [bufdaemon]
7 0 0 0 DL pgzero 0xffffffff81537e8c [pagezero]
6 0 0 0 DL psleep 0xffffffff815370d0 [vmdaemon]
5 0 0 0 DL psleep 0xffffffff815b3304 [pagedaemon]
4 0 0 0 DL ccb_scan 0xffffffff813038b0 [xpt_thrd]
3 0 0 0 DL waiting_ 0xffffffff815a73e0 [sctp_iterator]
2 0 0 0 DL balloon 0xffffffff8079d310 [balloon]

16 0 0 0 SL xbread 0xfffffe00feffc000 [xenstore_rcv]

15 0 0 0 SL waitev 0xffffffff81538098 [xenwatch]
14 0 0 0 DL - 0xffffffff8131f988 [yarrow]

13 0 0 0 DL (threaded) [geom]

100021 D - 0xffffffff8159bc70 [g_down]
100020 D - 0xffffffff8159bc68 [g_up]
100019 D - 0xffffffff8159bc60 [g_event]

12 0 0 0 WL (threaded) [intr]

100050 I [irq778: xn0]
100049 I [irq777: xbd1]
100048 I [irq776: xbd0]
100042 I [irq7: ppc0]
100041 I [swi0: uart]
100040 I [irq12: psm0]
100039 I [irq1: atkbd0]
100036 I [irq775: xenstore0]
100035 I [irq15: ata1]
100034 I [irq14: ata0]
100032 I [swi5: fast taskq]
100030 I [swi6: Giant taskq]
100029 I [swi6: task queue]
100024 I [swi2: cambio]
100018 I [swi3: vm]

100017 I [swi4: clock]
100016 I [swi4: clock]
100015 I [swi4: clock]
100014 I [swi4: clock]
100013 I [swi4: clock]
100012 I [swi4: clock]
100011 I [swi4: clock]

100010 I [swi1: netisr 0]

11 0 0 0 RL (threaded) [idle]

100009 Run CPU 6 [idle: cpu6]
100008 Run CPU 5 [idle: cpu5]
100007 Run CPU 4 [idle: cpu4]
100006 Run CPU 3 [idle: cpu3]

100005 CanRun [idle: cpu2]
100004 Run CPU 1 [idle: cpu1]

100003 Run CPU 0 [idle: cpu0]

1 0 1 0 SLs wait 0xfffffe000bc77970 [init]
10 0 0 0 DL audit_wo 0xffffffff815aaae0 [audit]

0 0 0 0 DLs (threaded) [kernel]

100044 D - 0xffffffff81342593 [deadlkres]
100043 D - 0xfffffe000bfc5300 [mca taskq]
100033 D - 0xfffffe000bcaa600 [kqueue taskq]
100031 D - 0xfffffe000bcaaa00 [thread taskq]
100028 D - 0xfffffe000be5c100 [ffs_trim taskq]
100027 D - 0xfffffe000bca1300 [acpi_task_2]
100026 D - 0xfffffe000bca1300 [acpi_task_1]
100025 D - 0xfffffe000bca1300 [acpi_task_0]
100022 D - 0xfffffe000bc6e300 [firmware taskq]
100000 D sched 0xffffffff8159bd68 [swapper]

[Roger Pau Monné]

I've been looking into this issue a little bit more, and the lock
dependencies look right to me, the lockup happens when the thread owning
the v_interlock mutex tries to acquire the vnode_free_list_mtx mutex
which is already owned by the syncer thread, at this point, the thread
owning the v_interlock mutex goes to sleep, and the syncer process will
start doing a sequence of:

VI_TRYLOCK -> mtx_unlock vnode_free_list_mtx -> kern_yield -> mtx_lock
vnode_free_list_mtx ...

It seems like kern_yield, which I assume is placed there in order to
allow the thread owning v_interlock to be able to also lock
vnode_free_list_mtx, doesn't get a window big enough to wake up the
waiting thread and get the vnode_free_list_mtx mutex. Since the syncer
is the only process runnable on the CPU there is no context switch, and
the syncer process continues to run.

Relying on kern_yield to provide a window big enough that allows any
other thread waiting on vnode_free_list_mtx to run doesn't seem like a
good idea on SMP systems. I've not tested this on bare metal, but waking
up an idle CPU in a virtualized environment might be more expensive than
doing it on bare metal.

Bear in mind that I'm not familiar with either the scheduler or the ufs
code, my proposed naive fix is to replace the kern_yield call with a
pause, that will allow any other threads waiting on vnode_free_list_mtx
to lock the vnode_free_list_mtx mutex and finish whatever they are doing
and release the v_interlock mutex, so the syncer thread can also finish
it's work. I've tested the patch for a couple of hours and seems to be
fine, I haven't been able to reproduce the issue anymore.

[Jilles Tjoelker]

Instead of a pause() that may be too short or too long, how about
waiting for the necessary lock? In other words, replace the kern_yield()
call with VI_LOCK(vp); VI_UNLOCK(vp);. This is also the usual approach
to acquire two locks without imposing an order between them.

I expect blocking on a mutex to be safe enough; a mutex may not be held
across waiting for hardware or other events.

> >From fec90f7bb9cdf05b49d11dbe4930d3c595c147f5 Mon Sep 17 00:00:00 2001
> From: Roger Pau Monne <roge...@citrix.com>
> Date: Sun, 26 May 2013 19:55:43 +0200
> Subject: [PATCH] mnt_vnode_next_active: replace kern_yield with pause
>
> On SMP systems there is no way to assure that a kern_yield will allow
> any other threads waiting on the vnode_free_list_mtx to be able to
> acquire it. The syncer process can get stuck in a loop trying to lock
> the v_interlock mutex, without allowing other threads waiting on
> vnode_free_list_mtx to run. Replace the kern_yield with a pause, that
> should allow any thread owning v_interlock and waiting on
> vnode_free_list_mtx to finish it's work and release v_interlock.
> ---
> sys/kern/vfs_subr.c | 10 +++++++++-
> 1 files changed, 9 insertions(+), 1 deletions(-)
>
> diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c
> index 0da6764..597f4b7 100644
> --- a/sys/kern/vfs_subr.c
> +++ b/sys/kern/vfs_subr.c
> @@ -4703,7 +4703,15 @@ restart:
> if (mp_ncpus == 1 || should_yield()) {
> TAILQ_INSERT_BEFORE(vp, *mvp, v_actfreelist);
> mtx_unlock(&vnode_free_list_mtx);
> - kern_yield(PRI_USER);
> + /*
> + * There is another thread owning the
> + * v_interlock mutex and possibly waiting on
> + * vnode_free_list_mtx, so pause in order for
> + * it to acquire the vnode_free_list_mtx
> + * mutex and finish the work, releasing
> + * v_interlock when finished.
> + */
> + pause("vi_lock", 1);
> mtx_lock(&vnode_free_list_mtx);
> goto restart;
> }
> --
> 1.7.7.5 (Apple Git-26)

--
Jilles Tjoelker

[Roger Pau Monné]

On 26/05/13 22:20, Jilles Tjoelker wrote:
> Instead of a pause() that may be too short or too long, how about
> waiting for the necessary lock? In other words, replace the kern_yield()
> call with VI_LOCK(vp); VI_UNLOCK(vp);. This is also the usual approach
> to acquire two locks without imposing an order between them.

Since there might be more than one locked vnode, waiting on a specific
locked vnode seemed rather arbitrary, but I agree that the pause is also
rather arbitrary.

Also, can we be sure that the v_interlock mutex will not be destroyed
while the syncer process is waiting for it to be unlocked?

> I expect blocking on a mutex to be safe enough; a mutex may not be held
> across waiting for hardware or other events.
>

[Jilles Tjoelker]

On Sun, May 26, 2013 at 10:52:07PM +0200, Roger Pau Monn� wrote:
> On 26/05/13 22:20, Jilles Tjoelker wrote:
> > Instead of a pause() that may be too short or too long, how about
> > waiting for the necessary lock? In other words, replace the kern_yield()
> > call with VI_LOCK(vp); VI_UNLOCK(vp);. This is also the usual approach
> > to acquire two locks without imposing an order between them.

> Since there might be more than one locked vnode, waiting on a specific
> locked vnode seemed rather arbitrary, but I agree that the pause is also
> rather arbitrary.

> Also, can we be sure that the v_interlock mutex will not be destroyed
> while the syncer process is waiting for it to be unlocked?

I think this is a major problem. My idea was too easy and will not work.

That said, the code in mnt_vnode_next_active() appears to implement some
sort of adaptive spinning for SMP. It tries VI_TRYLOCK for 200ms
(default value of hogticks) and then yields. This is far too long for a
mutex lock and if it takes that long it means that either the thread
owning the lock is blocked by us somehow or someone is abusing a mutex
to work like a sleepable lock such as by spinning or DELAY.

Given that it has been spinning for 200ms, it is not so bad to pause for
one additional microsecond.

The adaptive spinning was added fairly recently, so apparently it
happens fairly frequently that VI_TRYLOCK fails transiently.
Unfortunately, the real adaptive spinning code cannot be used because it
will spin forever as long as the thread owning v_interlock is running,
including when that is because it is spinning for vnode_free_list_mtx.
Perhaps we can try to VI_TRYLOCK a certain number of times. It is also
possible to check the contested bit of vnode_free_list_mtx
(sys/netgraph/netflow/netflow.c does something similar) and stop
spinning in that case.

A cpu_spinwait() invocation should also be added to the spin loop.

--
Jilles Tjoelker

[Roger Pau Monné]

On 27/05/13 08:07, Konstantin Belousov wrote:
> On Mon, May 27, 2013 at 12:22:54AM +0200, Jilles Tjoelker wrote:

> There are two 'proper' solutions for this issue:
>
> One is to change the handling of the vnode lifecycle to allow the
> safe block for the vnode interlock acquisition. In particular, the
> change would add some stability of the vnode memory when vnode is
> put on the free list. As example, the vnode zone could be marked as
> type-stable again, and then the vnode interlock can be obtained with
> dropped free list lock. Arguably, marking the zone as non-freeable would
> be a regression, esp. for the zone which accounts for largest allocation
> on the kernel memory.
>
> Another one is to somehow ensure that the priority is properly
> propagated from the spinning thread to the vnode interlock owner.
> I think that it is desirable to donate some amount of priority
> from the spinning thread. Unfortunately, I was unable to come
> up with elegant solution for this which would be also contained
> and did not require rewamp of the mutex interfaces.
>
> BTW, if anybody come up with the idea of the restructuring the free list
> handling to avoid the free list/vnode interlock LOR altogether, it would
> be the best.
>
> I do not have objections against the pause() addition, but I would
> argue that should_yield() should be removed then, switching the code to
> unconditionally pause when the collision detected.

Taking the idea from Jilles, what about replacing should_yield with a
check to see if the vnode_free_list_mtx mutex is contented?

That would prevent us from doing unnecessary pauses, and only releasing
the vnode_free_list_mtx mutex when there's someone else that actually
needs it.

[Roger Pau Monné]

On 27/05/13 12:23, Konstantin Belousov wrote:

> This would still be racy, and possibly allows the lock convoy in the same
> manner as the current code. Also, AFAIR, the real problem was when
> two iterators start synchronized, it usually ended in live-lock.
>
> If you are willing, try this, of course, but I tend to agree with just
> an addition of pause() for now.

OK, I've tested replacing kern_yield with a pause the whole night and
that seems to be working as expected, I did no longer see any lockups,
usually during a whole night run I saw at least 3 or 4 lockups.

If you are happy (and others) with the replacement you can commit it and
we can replace the should_yield call later if needed.

Thanks, Roger.

[Andriy Gapon]

In addition, should_yield() seems to have a problem:
http://article.gmane.org/gmane.os.freebsd.devel.cvs.src/167287

--
Andriy Gapon