ZFS DEADLKRES

5 views
Skip to first unread message

Marcin Cieslak

unread,
Feb 17, 2010, 6:26:36 PM2/17/10
to freebsd...@freebsd.org
My r203753 amd64 laptop falls into the deadlock situation
every night while running periodic daily script. I am pretty
certain this is related to ZFS.

I have enabled DEADLKRES in the kernel. I even have
a separate dump partition (not used for swap).

Indeed, the kernel deadlock detector gets triggered
during the periodic run, but I get I/O error when
calling doadump() from DDB.

Dumping works properly when DDB is invoked with keyboard
escape.

What's the best way to troubleshoot this deadlock?
Should I try to transcribe "alltrace" output?

--
<< Marcin Cieslak // sa...@saper.info >>

Attilio Rao

unread,
Feb 20, 2010, 12:46:10 PM2/20/10
to Marcin Cieslak, freebsd...@freebsd.org
2010/2/18 Marcin Cieslak <sa...@saper.info>:

> My r203753 amd64 laptop falls into the deadlock situation
> every night while running periodic daily script. I am pretty
> certain this is related to ZFS.
>
> I have enabled DEADLKRES in the kernel. I even have
> a separate dump partition (not used for swap).

May you reproduce the bug with WITNESS?
If you can, you should enable DEADLKRES too and once it panics let
extract a textdump(4) with the following commands:
bt, ps, show alllocks, show pcpu, allthreads

and report here.
We can then comment if further investigation is due.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein

Marcin Cieslak

unread,
Feb 21, 2010, 7:24:34 PM2/21/10
to Attilio Rao, freebsd...@freebsd.org


On Sat, 20 Feb 2010, Attilio Rao wrote:

> 2010/2/18 Marcin Cieslak <sa...@saper.info>:
>> My r203753 amd64 laptop falls into the deadlock situation
>> every night while running periodic daily script. I am pretty
>> certain this is related to ZFS.
>>
>> I have enabled DEADLKRES in the kernel. I even have
>> a separate dump partition (not used for swap).
>
> May you reproduce the bug with WITNESS?
> If you can, you should enable DEADLKRES too and once it panics let
> extract a textdump(4) with the following commands:
> bt, ps, show alllocks, show pcpu, allthreads

Unfortunately, there is no way to write anything to disk.
All attempts to talk to the ata subsystem from ddb(4)
fail with EIO (probably timeout).

I did what I could using broken firewire console.

At first, "periodic daily" processing progresses normally
(lots of disk activity).

Before deadlock detector kicks in the system, although
busy, keeps responding to keystrokes. Any access
to the ata disk blocks the process. The hard drive
LED is on all the time (no blinking) and the drive
itself appears to be quiet.

Anyway, "ls" and "find" operations are usually
very slow (and generate lots of I/O activity).
This happens to me since I am running ZFS
(ca. FreeBSD 7.2-something).

Below the output requested (separated by dashes):

----------------
bt:

[dcons connected]

db> bt
Tracing pid 0 tid 100067 td 0xffffff00037b8390
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x176
deadlkres() at deadlkres+0x229
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc5cd30, rbp = 0 ---

----------------
show pcpu:

db> show pcpu
cpuid = 0
dynamic pcpu = 0x14ced80
curthread = 0xffffff00037b8390: pid 0 "deadlkres"
curpcb = 0xffffff803dc5cd40
fpcurthread = none
idlethread = 0xffffff000336a390: pid 11 "idle: cpu0"
curpmap = 0
tssp = 0xffffffff80f3ec00
commontssp = 0xffffffff80f3ec00
rsp0 = 0xffffff803dc5cd40
gs32p = 0xffffffff80f3da38
ldt = 0xffffffff80f3da78
tss = 0xffffffff80f3da68
spin locks held:
db>

----------------
alllocks/vnodes:

db> shw ow alllocks
Process 3498 (sendmail) thread 0xffffff0009615390 (100158)
shared lockmgr zfs (zfs) r = 0 (0xffffff0006905098) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 3496 (sendmail) thread 0xffffff004fbd1720 (100235)
shared lockmgr zfs (zfs) r = 0 (0xffffff0006905098) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 3495 (sendmail) thread 0xffffff004fbd2ab0 (100226)
shared lockmgr zfs (zfs) r = 0 (0xffffff0079423098) locked @ /usr/src/sys/kern/vfs_syscalls.c:4102
Process 3494 (sendmail) thread 0xffffff004fbac390 (100224)
shared lockmgr zfs (zfs) r = 0 (0xffffff0006905098) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 3207 (find) thread 0xffffff002c6fb000 (100190)
shared lockmgr zfs (zfs) r = 0 (0xffffff0014c38cf0) locked @ /usr/src/sys/kern/vfs_lookup.c:502
Process 2796 (login) thread 0xffffff000974e000 (100186)
shared lockmgr zfs (zfs) r = 0 (0xffffff00095e4588) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 2560 (cron) thread 0xffffff002c69b720 (100196)
shared lockmgr zfs (zfs) r = 0 (0xffffff0076383588) locked @ /usr/src/sys/kern/vfs_lookup.c:502
Process 1991 (ntpd) thread 0xffffff002c64bab0 (100191)
exclusive lockmgr zfs (zfs) r = 0 (0xffffff0009541cf0) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 1946 (hcsecd) thread 0xffffff0009753390 (100181)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff002c6b3648) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148
Process 1731 (amd) thread 0xffffff0009615000 (100159)
shared lockmgr zfs (zfs) r = 0 (0xffffff0006905098) locked @ /usr/src/sys/kern/vfs_subr.c:2091
Process 1590 (syslogd) thread 0xffffff0009618000 (100151)
exclusive lockmgr zfs (zfs) r = 0 (0xffffff001e7e8800) locked @ /usr/src/sys/kern/vfs_vnops.c:607
Process 23 (syncer) thread 0xffffff00069d3000 (100073)
exclusive lockmgr syncer (syncer) r = 0 (0xffffff00095e4800) locked @ /usr/src/sys/kern/vfs_subr.c:1701
db> show lockedvnods
Locked vnodes

0xffffff0006905000: tag zfs, type VDIR
usecount 4, writecount 0, refcount 5 mountedhere 0
flags ()
v_object 0xffffff004fb97d80 ref 0 pages 0
lock type zfs: SHARED (count 4)
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff806a8764 at vget+0x6c
#5 0xffffffff80696579 at cache_lookup+0x4dd
#6 0xffffffff806968bd at vfs_cache_lookup+0xad
#7 0xffffffff8097098b at VOP_LOOKUP_APV+0xba
#8 0xffffffff8069d1cf at lookup+0x45e
#9 0xffffffff8069e14a at namei+0x4cb
#10 0xffffffff806aca3c at kern_statat_vnhook+0x82
#11 0xffffffff806acbe1 at kern_statat+0x15
#xffffffff806acd36 at stat+0x22
#13 0xffffffff80922e9b at syscall+0x118
#14 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff0014c38c58: tag zfs, type VDIR
usecount 3, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xffffff002936f948 ref 0 pages 0
lock type zfs: SHARED (count 1)
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff8069ce5a at lookup+0xe9
#5 0xffffffff8069e14a at namei+0x4cb
#6 0xffffffff806aca3c at kern_statat_vnhook+0x82
#7 0xffffffff806acbe1 at kern_statat+0x15
#8 0xffffffff806acc7f at lstat+0x22
#9 0xffffffff80922e9b at syscall+0x118
#10 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff00095e4768: tag syncer, type VNON
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
lock type syncer: EXCL by thread 0xffffff00069d3000 (pid 23)
#0 0xffffffff80603adb at __lockmgr_args+0x76e
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff806a9ea9 at sync_vnode+0x11c
#5 0xffffffff806aa12b at sched_sync+0x1c6
#6 0xffffffff805f1f8a at fork_exit+0x12a
#7 0xffffffff8090a47e at fork_trampoline+0xe

0xffffff00095e44f0: tag zfs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xffffff00094ef510 ref 0 pages 0
lock type zfs: SHARED (count 1)
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff806a8764 at vget+0x6c
#5 0xffffffff80696579 at cache_lookup+0x4dd
#6 0xffffffff806968bd at vfs_cache_lookup+0xad
#7 0xffffffff8097098b at VOP_LOOKUP_APV+0xba
#8 0xffffffff8069d1cf at lookup+0x45e
#9 0xffffffff8069e14a at namei+0x4cb
#10 0xffffffff806b411b at vn_open_cred+0x1d7
#11 0xffffffff806b2519 at kern_openat+0x163
#12 0xffffffff80922e9b at syscall+0x118
#13 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff0009541c58: tag zfs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
lock type zfs: EXCL by thread 0xffffff002c64bab0 (pid 1991)
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff806a8764 at vget+0x6c
#5 0xffffffff80696579 at cache_lookup+0x4dd
#6 0xffffffff806968bd at vfs_cache_lookup+0xad
#7 0xffffffff8097098b at VOP_LOOKUP_APV+0xba
#8 0xffffffff8069d1cf at lookup+0x45e
#9 0xffffffff8069e14a at namei+0x4cb
#10 0xffffffff806b4006 at vn_open_cred+0xc2
#11 0xffffffff806b2519 at kern_openat+0x163
#12 0xffffffff80922e9b at syscall+0x118
#13 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff0079423000: tag zfs, type VDIR
usecount 9, writecount 0, refcount 10 mountedhere 0
flags ()
v_object 0xffffff002c6b8948 ref 0 pages 0
lock type zfs: SHARED (count 1)
with exclusive waiters pending
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff8069ce5a at lookup+0xe9
#5 0xffffffff8069e14a at namei+0x4cb
#6 0xffffffff806b411b at vn_open_cred+0x1d7
#7 0xffffffff806b2519 at kern_openat+0x163
#8 0xffffffff80922e9b at syscall+0x118
#9 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff00763834f0: tag zfs, type VDIR
usecount 2, writecount 0, refcount 2 mountedhere 0
flags ()
lock type zfs: SHARED (count 1)
#0 0xffffffff80603886 at __lockmgr_args+0x519
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff8069ce5a at lookup+0xe9
#5 0xffffffff8069e14a at namei+0x4cb
#6 0xffffffff806aca3c at kern_statat_vnhook+0x82
#7 0xffffffff806acbe1 at kern_statat+0x15
#8 0xffffffff806acd36 at stat+0x22
#9 0xffffffff80922e9b at syscall+0x118
#10 0xffffffff8090a281 at Xfast_syscall+0xe1

0xffffff001e7e8768: tag zfs, type VREG
usecount 1, writecount 1, refcount 1 mountedhere 0
flags ()
v_object 0xffffff002360e360 ref 0 pages 0
lock type zfs: EXCL by thread 0xffffff0009618000 (pid 1590)
#0 0xffffffff80603adb at __lockmgr_args+0x76e
#1 0xffffffff80698b5a at vop_stdlock+0x39
#2 0xffffffff8096fb4f at VOP_LOCK1_APV+0x9b
#3 0xffffffff806b47cd at _vn_lock+0x5d
#4 0xffffffff806b5c8f at vn_write+0x1fb
#5 0xffffffff8065f5a9 at dofilewrite+0x7a
#6 0xffffffff80660c55 at kern_writev+0x52
#7 0xffffffff80660cd3 at writev+0x3a
#8 0xffffffff80922e9b at syscall+0x118
#9 0xffffffff8090a281 at Xfast_syscall+0xe1


----------------
attempt to execute minidump (this is from one of the previous
dumps, not the current one. The thread address pointed to
one of the cron threads that was locked @ /usr/src/sys/kern/vfs_subr.c:2091)

panic: deadlkres: possible deadlock detected for 0xffffff0006997000, blocked for 1800716 ticks

cpuid = 1
KDB: enter: panic
[thread pid 0 tid 100067]
Stopped at kdb_enter+0x3d: movq $0,0x6f923c(%rip)
db> call doadump()
Physical memeory: 2013 MB
Dumping 1784MB: Aborting dump due to I/O error.

** DUMP FAILED (ERROR 5) **
= 0
db>

----------------
attempt to execute full dump:

db> calld doadump()
Dumping 2046 MB (2 chunks)
Aborting dump due to I/O error.

** DUMP FAILED (ERROR 5) **
= 0
db>

----------------
kernel config:

cpu HAMMER
ident VAIO

# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.

# Use the following to compile in values accessible to the kernel
# through getenv() (or kenv(1) in userland). The format of the file
# is 'variable=value', see kenv(1)
#
# env "GENERIC.env"

makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols

options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options IPSEC # IP security
options IPSEC_NAT_T # IP security NAT traversal
options IPSEC_FILTERTUNNEL # Filter IPsec packets
options SCTP # Stream Control Transmission Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options MD_ROOT # MD is a potential root device
options NFSCL
options NFSCLIENT
options NFSSERVER
options NFSD
options NFSLOCKD # Network Lock Manager
options KGSSAPI
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options NTFS # NT File SYstem
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty)
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options P1003_1B_SEMAPHORES # POSIX-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
options AUDIT # Security event auditing
options MAC # TrustedBSD MAC Framework
options FLOWTABLE # per-cpu routing cache
#options KDTRACE_FRAME # Ensure frames are compiled in
#options KDTRACE_HOOKS # Kernel DTrace hooks

# Debugging for use in -current
options KDB # Enable kernel debugger support.
options DDB # Support DDB.
options GDB # Support remote GDB.
options INVARIANTS # Enable calls of extra sanity checking
options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS
options WITNESS # Enable checks to detect deadlocks and cycles
options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed
options DEADLKRES
options CAMDEBUG
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC

# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel

# CPU frequency control
device cpufreq

# Bus support.
device acpi
options ACPI_DEBUG
device pci

# Floppy drives
#device fdc

# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
device ahci

#device atacore
#device atacard
#device atapci
#device ataahci
#device ataintel

# SCSI Controllers
device ahc # AHA2940 and onboard AIC7xxx devices
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
device ahd # AHA39320/29320 and onboard AIC79xx devices
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
device amd # AMD 53C974 (Tekram DC-390(T))
device hptiop # Highpoint RocketRaid 3xxx series
device isp # Qlogic family
#device ispfw # Firmware for QLogic HBAs- normally a module
device mpt # LSI-Logic MPT-Fusion
#device ncr # NCR/Symbios Logic
device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
device trm # Tekram DC395U/UW/F DC315U adapters

device adv # Advansys SCSI adapters
device adw # Advansys wide SCSI adapters
device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
device bt # Buslogic/Mylex MultiMaster SCSI adapters

# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
device amr # AMI MegaRAID
device arcmsr # Areca SATA II RAID
#XXX it is not 64-bit clean, -scottl
#device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID
device ciss # Compaq Smart RAID 5*
device dpt # DPT Smartcache III, IV - See NOTES for options
device hptmv # Highpoint RocketRAID 182x
device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
device iir # Intel Integrated RAID
device ips # IBM (Adaptec) ServeRAID
device mly # Mylex AcceleRAID/eXtremeRAID
device twa # 3ware 9000 series PATA/SATA RAID

# RAID controllers
device aac # Adaptec FSA RAID
device aacp # SCSI passthrough for aac (requires CAM)
device ida # Compaq Smart RAID
device mfi # LSI MegaRAID SAS
device mlx # Mylex DAC960 family
#XXX pointer/int warnings
#device pst # Promise Supertrak SX6000
device twe # 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse

device kbdmux # keyboard multiplexer

device vga # VGA video card driver

device splash # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device sc

device agp # support several AGP chipsets

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
device cbb # cardbus (yenta) bridge
device pccard # PC Card (16-bit) bus
device cardbus # CardBus (32-bit) bus

# Serial (COM) ports
device uart # Generic UART driver

# Parallel port
device ppc
device ppbus # Parallel port bus (required)
device lpt # Printer
device plip # TCP/IP over parallel
device ppi # Parallel port interface device
#device vpo # Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to sio, uart and/or ppc drivers):
#device puc

# PCI Ethernet NICs.
device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 Gigabit Ethernet Family
device igb # Intel PRO/1000 PCIE Server Gigabit Family
device ixgbe # Intel PRO/10GbE PCIE Ethernet Family
device le # AMD Am7900 LANCE and Am79C9xx PCnet
device ti # Alteon Networks Tigon I/II gigabit Ethernet
device txp # 3Com 3cR990 (``Typhoon'')
device vx # 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device ae # Attansic/Atheros L2 FastEthernet
device age # Attansic/Atheros L1 Gigabit Ethernet
device alc # Atheros AR8131/AR8132 Ethernet
device ale # Atheros AR8121/AR8113/AR8114 Ethernet
device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device bfe # Broadcom BCM440x 10/100 Ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device dc # DEC/Intel 21143 and various workalikes
device et # Agere ET1310 10/100/Gigabit Ethernet
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device jme # JMicron JMC250 Gigabit/JMC260 Fast Ethernet
device lge # Level 1 LXT1001 gigabit Ethernet
device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet
device nfe # nVidia nForce MCP on-board Ethernet
device nge # NatSemi DP83820 gigabit Ethernet
#device nve # nVidia nForce MCP on-board Ethernet Networking
device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le')
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
device sf # Adaptec AIC-6915 (``Starfire'')
device sis # Silicon Integrated Systems SiS 900/SiS 7016
device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
device ste # Sundance ST201 (D-Link DFE-550TX)
device stge # Sundance/Tamarack TC9021 gigabit Ethernet
device tl # Texas Instruments ThunderLAN
device tx # SMC EtherPower II (83c170 ``EPIC'')
device vge # VIA VT612x gigabit Ethernet
device vr # VIA Rhine, Rhine II
device wb # Winbond W89C840F
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs. pccard NICs included.
device cs # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
device ex # Intel EtherExpress Pro/10 and Pro/10+
device ep # Etherlink III based cards
device fe # Fujitsu MB8696x based cards
device sn # SMC's 9000 series of Ethernet chips
device xe # Xircom pccard Ethernet

# Wireless NIC cards
device wlan # 802.11 support
options IEEE80211_DEBUG # enable debug msgs
options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
options IEEE80211_SUPPORT_MESH # enable 802.11s draft support
device wlan_wep # 802.11 WEP support
device wlan_ccmp # 802.11 CCMP support
device wlan_tkip # 802.11 TKIP support
device wlan_amrr # AMRR transmit rate control algorithm
device an # Aironet 4500/4800 802.11 wireless NICs.
device ath # Atheros pci/cardbus NIC's
device ath_hal # pci/cardbus chip support
options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
device ath_rate_sample # SampleRate tx rate control for ath
device ral # Ralink Technology RT2500 wireless NICs.
device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.

# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device tun # Packet tunnel.
device pty # BSD-style compatibility pseudo ttys
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
device crypto # for IP security

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter

# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
options USB_DEBUG
#device udbp # USB Double Bulk Pipe devices
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device ulpt # Printer
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
device urio # Diamond Rio 500 MP3 player
# USB Serial devices
device uark # Technologies ARK3116 based serial adapters
device ubsa # Belkin F5U103 and compatible serial adapters
device uftdi # For FTDI usb serial adapters
device uipaq # Some WinCE based devices
device uplcom # Prolific PL-2303 serial adapters
device uslcom # SI Labs CP2101/CP2102 serial adapters
device uvisor # Visor and Palm devices
device uvscom # USB serial support for DDI pocket's PHS
# USB Ethernet, requires miibus
device aue # ADMtek USB Ethernet
device axe # ASIX Electronics USB Ethernet
device cdce # Generic USB over Ethernet
device cue # CATC USB Ethernet
device kue # Kawasaki LSI USB Ethernet
device rue # RealTek RTL8150 USB Ethernet
device udav # Davicom DM9601E USB
# USB Wireless
device rum # Ralink Technology RT2501USB wireless NICs
device uath # Atheros AR5523 wireless NICs
device ural # Ralink Technology RT2500USB wireless NICs
device zyd # ZyDAS zb1211/zb1211b wireless NICs

# FireWire support
device firewire # FireWire bus code
device sbp # SCSI over FireWire (Requires scbus and da)
device fwe # Ethernet over FireWire (non-standard!)
device fwip # IP over FireWire (RFC 2734,3146)
device dcons # Dumb console driver
device dcons_crom # Configuration ROM for dcons

----------------
dmesg since boot:

[dcons connected]
Consoles: internal video/keyboard dumb console port
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS 638kB/2094592kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(sa...@radziecki.saper.info, Wed Jan 20 22:26:53 CET 2010)
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ Loading /boot/defaults/loader.conf
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/kernel text=0x9acf28 | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /!
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ data=0x13ac18+0x279ac0 | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - syms=[0x8+0xfbd60\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / +0x8+0xe35e5- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /!
- \ | / - \ | / - ]
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/zfs.ko / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - size 0x18ee60 at 0x1141000
loading required module 'opensolaris'
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/opensolaris.ko size 0x38a8 at 0x12d0000
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/linux.ko / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / size 0x431f8 at 0x12d4000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/if_wpi.ko | / - \ | / - \ | / - \ | / - \ | / - \ | size 0x12678 at 0x1318000
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - /boot/kernel/snd_hda.ko \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - size 0x24668 at 0x132b000
loading required module 'sound'
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/sound.ko / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / size 0x86030 at 0x1350000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/acpi_video.ko - \ | / - \ | / - \ | / size 0x6968 at 0x13d7000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/wpifw.ko / - \ | / - \ | / - \ | / - \ | / size 0x25588 at 0x13de000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/acpi_sony.ko size 0x1fb0 at 0x1404000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/smbus.ko size 0x2498 at 0x1406000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/smb.ko size 0x2dc8 at 0x1409000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/iicsmb.ko size 0x3cf0 at 0x140c000
loading required module 'iicbus'
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/iicbus.ko size 0x43b0 at 0x1410000
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / /boot/kernel/ichsmb.ko - \ | / - \ | / - \ size 0x59c8 at 0x1415000
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/ng_ubt.ko | / - \ | / - \ | / - \ | size 0x82c8 at 0x141b000
loading required module 'netgraph'
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/netgraph.ko / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ size 0x15b48 at 0x1424000
loading required module 'ng_hci'
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/ng_hci.ko | / - \ | / - \ | / - \ | / - \ | / - \ | size 0x12f08 at 0x143a000
loading required module 'ng_bluetooth'
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/ng_bluetooth.ko size 0x2d20 at 0x144d000
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /boot/kernel/atapicam.ko / - \ | / - \ | / - \ | / - size 0x6de8 at 0x1450000
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - /boot/kernel/cmx.ko \ | / - \ | / - size 0x4438 at 0x1457000
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/modules/nvidia.ko | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /!
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \!
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -!
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | size 0xd3d630 at 0x145c000
/ - \ | / - \ | / - \ | / - \ | / - \ | /
Hit [Enter] to boot immediately, or any other key for command prompt.

Booting [/boot/kernel/kernel]...
cryptosoft0: <software crypto> on motherboard
acpi0: <Sony> on motherboard
acpi0: [ITHREAD]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x17> port 0x62,0x66 on acpi0
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 0xcd000000-0xcdffffff,0xd0000000-0xdfffffff,0xce000000-0xceffffff irq 16 at device 0.0 on pci1
nvidia0: <GeForce Go 7400> on vgapci0
vgapci0: child nvidia0 requested pci_enable_busmaster
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
nvidia0: [GIANT-LOCKED]
nvidia0: [ITHREAD]
hdac0: <Intel 82801G High Definition Audio Controller> mem 0xf8500000-0xf8503fff irq 21 at device 27.0 on pci0
hdac0: HDA Driver Revision: 20100122_0141
hdac0: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci6: <ACPI PCI bus> on pcib3
wpi0: <Intel(R) PRO/Wireless 3945ABG> mem 0xf8100000-0xf8100fff irq 17 at device 0.0 on pci6
wpi0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci7: <ACPI PCI bus> on pcib4
mskc0: <Marvell Yukon 88E8036 Fast Ethernet> port 0x3000-0x30ff mem 0xf8000000-0xf8003fff irq 18 at device 0.0 on pci7
msk0: <Marvell Technology Group Ltd. Yukon FE Id 0xb7 Rev 0x01> on mskc0
msk0: disabling jumbo frame support
msk0: Ethernet address: 00:1a:80:55:25:5e
miibus0: <MII bus> on msk0
e1000phy0: <Marvell 88E3082 10/100 Fast Ethernet PHY> PHY 0 on miibus0
e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
mskc0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci8: <ACPI PCI bus> on pcib5
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0x1800-0x181f irq 19 at device 29.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801G (ICH7) USB controller USB-A> on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801G (ICH7) USB controller USB-B> on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0x1840-0x185f irq 19 at device 29.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801G (ICH7) USB controller USB-C> on uhci2
uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0x1860-0x187f irq 19 at device 29.3 on pci0
uhci3: [ITHREAD]
usbus3: <Intel 82801G (ICH7) USB controller USB-D> on uhci3
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xf8504000-0xf85043ff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib6
cbb0: <PCI-CardBus Bridge> at device 4.0 on pci9
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
cbb0: [FILTER]
fwohci0: <1394 Open Host Controller Interface> mem 0xf8205000-0xf82057ff,0xf8200000-0xf8203fff irq 21 at device 4.1 on pci9
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 08:00:46:03:02:91:b4:4d
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x41000
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 0a:00:46:91:b4:4d
fwe0: Ethernet address: 0a:00:46:91:b4:4d
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 08:00:46:03:02:91:b4:4d @ 0xfffe00000000, S400, maxrec 2048
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
pci9: <mass storage> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ahci0: <Intel ICH7M AHCI SATA controller> port 0x18c8-0x18cf,0x18ac-0x18af,0x18c0-0x18c7,0x18a8-0x18ab,0x18b0-0x18bf mem 0xf8504400-0xf85047ff irq 22 at device 31.2 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.10 with 4 1.5Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 2 on ahci0
ahcich1: [ITHREAD]
ichsmb0: <Intel 82801GB (ICH7) SMBus controller> port 0x18e0-0x18ff at device 31.3 on pci0
ichsmb0: can't get IRQ
device_attach: ichsmb0 attach returned 6
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
acpi_tz2: <Thermal Zone> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
acpi_sony0: <Sony notebook controller> on acpi0
acpi_sony0: PID 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model GlidePoint, device ID 0
battery0: <ACPI Control Method Battery> on acpi0
acpi_acad0: <AC Adapter> on acpi0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xdc000-0xdffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
battery0: battery initialization start
acpi_acad0: acline initialization start
acpi_acad0: On Line
acpi_acad0: acline initialization done, tried 1 times
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
acd0: DVDR <MATSHITADVD-RAM UJ-852S/1.01> at ata0-master UDMA33
hdac0: HDA Codec #0: Sigmatel STAC9872AK
hdac0: HDA Codec #1: Conexant (Unknown)
pcm0: <HDA Sigmatel STAC9872AK PCM #0 Analog> at cad 0 nid 1 on hdac0
pcm1: <HDA Sigmatel STAC9872AK PCM #1 Analog> at cad 0 nid 1 on hdac0
cmx0: <OMNIKEY CardMan 4040> at port 0x100-0x107 irq 20 function 0 config 35 on pccard0
cmx0: attached
Expensive timeout(9) function: 0xffffffff80653cff(0xffffff0003757000) 0.036358531 s
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
acd0: FAILURE - READ_BIG ILLEGAL REQUEST asc=0x64 ascq=0x00
acd0: FAILURE - READ_BIG ILLEGAL REQUEST asc=0x64 ascq=0x00
uhub4: 8 ports with 8 removable, self powered
acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00
ugen4.2: <USBest Technology> at usbus4
umass0: <USBest Technology USB Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on usbus4
umass0: SCSI over Bulk-Only; quirks = 0x0000
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <FUJITSU MHW2120BH 00000012> ATA-8cd0 at ata0 bus 0 scbus3 target 0 lun 0
cd0: <MATSHITA DVD-RAM UJ-852S 1.01> Removable CD-ROM SCSI-0 device
cd0: 33.000MB/s transfers
cd0: cd present [64816 x 2048 byte records]
SATA 1.x device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
acd0: FAILURE - READ_BIG ILLEGAL REQUEST asc=0x64 ascq=0x00
(cd0:ata0:0:0:0): READ(10). CDB: 28 0 0 0 fd 2f 0 0 1 0
(cd0:ata0:0:0:0): CAM status: SCSI Status Error
(cd0:ata0:0:0:0): SCSI status: Check Condition
(cd0:ata0:0:0:0): SCSI sense: ILLEGAL REQUEST csi:0,1d,0,0 asc:64,0 (Illegal mode for this track)
(cd0:ata0:0:0:0): cddone: got error 0x6 back
umass0:4:0:-1: Attached to scbus4
da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
da0: <Ut163 USB2FlashStorage 0.00> Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 250MB (512000 512 byte sectors: 64H 32S/T 250C)
GEOM: ada0: partition 3 does not start on a track boundary.
GEOM: ada0: partition 3 does not end on a track boundary.
GEOM: ada0: partition 2 does not start on a track boundary.
GEOM: ada0: partition 2 does not end on a track boundary.
ugen4.3: <Sony> at usbus4
umass1: <Sony UMH-U09, class 0/0, rev 2.00/4.82, addr 3> on usbus4
umass1: SCSI over Bulk-Only; quirks = 0x0000
GEOM: da0: partition 1 does not start on a track boundary.
GEOM: da0: partition 1 does not end on a track boundary.
Root mount waiting for: usbus4
umass1:5:1:-1: Attached to scbus5
Root mount waiting for: usbus4
(probe0:umass-sim1:1:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
da1 at umass-sim1 bus 1 scbus5 target 0 lun 0
da1: <Sony USB HS-CARD 4.82> Removable Direct Access SCSI-0 device
da1: 40.000MB/s transfers
da1: Attempt to query device size failed: NOT READY, Medium not present
ugen4.4: <vendor 0x05ca> at usbus4
Trying to mount root from zfs:tank
ugen3.2: <ALPS> at usbus3
ubt0: <ALPS UGX, class 224/1, rev 2.00/19.15, addr 2> on usbus3
WARNING: /crash was not properly dismounted
wlan0: Ethernet address: 00:1b:77:bc:a5:44
wpi0: need multicast update callback
msk0: link state changed to DOWN
WARNING: attempt to domain_add(bluetooth) after domainfinalize()
WARNING: attempt to domain_add(netgraph) after domainfinalize()
lock order reversal:
1st 0xffffff80003ad018 wpi0_com_lock (wpi0_com_lock) @ /usr/src/sys/modules/wpi/../../dev/wpi/if_wpi.c:1292
2nd 0xffffff00034f5010 wpi0 (network driver) @ /usr/src/sys/modules/wpi/../../dev/wpi/if_wpi.c:2069
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x49
witness_checkorder() at witness_checkorder+0x7ea
_mtx_lock_flags() at _mtx_lock_flags+0x68
wpi_raw_xmit() at wpi_raw_xmit+0x68
ieee80211_send_mgmt() at ieee80211_send_mgmt+0x49c
sta_newstate() at sta_newstate+0x42b
wpi_newstate() at wpi_newstate+0xe3
ieee80211_newstate_cb() at ieee80211_newstate_cb+0xac
taskqueue_run() at taskqueue_run+0x90
taskqueue_thread_loop() at taskqueue_thread_loop+0x3f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000183d30, rbp = 0 ---
wlan0: link state changed to UP
wpi0: need multicast update callback
wpi0: need multicast update callback
wpi0: need multicast update callback
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled
lock order reversal:
1st 0xffffff0009adf310 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1058
2nd 0xffffff0009af8cf0 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2091
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x49
witness_checkorder() at witness_checkorder+0x7ea
__lockmgr_args() at __lockmgr_args+0xd5c
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x5d
vget() at vget+0x6c
devfs_allocv() at devfs_allocv+0xee
devfs_root() at devfs_root+0x41
vfs_donmount() at vfs_donmount+0xfa8
nmount() at nmount+0x74
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007ac37c, rsp = 0x7fffffffdd28, rbp = 0x800a06048 ---
wpi0: need multicast update callback
battery0: battery initialization failed, giving up
kqemu version 0x00010400
kqemu: KQEMU installed, max_locked_mem=1030772kB.

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login: fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=2, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
firewire0: fw_explore_node: Pre 1394a-2000 detected
firewire0: New S400 device ID:001106005350d195


FreeBSD/amd64 (radziecki.saper.info) (dcons)

login:

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login: acpi_lid0: Lid closed
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=3, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=4, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=5, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=6, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
firewire0: fw_attach_dev:Removing missing device ID:001106005350d195
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000001, SelfID Count=7, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1) (me)
firewire0: bus manager 1
firewire0: fw_explore_node: Pre 1394a-2000 detected
firewire0: New S400 device ID:001106005350d195
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=8, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
fwohci0: too many cycle lost, no cycle master presents?


FreeBSD/amd64 (radziecki.saper.info) (dcons)

login: fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=9, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1 cable IRM irm(1)
fwohci0: too many cycle lost, no cycle master presents?
acpi_lid0: Lid opened
acpi_lid0: Lid closed
acpi_lid0: Lid opened
~.

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login:

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login:

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login:

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login:

FreeBSD/amd64 (radziecki.saper.info) (dcons)

login: ~.[dcons disconnected (read buffer failed)]

----------------
alltrace:


[dcons connected]


Tracing command sendmail pid 3499 tid 100166 td 0xffffff0009747390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
__lockmgr_args() at __lockmgr_args+0x77e
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x5d
cache_lookup() at cache_lookup+0x6a0
vfs_cache_lookup() at vfs_cache_lookup+0xad
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
vn_open_cred() at vn_open_cred+0x1d7
kern_openat() at kern_openat+0x163
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800ec012c, rsp = 0x7fffffffc238, rbp = 0x3 ---

Tracing command sendmail pid 3498 tid 100158 td 0xffffff0009615390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
dbuf_read() at dbuf_read+0x34e
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
kern_statat_vnhook() at kern_statat_vnhook+0x82
kern_statat() at kern_statat+0x15
stat() at stat+0x22
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x800ec00cc, rsp = 0x7fffffffbec8, rbp = 0 ---

Tracing command sendmail pid 3497 tid 100183 td 0xffffff000974eab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
__lockmgr_args() at __lockmgr_args+0x77e
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x5d
cache_lookup() at cache_lookup+0x6a0
vfs_cache_lookup() at vfs_cache_lookup+0xad
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
vn_open_cred() at vn_open_cred+0x1d7
kern_openat() at kern_openat+0x163
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800ec012c, rsp = 0x7fffffffc238, rbp = 0x3 ---

Tracing command sendmail pid 3496 tid 100235 td 0xffffff004fbd1720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
dbuf_read() at dbuf_read+0x34e
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
kern_statat_vnhook() at kern_statat_vnhook+0x82
kern_statat() at kern_statat+0x15
stat() at stat+0x22
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x800ec00cc, rsp = 0x7fffffffbec8, rbp = 0 ---

Tracing command sendmail pid 3495 tid 100226 td 0xffffff004fbd2ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_cursor_retrieve() at zap_cursor_retrieve+0x190
zfs_freebsd_readdir() at zfs_freebsd_readdir+0x289
VOP_READDIR_APV() at VOP_READDIR_APV+0xb5
kern_getdirentries() at kern_getdirentries+0x122
getdirentries() at getdirentries+0x23
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (196, FreeBSD ELF64, getdirentries), rip = 0x800e52e2c, rsp = 0x7fffffffc2b8, rbp = 0x1 ---

Tracing command sendmail pid 3494 tid 100224 td 0xffffff004fbac390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
kern_statat_vnhook() at kern_statat_vnhook+0x82
kern_statat() at kern_statat+0x15
stat() at stat+0x22
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x800ec00cc, rsp = 0x7fffffffbec8, rbp = 0 ---

Tracing command cat pid 3210 tid 100222 td 0xffffff002c6fbab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
pipe_read() at pipe_read+0x495
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80073022c, rsp = 0x7fffffffea38, rbp = 0 ---

Tracing command sh pid 3208 tid 100189 td 0xffffff002c6fb390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffdf68, rbp = 0x7fffffffe128 ---

Tracing command find pid 3207 tid 100190 td 0xffffff002c6fb000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dnode_hold_impl() at dnode_hold_impl+0xd8
dmu_bonus_hold() at dmu_bonus_hold+0x23
zfs_zget() at zfs_zget+0x213
zfs_dirent_lock() at zfs_dirent_lock+0x49b
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
kern_statat_vnhook() at kern_statat_vnhook+0x82
kern_statat() at kern_statat+0x15
lstat() at lstat+0x22
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x8007290ac, rsp = 0x7fffffffe678, rbp = 0x800a6ddd0 ---

Tracing command sh pid 3203 tid 100194 td 0xffffff002c64b000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe828, rbp = 0 ---

Tracing command mail pid 3202 tid 100162 td 0xffffff0009613390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
pipe_read() at pipe_read+0x495
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80073f22c, rsp = 0x7fffffffdcf8, rbp = 0x400 ---

Tracing command sh pid 3201 tid 100230 td 0xffffff004fbc3ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe3f8, rbp = 0x7fffffffe5b8 ---

Tracing command sh pid 3200 tid 100227 td 0xffffff004fbd2720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe0f8, rbp = 0x7fffffffe2b8 ---

Tracing command sh pid 3193 tid 100171 td 0xffffff0009745000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe8c8, rbp = 0 ---

Tracing command sh pid 3192 tid 100199 td 0xffffff002c6f9ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe6b8, rbp = 0x7fffffffe878 ---

Tracing command mail pid 2850 tid 100119 td 0xffffff00093c6000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
pipe_read() at pipe_read+0x495
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80073f22c, rsp = 0x7fffffffdd18, rbp = 0x400 ---

Tracing command sh pid 2849 tid 100173 td 0xffffff0009753720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe438, rbp = 0x7fffffffe5f8 ---

Tracing command sh pid 2848 tid 100129 td 0xffffff00093c5390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe138, rbp = 0x7fffffffe2f8 ---

Tracing command sh pid 2841 tid 100154 td 0xffffff0009616390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009304fc, rsp = 0x7fffffffe908, rbp = 0 ---

Tracing command csh pid 2804 tid 100157 td 0xffffff0009615720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigsuspend() at kern_sigsuspend+0x93
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip = 0x8009475fc, rsp = 0x7fffffffe628, rbp = 0x800c5d300 ---

Tracing command getty pid 2803 tid 100153 td 0xffffff0009616720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2802 tid 100108 td 0xffffff00093ca390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2801 tid 100164 td 0xffffff00095a5ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2800 tid 100201 td 0xffffff002c6f9390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2799 tid 100203 td 0xffffff0009747ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2798 tid 100193 td 0xffffff002c64b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 2797 tid 100120 td 0xffffff0006a75ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
tty_wait() at tty_wait+0x41
ttydisc_read() at ttydisc_read+0x225
ttydev_read() at ttydev_read+0x89
devfs_read_f() at devfs_read_f+0x7f
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084222c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command login pid 2796 tid 100186 td 0xffffff000974e000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
vn_open_cred() at vn_open_cred+0x1d7
kern_openat() at kern_openat+0x163
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x80050fe1c, rsp = 0x7fffffffe348, rbp = 0x4010e1 ---

Tracing command login pid 2795 tid 100197 td 0xffffff002c69b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e64fc, rsp = 0x7fffffffec48, rbp = 0 ---

Tracing command inetd pid 2765 tid 100109 td 0xffffff00093ca000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a571ac, rsp = 0x7fffffffddc8, rbp = 0x7878787878787878 ---

Tracing command nfscbd pid 2746 tid 100216 td 0xffffff0079d09390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069b1dc, rsp = 0x7fffffffe9c8, rbp = 0x7fffffffeb60 ---

Tracing command nfscbd pid 2746 tid 100215 td 0xffffff00093ca720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069b1dc, rsp = 0x7fffffffe9c8, rbp = 0x7fffffffeb60 ---

Tracing command nfscbd pid 2746 tid 100214 td 0xffffff00093caab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069b1dc, rsp = 0x7fffffffe9c8, rbp = 0x7fffffffeb60 ---

Tracing command nfscbd pid 2746 tid 100195 td 0xffffff002c69bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_run() at svc_run+0x94
nfscbd_nfsd() at nfscbd_nfsd+0xe8
nfssvc_nfscl() at nfssvc_nfscl+0x60
nfssvc() at nfssvc+0x75
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x80069b1dc, rsp = 0x7fffffffe9c8, rbp = 0x7fffffffeb60 ---

Tracing command nfscbd pid 2745 tid 100206 td 0xffffff002c67d000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_accept() at kern_accept+0x1a7
accept() at accept+0x61
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (30, FreeBSD ELF64, accept), rip = 0x8006d08ac, rsp = 0x7fffffffe9c8, rbp = 0 ---

Tracing command gdomap pid 2670 tid 100152 td 0xffffff0009616ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80164c1ac, rsp = 0x7fffffffd018, rbp = 0x400 ---

Tracing command cron pid 2560 tid 100196 td 0xffffff002c69b720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
kern_statat_vnhook() at kern_statat_vnhook+0x82
kern_statat() at kern_statat+0x15
stat() at stat+0x22
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x80093f0cc, rsp = 0x7fffffffe7e8, rbp = 0 ---

Tracing command sendmail pid 2553 tid 100205 td 0xffffff002c67d390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigsuspend() at kern_sigsuspend+0x93
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip = 0x800e395fc, rsp = 0x7fffffffcd88, rbp = 0x1 ---

Tracing command sendmail pid 2547 tid 100204 td 0xffffff0009747720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800ece1ac, rsp = 0x7fffffffc208, rbp = 0x7fffffffc2a0 ---

Tracing command cupsd pid 2499 tid 100125 td 0xffffff00093c5720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_kevent() at kern_kevent+0x333
kevent() at kevent+0x85
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x8011a8dec, rsp = 0x7fffffffe7e8, rbp = 0x1 ---

Tracing command pcscd pid 2188 tid 100213 td 0xffffff002c73c390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
pipe_read() at pipe_read+0x495
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x52
read() at read+0x4e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095f22c, rsp = 0x7fffff9fdf68, rbp = 0x800c0ac88 ---

Tracing command pcscd pid 2188 tid 100211 td 0xffffff002c73cab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x33e
kern_nanosleep() at kern_nanosleep+0x10b
nanosleep() at nanosleep+0x61
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800945f5c, rsp = 0x7fffffbfef28, rbp = 0 ---

Tracing command pcscd pid 2188 tid 100122 td 0xffffff00093c5ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80095f1ac, rsp = 0x7fffffffeac8, rbp = 0x7 ---

Tracing command dkim-filter pid 2170 tid 100210 td 0xffffff002c738000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigtimedwait() at kern_sigtimedwait+0x4dd
sigwait() at sigwait+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800dc5f1c, rsp = 0x7fffff7fcf38, rbp = 0x80100ac80 ---

Tracing command dkim-filter pid 2170 tid 100209 td 0xffffff002c738390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigtimedwait() at kern_sigtimedwait+0x4dd
sigwait() at sigwait+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800dc5f1c, rsp = 0x7fffff9fdf48, rbp = 0x80100ae40 ---

Tracing command dkim-filter pid 2170 tid 100208 td 0xffffff002c738720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800e5b1ac, rsp = 0x7fffffbfeca8, rbp = 0x8 ---

Tracing command dkim-filter pid 2170 tid 100169 td 0xffffff0009745720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
poll() at poll+0x3f4
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800e0ad3c, rsp = 0x7ffffffee148, rbp = 0 ---

Tracing command racoon pid 2091 tid 100188 td 0xffffff0009618720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800eac1ac, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffecb0 ---

Tracing command privoxy pid 2083 tid 100128 td 0xffffff0009503000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_accept() at kern_accept+0x1a7
accept() at accept+0x61
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800b6f8ac, rsp = 0x7fffffffd658, rbp = 0x7fffffffd6d0 ---

Tracing command ifdhandler pid 2071 tid 100168 td 0xffffff0009745ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
poll() at poll+0x3f4
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x80091cd3c, rsp = 0x7fffffffd378, rbp = 0x7fffffffe3c0 ---

Tracing command rarpd pid 2003 tid 100185 td 0xffffff000974e390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007311ac, rsp = 0x7fffffffea88, rbp = 0 ---

Tracing command ntpd pid 1991 tid 100191 td 0xffffff002c64bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
vn_open_cred() at vn_open_cred+0xc2
kern_openat() at kern_openat+0x163
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800d2c12c, rsp = 0x7fffffffeaa8, rbp = 0x8 ---

Tracing command hcsecd pid 1946 tid 100181 td 0xffffff0009753390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
soreceive_generic() at soreceive_generic+0x1005
kern_recvit() at kern_recvit+0x17b
recvit() at recvit+0x17
recvfrom() at recvfrom+0x7b
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (29, FreeBSD ELF64, recvfrom), rip = 0x8007f5abc, rsp = 0x7fffffffea38, rbp = 0x3 ---

Tracing command bootparamd pid 1939 tid 100110 td 0xffffff00093c8ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007301ac, rsp = 0x7fffffffeaf8, rbp = 0x800879a80 ---

Tracing command tor pid 1911 tid 100121 td 0xffffff000950a390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x33e
kern_kevent() at kern_kevent+0x333
kevent() at kevent+0x85
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800ee6dec, rsp = 0x7fffffffe9d8, rbp = 0x801241000 ---

Tracing command rpc.statd pid 1837 tid 100118 td 0xffffff00093c6390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80083a1ac, rsp = 0x7fffffffeb58, rbp = 0x800983a80 ---

Tracing command nfsd pid 1830 tid 100180 td 0xffffff0009510000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069e1dc, rsp = 0x7fffffffe6d8, rbp = 0x5 ---

Tracing command nfsd pid 1830 tid 100179 td 0xffffff0009510390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069e1dc, rsp = 0x7fffffffe6d8, rbp = 0x5 ---

Tracing command nfsd pid 1830 tid 100178 td 0xffffff0009510720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_thread_start() at svc_thread_start+0xb
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0xc, rip = 0x80069e1dc, rsp = 0x7fffffffe6d8, rbp = 0x5 ---

Tracing command nfsd pid 1830 tid 100117 td 0xffffff0006a79000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
svc_run_internal() at svc_run_internal+0x803
svc_run() at svc_run+0x94
nfssvc_nfsd() at nfssvc_nfsd+0x139
nfssvc_nfsserver() at nfssvc_nfsserver+0x53
nfssvc() at nfssvc+0x75
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x80069e1dc, rsp = 0x7fffffffe6d8, rbp = 0x5 ---

Tracing command nfsd pid 1829 tid 100155 td 0xffffff0009616000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007321ac, rsp = 0x7fffffffe978, rbp = 0x7fffffffec50 ---

Tracing command mountd pid 1827 tid 100130 td 0xffffff0009511ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8008451ac, rsp = 0x7fffffffec48, rbp = 0x7fffffffed40 ---

Tracing command gssd pid 1794 tid 100116 td 0xffffff00093c6720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80083b1ac, rsp = 0x7fffffffea28, rbp = 0x800984a80 ---

Tracing command nfsuserd pid 1788 tid 100127 td 0xffffff0009503390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007301ac, rsp = 0x7fffffffe9f8, rbp = 0x800879a80 ---

Tracing command nfsuserd pid 1787 tid 100163 td 0xffffff0009613000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007301ac, rsp = 0x7fffffffe9f8, rbp = 0x800879a80 ---

Tracing command nfsuserd pid 1786 tid 100170 td 0xffffff0009745390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007301ac, rsp = 0x7fffffffe9f8, rbp = 0x800879a80 ---

Tracing command nfsuserd pid 1785 tid 100111 td 0xffffff00093c8720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8007301ac, rsp = 0x7fffffffe9f8, rbp = 0x800879a80 ---

Tracing command nfsuserd pid 1784 tid 100161 td 0xffffff0009613720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigsuspend() at kern_sigsuspend+0x93
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip = 0x80069b5fc, rsp = 0x7fffffffeb68, rbp = 0x4 ---

Tracing command amd pid 1731 tid 100159 td 0xffffff0009615000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
dbuf_read() at dbuf_read+0x34e
dmu_buf_hold() at dmu_buf_hold+0xb7
zap_lockdir() at zap_lockdir+0x48
zap_lookup_norm() at zap_lookup_norm+0x3a
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4f8
zfs_dirlook() at zfs_dirlook+0x5a
zfs_lookup() at zfs_lookup+0x1de
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x7a
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xb1
vfs_cache_lookup() at vfs_cache_lookup+0xda
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xba
lookup() at lookup+0x45e
namei() at namei+0x4cb
vn_open_cred() at vn_open_cred+0x1d7
kern_openat() at kern_openat+0x163
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x80084c12c, rsp = 0x7fffffffe848, rbp = 0x4 ---

Tracing command rpcbind pid 1700 tid 100112 td 0xffffff00093c8390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
poll() at poll+0x3f4
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8008fed3c, rsp = 0x7fffffffcad8, rbp = 0x800c120c0 ---

Tracing command named pid 1683 tid 100177 td 0xffffff0009510ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_kevent() at kern_kevent+0x333
kevent() at kevent+0x85
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c5adec, rsp = 0x7fffff5fbe08, rbp = 0x14 ---

Tracing command named pid 1683 tid 100176 td 0xffffff0009511000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x33e
do_cv_wait() at do_cv_wait+0x60c
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800a83adc, rsp = 0x7fffff7fcdb8, rbp = 0x800e0ac80 ---

Tracing command named pid 1683 tid 100175 td 0xffffff0009511390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
do_cv_wait() at do_cv_wait+0x7fd
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800a83adc, rsp = 0x7fffff9fdeb8, rbp = 0x800e0ae40 ---

Tracing command named pid 1683 tid 100174 td 0xffffff0009511720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
do_cv_wait() at do_cv_wait+0x7fd
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800a83adc, rsp = 0x7fffffbfeeb8, rbp = 0x800e9d1c0 ---

Tracing command named pid 1683 tid 100160 td 0xffffff0009613ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigtimedwait() at kern_sigtimedwait+0x4dd
sigwait() at sigwait+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800bddf1c, rsp = 0x7fffffffeb88, rbp = 0x800e041c0 ---

Tracing command syslogd pid 1590 tid 100151 td 0xffffff0009618000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dmu_tx_check_ioerr() at dmu_tx_check_ioerr+0x84
dmu_tx_count_write() at dmu_tx_count_write+0x4c
dmu_tx_hold_write() at dmu_tx_hold_write+0x43
zfs_freebsd_write() at zfs_freebsd_write+0x3e6
VOP_WRITE_APV() at VOP_WRITE_APV+0x129
vn_write() at vn_write+0x281
dofilewrite() at dofilewrite+0x7a
kern_writev() at kern_writev+0x52
writev() at writev+0x3a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (121, FreeBSD ELF64, writev), rip = 0x80082f39c, rsp = 0x7fffffffd398, rbp = 0 ---

Tracing command dhclient pid 1363 tid 100115 td 0xffffff00093c6ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x177
seltdwait() at seltdwait+0x4f
poll() at poll+0x3f4
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006eed3c, rsp = 0x7fffffffece8, rbp = 0x2 ---

Tracing command dhclient pid 1260 tid 100165 td 0xffffff00095a5720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
poll() at poll+0x3f4
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006eed3c, rsp = 0x7fffffffece8, rbp = 0x5 ---

Tracing command devd pid 1209 tid 100126 td 0xffffff0009503720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x43da6c, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 ---

Tracing command wpa_supplicant pid 545 tid 100124 td 0xffffff0009503ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x169
seltdwait() at seltdwait+0xa5
kern_select() at kern_select+0x641
select() at select+0x54
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800c7d1ac, rsp = 0x7fffffffeb38, rbp = 0x800e07080 ---

Tracing command adjkerntz pid 133 tid 100114 td 0xffffff0006a79390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_sigsuspend() at kern_sigsuspend+0x93
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip = 0x80069a5fc, rsp = 0x7fffffffec58, rbp = 0x7fffffffecd0 ---

Tracing command flowcleaner pid 26 tid 100076 td 0xffffff00069d2390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_cv_timedwait() at _cv_timedwait+0x177
flowtable_cleaner() at flowtable_cleaner+0xdc
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dcb9d30, rbp = 0 ---

Tracing command softdepflush pid 25 tid 100075 td 0xffffff00069d2720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
softdep_flush() at softdep_flush+0x253
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dcb4d30, rbp = 0 ---

Tracing command vnlru pid 24 tid 100074 td 0xffffff00069d2ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
vnlru_proc() at vnlru_proc+0x62e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dcafd30, rbp = 0 ---

Tracing command syncer pid 23 tid 100073 td 0xffffff00069d3000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
zil_commit() at zil_commit+0x3de
zfs_sync() at zfs_sync+0xa5
sync_fsync() at sync_fsync+0x134
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5
sync_vnode() at sync_vnode+0x143
sched_sync() at sched_sync+0x1c6
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dcaad30, rbp = 0 ---

Tracing command bufdaemon pid 22 tid 100072 td 0xffffff00069d3390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
buf_daemon() at buf_daemon+0x16a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dca5d30, rbp = 0 ---

Tracing command pagezero pid 21 tid 100071 td 0xffffff00069d3720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
vm_pagezero() at vm_pagezero+0x71
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dca0d30, rbp = 0 ---

Tracing command vmdaemon pid 20 tid 100070 td 0xffffff00069d3ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc9bd30, rbp = 0 ---

Tracing command pagedaemon pid 19 tid 100069 td 0xffffff00069d6000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
vm_pageout() at vm_pageout+0x89f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc96d30, rbp = 0 ---

Tracing command xpt_thrd pid 18 tid 100068 td 0xffffff00037b8000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
xpt_scanner_thread() at xpt_scanner_thread+0xd5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc61d30, rbp = 0 ---

Tracing command zfskern pid 9 tid 100103 td 0xffffff002ad0d390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
vdev_geom_worker() at vdev_geom_worker+0xed
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040039d30, rbp = 0 ---

Tracing command zfskern pid 9 tid 100106 td 0xffffff00069d1390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x3d6
dbuf_findbp() at dbuf_findbp+0xe5
dbuf_hold_impl() at dbuf_hold_impl+0xc2
dbuf_hold() at dbuf_hold+0x1b
dmu_buf_hold() at dmu_buf_hold+0x75
bplist_iterate() at bplist_iterate+0xa1
spa_sync() at spa_sync+0x261
txg_sync_thread() at txg_sync_thread+0x285
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040059d30, rbp = 0 ---

Tracing command zfskern pid 9 tid 100105 td 0xffffff00069d1720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
txg_thread_wait() at txg_thread_wait+0x7d
txg_quiesce_thread() at txg_quiesce_thread+0xb5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040055d30, rbp = 0 ---

Tracing command zfskern pid 9 tid 100066 td 0xffffff00037b8720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_cv_timedwait() at _cv_timedwait+0x177
l2arc_feed_thread() at l2arc_feed_thread+0x143
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc57d30, rbp = 0 ---

Tracing command zfskern pid 9 tid 100065 td 0xffffff00037b8ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_cv_timedwait() at _cv_timedwait+0x177
arc_reclaim_thread() at arc_reclaim_thread+0x2a5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc52d30, rbp = 0 ---

Tracing command acpi_cooling0 pid 17 tid 100062 td 0xffffff00037b9720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
acpi_tz_cooling_thread() at acpi_tz_cooling_thread+0x11a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc3fd30, rbp = 0 ---

Tracing command acpi_thermal pid 16 tid 100061 td 0xffffff00037b9ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
acpi_tz_thread() at acpi_tz_thread+0x200
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc3ad30, rbp = 0 ---

Tracing command fw0_probe pid 8 tid 100058 td 0xffffff00037bc720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
fw_bus_probe_thread() at fw_bus_probe_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bd00d30, rbp = 0 ---

Tracing command cbb0 event thread pid 7 tid 100055 td 0xffffff0003757720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
cbb_event_thread() at cbb_event_thread+0x1d3
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bceed30, rbp = 0 ---

Tracing command usb pid 15 tid 100054 td 0xffffff0003757ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bce9d30, rbp = 0 ---

Tracing command usb pid 15 tid 100053 td 0xffffff000378b000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bce4d30, rbp = 0 ---

Tracing command usb pid 15 tid 100052 td 0xffffff000378b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcdfd30, rbp = 0 ---

Tracing command usb pid 15 tid 100051 td 0xffffff000378b720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcdad30, rbp = 0 ---

Tracing command usb pid 15 tid 100049 td 0xffffff000378c000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bccfd30, rbp = 0 ---

Tracing command usb pid 15 tid 100048 td 0xffffff000378c390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bccad30, rbp = 0 ---

Tracing command usb pid 15 tid 100047 td 0xffffff000378c720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcc5d30, rbp = 0 ---

Tracing command usb pid 15 tid 100046 td 0xffffff000378cab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcc0d30, rbp = 0 ---

Tracing command usb pid 15 tid 100045 td 0xffffff000353fab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcbad30, rbp = 0 ---

Tracing command usb pid 15 tid 100044 td 0xffffff0003754000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcb5d30, rbp = 0 ---

Tracing command usb pid 15 tid 100043 td 0xffffff0003754390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcb0d30, rbp = 0 ---

Tracing command usb pid 15 tid 100042 td 0xffffff0003754720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcabd30, rbp = 0 ---

Tracing command usb pid 15 tid 100041 td 0xffffff0003754ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bca5d30, rbp = 0 ---

Tracing command usb pid 15 tid 100040 td 0xffffff0003755000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bca0d30, rbp = 0 ---

Tracing command usb pid 15 tid 100039 td 0xffffff0003755390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc9bd30, rbp = 0 ---

Tracing command usb pid 15 tid 100038 td 0xffffff0003755720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc96d30, rbp = 0 ---

Tracing command usb pid 15 tid 100037 td 0xffffff0003755ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc90d30, rbp = 0 ---

Tracing command usb pid 15 tid 100036 td 0xffffff0003757000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc8bd30, rbp = 0 ---

Tracing command usb pid 15 tid 100035 td 0xffffff000350b720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc86d30, rbp = 0 ---

Tracing command usb pid 15 tid 100034 td 0xffffff000350bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
usb_process() at usb_process+0x15f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc81d30, rbp = 0 ---

Tracing command yarrow pid 14 tid 100017 td 0xffffff0003383000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
random_kthread() at random_kthread+0x1ba
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000071d30, rbp = 0 ---

Tracing command crypto returns pid 6 tid 100016 td 0xffffff0003383390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
crypto_ret_proc() at crypto_ret_proc+0x1b3
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800006cd30, rbp = 0 ---

Tracing command crypto pid 5 tid 100015 td 0xffffff0003383720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
crypto_proc() at crypto_proc+0x1cf
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000067d30, rbp = 0 ---

Tracing command g_down pid 4 tid 100013 td 0xffffff0003384000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
g_io_schedule_down() at g_io_schedule_down+0x1f4
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800005dd30, rbp = 0 ---

Tracing command g_up pid 3 tid 100012 td 0xffffff0003384390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
g_io_schedule_up() at g_io_schedule_up+0x12e
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000058d30, rbp = 0 ---

Tracing command g_event pid 2 tid 100011 td 0xffffff000336b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
g_event_procbody() at g_event_procbody+0xc3
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000053d30, rbp = 0 ---

Tracing command ng_queue pid 13 tid 100010 td 0xffffff000336b720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
ngthread() at ngthread+0x59
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800004ed30, rbp = 0 ---

Tracing command ng_queue pid 13 tid 100009 td 0xffffff000336bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
ngthread() at ngthread+0x59
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000049d30, rbp = 0 ---

Tracing command intr pid 12 tid 100064 td 0xffffff00037b9000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100063 td 0xffffff00037b9390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc44d30, rbp = 0 ---

Tracing command intr pid 12 tid 100060 td 0xffffff00037bc000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bd3dd30, rbp = 0 ---

Tracing command intr pid 12 tid 100059 td 0xffffff00037bc390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bd38d30, rbp = 0 ---

Tracing command intr pid 12 tid 100056 td 0xffffff0003757390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcf3d30, rbp = 0 ---

Tracing command intr pid 12 tid 100050 td 0xffffff000378bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcd5d30, rbp = 0 ---

Tracing command intr pid 12 tid 100033 td 0xffffff000353d000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc7cd30, rbp = 0 ---

Tracing command intr pid 12 tid 100031 td 0xffffff000353d720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000188d30, rbp = 0 ---

Tracing command intr pid 12 tid 100029 td 0xffffff000353f000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000add30, rbp = 0 ---

Tracing command intr pid 12 tid 100028 td 0xffffff000353f390
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100027 td 0xffffff000353f720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000a3d30, rbp = 0 ---

Tracing command intr pid 12 tid 100023 td 0xffffff0003508390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800008fd30, rbp = 0 ---

Tracing command intr pid 12 tid 100022 td 0xffffff0003508720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800008ad30, rbp = 0 ---

Tracing command intr pid 12 tid 100020 td 0xffffff000350b000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100018 td 0xffffff000337aab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000076d30, rbp = 0 ---

Tracing command intr pid 12 tid 100008 td 0xffffff000337a000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100007 td 0xffffff000337a390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800003fd30, rbp = 0 ---

Tracing command intr pid 12 tid 100006 td 0xffffff000337a720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800003ad30, rbp = 0 ---

Tracing command intr pid 12 tid 100005 td 0xffffff000336a000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
ithread_loop() at ithread_loop+0x23e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000035d30, rbp = 0 ---

Tracing command idle pid 11 tid 100004 td 0xffffff000336a390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sched_preempt() at sched_preempt+0xbc
ipi_bitmap_handler() at ipi_bitmap_handler+0x6d
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b
--- interrupt, rip = 0xffffffff8090154e, rsp = 0xffffff8000030b50, rbp = 0xffffff8000030b60 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1f0
sched_idletd() at sched_idletd+0x11f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000030d30, rbp = 0 ---

Tracing command idle pid 11 tid 100003 td 0xffffff000336a720
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x32
trap() at trap+0x28c
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff8090154e, rsp = 0xffffff800000bfe0, rbp = 0xffffff800002bb60 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1f0
sched_idletd() at sched_idletd+0x11f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800002bd30, rbp = 0 ---

Tracing command init pid 1 tid 100002 td 0xffffff000336aab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_catch_signals() at sleepq_catch_signals+0x307
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x32b
kern_wait() at kern_wait+0x3e7
wait4() at wait4+0x33
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c04c, rsp = 0x7fffffffe808, rbp = 0x401baa ---

Tracing command audit pid 10 tid 100001 td 0xffffff000336b000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_cv_wait() at _cv_wait+0x15d
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000021d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100212 td 0xffffff002c73c720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804026bd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100150 td 0xffffff0009618390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040135d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100149 td 0xffffff000950a720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040130d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100148 td 0xffffff000950aab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804012bd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100147 td 0xffffff00095a2000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040126d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100146 td 0xffffff00095a2390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040121d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100145 td 0xffffff00095a2720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804011cd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100144 td 0xffffff00095a2ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040117d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100143 td 0xffffff00095a3000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040112d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100142 td 0xffffff00095a3390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804010dd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100141 td 0xffffff00095a3720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040108d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100140 td 0xffffff00095a3ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040103d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100139 td 0xffffff00095a4000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400fed30, rbp = 0 ---

Tracing command kernel pid 0 tid 100138 td 0xffffff00095a4390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400f9d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100137 td 0xffffff00095a4720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400f4d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100136 td 0xffffff00095a4ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400efd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100135 td 0xffffff00095a5000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400ead30, rbp = 0 ---

Tracing command kernel pid 0 tid 100134 td 0xffffff00095a5390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400e5d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100133 td 0xffffff0006a7c720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400e0d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100132 td 0xffffff0006a7cab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400dbd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100131 td 0xffffff00093c5000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80400d6d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100107 td 0xffffff00069d1000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804005ed30, rbp = 0 ---

Tracing command kernel pid 0 tid 100104 td 0xffffff0006a71000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804003ed30, rbp = 0 ---

Tracing command kernel pid 0 tid 100102 td 0xffffff0006a71720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040034d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100101 td 0xffffff0006a74000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804002fd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100100 td 0xffffff0006a74720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804002ad30, rbp = 0 ---

Tracing command kernel pid 0 tid 100099 td 0xffffff0006a74ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040025d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100098 td 0xffffff0006a71ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040020d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100097 td 0xffffff0006a75000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804000cd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100096 td 0xffffff0006a75390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040002d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100095 td 0xffffff0006a75720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff804001bd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100094 td 0xffffff00069d2000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040016d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100093 td 0xffffff00069d1ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040011d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100092 td 0xffffff0006a79720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040007d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100091 td 0xffffff0006a79ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803fffdd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100090 td 0xffffff0006a7a000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803fff8d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100089 td 0xffffff0006a7a390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffdad30, rbp = 0 ---

Tracing command kernel pid 0 tid 100088 td 0xffffff0006a7a720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803fff3d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100087 td 0xffffff0006a7aab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffeed30, rbp = 0 ---

Tracing command kernel pid 0 tid 100086 td 0xffffff0006a7b000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffe9d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100085 td 0xffffff0006a7c000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffdfd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100084 td 0xffffff00069d6390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffd5d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100083 td 0xffffff0006a7b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffd0d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100082 td 0xffffff0006a7b720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffe4d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100081 td 0xffffff0006a7bab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffcbd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100080 td 0xffffff0006a7c390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffc6d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100079 td 0xffffff00069d6720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803ffc1d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100078 td 0xffffff00069d6ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040043d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100077 td 0xffffff0006a74390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8040048d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100067 td 0xffffff00037b8390
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x176
deadlkres() at deadlkres+0x229
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803dc5cd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100057 td 0xffffff00037bcab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
msleep_spin() at msleep_spin+0x201
taskqueue_thread_loop() at taskqueue_thread_loop+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bcfbd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100032 td 0xffffff000353d390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
msleep_spin() at msleep_spin+0x201
taskqueue_thread_loop() at taskqueue_thread_loop+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff803bc76d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100030 td 0xffffff000353dab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000183d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100026 td 0xffffff0003384720
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
msleep_spin() at msleep_spin+0x201
taskqueue_thread_loop() at taskqueue_thread_loop+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800009ed30, rbp = 0 ---

Tracing command kernel pid 0 tid 100025 td 0xffffff0003384ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
msleep_spin() at msleep_spin+0x201
taskqueue_thread_loop() at taskqueue_thread_loop+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000099d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100024 td 0xffffff0003508000
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
msleep_spin() at msleep_spin+0x201
taskqueue_thread_loop() at taskqueue_thread_loop+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000094d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100021 td 0xffffff0003508ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000085d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100019 td 0xffffff000350b390
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800007bd30, rbp = 0 ---

Tracing command kernel pid 0 tid 100014 td 0xffffff0003383ab0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sleep() at _sleep+0x241
taskqueue_thread_loop() at taskqueue_thread_loop+0xb0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000062d30, rbp = 0 ---

Tracing command kernel pid 0 tid 100000 td 0xffffffff80d252e0
sched_switch() at sched_switch+0x149
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_timedwait() at sleepq_timedwait+0x46
_sleep() at _sleep+0x226
scheduler() at scheduler+0x28e
mi_startup() at mi_startup+0x59
btext() at btext+0x2c

----------------
end

Attilio Rao

unread,
Apr 8, 2010, 11:05:39 AM4/8/10
to Marcin Cieslak, freebsd...@freebsd.org
2010/2/22 Marcin Cieslak <sa...@saper.info>:

>
>
>
> On Sat, 20 Feb 2010, Attilio Rao wrote:
>
>> 2010/2/18 Marcin Cieslak <sa...@saper.info>:
>>>
>>> My r203753 amd64 laptop falls into the deadlock situation
>>> every night while running periodic daily script. I am pretty
>>> certain this is related to ZFS.
>>>
>>> I have enabled DEADLKRES in the kernel. I even have
>>> a separate dump partition (not used for swap).
>>
>> May you reproduce the bug with WITNESS?
>> If you can, you should enable DEADLKRES too and once it panics let
>> extract a textdump(4) with the following commands:
>> bt, ps, show alllocks, show pcpu, allthreads
>
> Unfortunately, there is no way to write anything to disk.
> All attempts to talk to the ata subsystem from ddb(4)
> fail with EIO (probably timeout).

This may be a false positive.
May you please try the following patch and report if you can fix it
does fix it or not?:
http://www.freebsd.org/~attilio/Sandvine/deadlkres/deadlkres-blessed.diff

Reply all
Reply to author
Forward
0 new messages