Crashes (with dumps) in zap_leaf.c

[Zaphod Beeblebrox]

I'm getting multiple crashes that look rather like:

#0 0xffffffff8098e390 at kdb_backtrace+0x60
#1 0xffffffff80951066 at vpanic+0x126
#2 0xffffffff80950f33 at panic+0x43
#3 0xffffffff80bcfa4c at vm_fault_hold+0x1b2c
#4 0xffffffff80bcded7 at vm_fault+0x77
#5 0xffffffff80d5612c at trap_pfault+0x19c
#6 0xffffffff80d558fa at trap+0x47a
#7 0xffffffff80d3b8d2 at calltrap+0x8
#8 0xffffffff81a49a5a at zap_entry_create+0x27a
#9 0xffffffff81a45eee at fzap_add_cd+0xde
#10 0xffffffff81a4c051 at zap_add+0x101
#11 0xffffffff81a6bfb5 at zfs_link_create+0x415
#12 0xffffffff81a87a34 at zfs_freebsd_rename+0xac4
#13 0xffffffff80e81e1b at VOP_RENAME_APV+0xab
#14 0xffffffff80a03476 at kern_renameat+0x4a6
#15 0xffffffff80d5694f at amd64_syscall+0x40f
#16 0xffffffff80d3bbbb at Xfast_syscall+0xfb

Analyzing one of those crashes:

#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff80950cc2 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:486
#2 0xffffffff809510a5 in vpanic (fmt=<value optimized out>,
ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889
#3 0xffffffff80950f33 in panic (fmt=0x0)
at /usr/src/sys/kern/kern_shutdown.c:818
#4 0xffffffff80bcfa4c in vm_fault_hold (map=<value optimized out>,
vaddr=<value optimized out>, fault_type=<value optimized out>,
fault_flags=<value optimized out>, m_hold=<value optimized out>)
at /usr/src/sys/vm/vm_fault.c:329
#5 0xffffffff80bcded7 in vm_fault (map=0xfffff80002000000,
vaddr=<value optimized out>, fault_type=1 '\001', fault_flags=0)
at /usr/src/sys/vm/vm_fault.c:273
#6 0xffffffff80d5612c in trap_pfault (frame=0xfffffe0c56854320, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:757
#7 0xffffffff80d558fa in trap (frame=0xfffffe0c56854320)
at /usr/src/sys/amd64/amd64/trap.c:447
#8 0xffffffff80d3b8d2 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:236
#9 0xffffffff81a494c5 in zap_leaf_array_create (l=0xfffff8008b3a3b00,
buf=<value optimized out>, integer_size=1,
num_integers=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:198
#10 0xffffffff81a49a5a in zap_entry_create (l=0xfffff8008b3a3b00,
zn=0xfffff8008bf19e00, cd=Cannot access memory at address 0x0
)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:644
#11 0xffffffff81a45eee in fzap_add_cd (zn=0xfffff8008bf19e00,
integer_size=<value optimized out>, num_integers=1,
val=0xfffffe0c568546d0, cd=4294967295, tx=0xfffff80787758c00)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:814
#12 0xffffffff81a4c051 in zap_add (os=<value optimized out>,
zapobj=<value optimized out>, key=<value optimized out>,
integer_size=<value optimized out>, num_integers=<value optimized out>,
val=<value optimized out>, tx=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1014
#13 0xffffffff81a6bfb5 in zfs_link_create (dl=0xfffff800966cce00,
zp=<value optimized out>, tx=0xfffff80787758c00,
flag=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:767
#14 0xffffffff81a87a34 in zfs_freebsd_rename (ap=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4067
#15 0xffffffff80e81e1b in VOP_RENAME_APV (vop=<value optimized out>,
a=<value optimized out>) at vnode_if.c:1546
#16 0xffffffff80a03476 in kern_renameat (td=<value optimized out>,
oldfd=<value optimized out>, old=<value optimized out>,
newfd=<value optimized out>, new=<value optimized out>,
pathseg=<value optimized out>) at vnode_if.h:636
#17 0xffffffff80d5694f in amd64_syscall (td=0xfffff8048f7fd000, traced=0)
at subr_syscall.c:141
#18 0xffffffff80d3bbbb in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:396
#19 0x000000080381fbaa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) frame 9
#9 0xffffffff81a494c5 in zap_leaf_array_create (l=0xfffff8008b3a3b00,
buf=<value optimized out>, integer_size=1,
num_integers=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:198
198 ZAP_LEAF_CHUNK(l, chunk).l_free.lf_next;
(kgdb) p chunk
No symbol "chunk" in current context.

... now I don't debug kernels too often, but I don't know why I can't see
chunk. It's declared as an int in this function.

Does this give anyone insight?
_______________________________________________
freeb...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Zaphod Beeblebrox]

Replying to my own post and adding some detail. I didn't realize at first
that was an l as-in the lowercase L. But looking at the macro, it is.

(kgdb) p l
$1 = (zap_leaf_t *) 0xfffff8008b3a3b00
(kgdb) p *l
$2 = {l_dbu = {dbu_tqent = {tqent_task = {ta_link = {stqe_next = 0x0},
ta_pending = 0, ta_priority = 0, ta_func = 0, ta_context = 0x0},
tqent_func = 0, tqent_arg = 0x0},
dbu_evict_func = 0xffffffff81a482b0 <zap_leaf_pageout>}, l_rwlock = {
lock_object = {lo_name = 0xffffffff81af06d9 "l->l_rwlock",
lo_flags = 40960000, lo_data = 0, lo_witness = 0x0},
sx_lock = 18446735297203916800}, l_blkid = 1311, l_bs = 14,
l_dbuf = 0xfffff806255e11b0}

Now the macro starts by dereferencing l->l_phys->l_hash ... I can get it to
spit out l->l_bs, but the debugger won't spit out l->l_phys.

At this point ... I'm wondering if I'm confused and missing something.