Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

panic: vm_fault: fault on nofault entry

90 views
Skip to first unread message

Glen Barber

unread,
Mar 9, 2014, 12:56:48 PM3/9/14
to
We are having regular panics on several machines in the cluster.

Below follows the script from the kgdb(1) session, hopefully providing
enough information. This machine runs 11.0-CURRENT #2 r262892, from
2 days ago.

It uses tmpfs(5) for the port build workspace. I have an unconfirmed
suspicion that use of sysutils/lsof is involved somehow, but cannot be
sure. (In my experience with panics with port building, removing lsof
from the system did have an effect, but I may be going down the wrong
rabbit hole.)


Script started on Sun Mar 9 16:40:07 2014
ro...@redbuild01.nyi:/usr/obj/usr/src/sys/REDBUILD # sh
# kgdb ./kernel.debug /var/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: vm_fault: fault on nofault entry, addr: fffffe035021a000
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1839a54180
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1839a54230
panic() at panic+0x155/frame 0xfffffe1839a542b0
vm_fault_hold() at vm_fault_hold+0x1e7a/frame 0xfffffe1839a54500
vm_fault() at vm_fault+0x77/frame 0xfffffe1839a54540
trap_pfault() at trap_pfault+0x199/frame 0xfffffe1839a545e0
trap() at trap+0x4a0/frame 0xfffffe1839a547f0
calltrap() at calltrap+0x8/frame 0xfffffe1839a547f0
--- trap 0xc, rip = 0xffffffff80d97bab, rsp = 0xfffffe1839a548b0, rbp = 0xfffffe1839a54910 ---
copyout() at copyout+0x3b/frame 0xfffffe1839a54910
memrw() at memrw+0x19f/frame 0xfffffe1839a54950
giant_read() at giant_read+0xa4/frame 0xfffffe1839a54990
devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe1839a549f0
dofileread() at dofileread+0x95/frame 0xfffffe1839a54a40
kern_readv() at kern_readv+0x68/frame 0xfffffe1839a54a90
sys_read() at sys_read+0x63/frame 0xfffffe1839a54ae0
amd64_syscall() at amd64_syscall+0x3fb/frame 0xfffffe1839a54bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1839a54bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b8444a, rsp = 0x7fffffffd088, rbp = 0x7fffffffd0d0 ---
KDB: enter: panic

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/tmpfs.ko.symbols...done.
Loaded symbols for /boot/kernel/tmpfs.ko.symbols
Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
#0 doadump (textdump=-967130448) at pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0 doadump (textdump=-967130448) at pcpu.h:219
#1 0xffffffff8034a1a5 in db_fncall (dummy1=<value optimized out>,
dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>)
at /usr/src/sys/ddb/db_command.c:578
#2 0xffffffff80349e8d in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:449
#3 0xffffffff80349c04 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
#4 0xffffffff8034c660 in db_trap (type=<value optimized out>, code=0)
at /usr/src/sys/ddb/db_main.c:231
#5 0xffffffff80987ae9 in kdb_trap (type=3, code=0, tf=<value optimized out>)
at /usr/src/sys/kern/subr_kdb.c:656
#6 0xffffffff80d999b9 in trap (frame=0xfffffe1839a54160)
at /usr/src/sys/amd64/amd64/trap.c:571
#7 0xffffffff80d7e6e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff8098724e in kdb_enter (why=0xffffffff8100f4ba "panic", msg=<value optimized out>)
at cpufunc.h:63
#9 0xffffffff80946a75 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:752
#10 0xffffffff80c0a1fa in vm_fault_hold (map=<value optimized out>,
vaddr=<value optimized out>, fault_type=<value optimized out>,
fault_flags=<value optimized out>, m_hold=<value optimized out>)
at /usr/src/sys/vm/vm_fault.c:272
#11 0xffffffff80c08337 in vm_fault (map=0xfffff80002000000, vaddr=<value optimized out>,
fault_type=1 '\001', fault_flags=128) at /usr/src/sys/vm/vm_fault.c:217
#12 0xffffffff80d9a1a9 in trap_pfault (frame=0xfffffe1839a54800, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:767
#13 0xffffffff80d999d0 in trap (frame=0xfffffe1839a54800)
at /usr/src/sys/amd64/amd64/trap.c:455
#14 0xffffffff80d7e6e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#15 0xffffffff80d97bab in copyout () at /usr/src/sys/amd64/amd64/support.S:246
#16 0xffffffff8099c2f5 in uiomove_faultflag (cp=<value optimized out>,
n=<value optimized out>, uio=0xfffffe1839a54ab0, nofault=<value optimized out>)
at /usr/src/sys/kern/subr_uio.c:192
#17 0xffffffff80d8612f in memrw (dev=0xfffff8000dbd0400, uio=0xfffffe1839a54ab0,
flags=113246208) at /usr/src/sys/amd64/amd64/mem.c:101
#18 0xffffffff808ecf04 in giant_read (dev=0xfffff8000dbd0400, uio=0xfffffe1839a54ab0, ioflag=0)
at /usr/src/sys/kern/kern_conf.c:442
#19 0xffffffff808185cb in devfs_read_f (fp=0xfffff80083439230, uio=0xfffffe1839a54ab0,
cred=<value optimized out>, flags=0, td=0xfffff80e4edb8490)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1193
#20 0xffffffff809a15e5 in dofileread (td=0xfffff80e4edb8490, fd=4, fp=0xfffff80083439230,
auio=0xfffffe1839a54ab0, offset=<value optimized out>, flags=1172307968) at file.h:299
#21 0xffffffff809a1308 in kern_readv (td=0xfffff80e4edb8490, fd=4, auio=0xfffffe1839a54ab0)
at /usr/src/sys/kern/sys_generic.c:256
#22 0xffffffff809a1293 in sys_read (td=<value optimized out>, uap=<value optimized out>)
at /usr/src/sys/kern/sys_generic.c:171
#23 0xffffffff80d9a9fb in amd64_syscall (td=0xfffff80e4edb8490, traced=0) at subr_syscall.c:133
#24 0xffffffff80d7e9cb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
#25 0x0000000800b8444a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) frame 19
#19 0xffffffff808185cb in devfs_read_f (fp=0xfffff80083439230, uio=0xfffffe1839a54ab0,
cred=<value optimized out>, flags=0, td=0xfffff80e4edb8490)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1193
1193 error = dsw->d_read(dev, uio, ioflag);
(kgdb) list
1188 ioflag = fp->f_flag & (O_NONBLOCK | O_DIRECT);
1189 if (ioflag & O_DIRECT)
1190 ioflag |= IO_DIRECT;
1191
1192 foffset_lock_uio(fp, uio, flags | FOF_NOLOCK);
1193 error = dsw->d_read(dev, uio, ioflag);
1194 if (uio->uio_resid != resid || (error == 0 && resid != 0))
1195 vfs_timestamp(&dev->si_atime);
1196 td->td_fpop = fpop;
1197 dev_relthread(dev, ref);
(kgdb) down
#18 0xffffffff808ecf04 in giant_read (dev=0xfffff8000dbd0400, uio=0xfffffe1839a54ab0, ioflag=0)
at /usr/src/sys/kern/kern_conf.c:442
442 retval = dsw->d_gianttrick->d_read(dev, uio, ioflag);
(kgdb) list
437
438 dsw = dev_refthread(dev, &ref);
439 if (dsw == NULL)
440 return (ENXIO);
441 mtx_lock(&Giant);
442 retval = dsw->d_gianttrick->d_read(dev, uio, ioflag);
443 mtx_unlock(&Giant);
444 dev_relthread(dev, ref);
445 return (retval);
446 }
(kgdb) p *dev
$1 = {si_spare0 = 0x0, si_flags = 4, si_atime = {tv_sec = 1394286776, tv_nsec = 0},
si_ctime = {tv_sec = 1394236183, tv_nsec = 584945000}, si_mtime = {tv_sec = 1394236183,
tv_nsec = 584945000}, si_uid = 0, si_gid = 2, si_mode = 416, si_cred = 0x0, si_drv0 = 1,
si_refcount = 9, si_list = {le_next = 0xfffff8000dbd0600, le_prev = 0xffffffff8144db18},
si_clone = {le_next = 0x0, le_prev = 0x0}, si_children = {lh_first = 0x0}, si_siblings = {
le_next = 0x0, le_prev = 0x0}, si_parent = 0x0, si_mountpt = 0x0, si_drv1 = 0x0,
si_drv2 = 0x0, si_devsw = 0xffffffff8144da78, si_iosize_max = 65536, si_usecount = 1,
si_threadcount = 2, __si_u = {__sid_snapdata = 0x0},
si_name = "kmem", '\0' <repeats 59 times>}
(kgdb) p *uio
$2 = {uio_iov = 0xfffffe1839a54aa0, uio_iovcnt = 1, uio_offset = -2184830705664,
uio_resid = 113246208, uio_segflg = UIO_USERSPACE, uio_rw = UIO_READ,
uio_td = 0xfffff80e4edb8490}
(kgdb) p *ioflag
Cannot access memory at address 0x0
(kgdb) p Giant
$3 = {lock_object = {lo_name = 0xffffffff8100e05a "Giant", lo_flags = 17498112, lo_data = 0,
lo_witness = 0x0}, mtx_lock = 18446735339069080720}
(kgdb) down
#17 0xffffffff80d8612f in memrw (dev=0xfffff8000dbd0400, uio=0xfffffe1839a54ab0,
flags=113246208) at /usr/src/sys/amd64/amd64/mem.c:101
101 error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
(kgdb) list
96 if (dev2unit(dev) == CDEV_MINOR_MEM) {
97 v = uio->uio_offset;
98 kmemphys:
99 o = v & PAGE_MASK;
100 c = min(uio->uio_resid, (u_int)(PAGE_SIZE - o));
101 error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
102 continue;
103 }
104 else if (dev2unit(dev) == CDEV_MINOR_KMEM) {
105 v = uio->uio_offset;
(kgdb) p *v
$4 = 0
(kgdb) p *c
$5 = 0
(kgdb) p *uio
$6 = {uio_iov = 0xfffffe1839a54aa0, uio_iovcnt = 1, uio_offset = -2184830705664,
uio_resid = 113246208, uio_segflg = UIO_USERSPACE, uio_rw = UIO_READ,
uio_td = 0xfffff80e4edb8490}
(kgdb) down
#16 0xffffffff8099c2f5 in uiomove_faultflag (cp=<value optimized out>,
n=<value optimized out>, uio=0xfffffe1839a54ab0, nofault=<value optimized out>)
at /usr/src/sys/kern/subr_uio.c:192
192 error = copyout(cp, iov->iov_base, cnt);
(kgdb) list
187 switch (uio->uio_segflg) {
188
189 case UIO_USERSPACE:
190 maybe_yield();
191 if (uio->uio_rw == UIO_READ)
192 error = copyout(cp, iov->iov_base, cnt);
193 else
194 error = copyin(iov->iov_base, cp, cnt);
195 if (error)
196 goto out;
(kgdb) p *cp
Attempt to dereference a generic pointer.
(kgdb) p cp
$7 = <value optimized out>
(kgdb) down
#15 0xffffffff80d97bab in copyout () at /usr/src/sys/amd64/amd64/support.S:246
246 cld
Current language: auto; currently asm
(kgdb) list
241 xchgq %rdi,%rsi
242 /* bcopy(%rsi, %rdi, %rdx) */
243 movq %rdx,%rcx
244
245 shrq $3,%rcx
246 cld
247 rep
248 movsq
249 movb %dl,%cl
250 andb $7,%cl
(kgdb) down
#14 0xffffffff80d7e6e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
231 call trap
(kgdb) list
226 #endif
227 .globl calltrap
228 .type calltrap,@function
229 calltrap:
230 movq %rsp,%rdi
231 call trap
232 MEXITCOUNT
233 jmp doreti /* Handle any pending ASTs */
234
235 /*
(kgdb) quit

Script done on Sun Mar 9 16:46:04 2014

Glen

Konstantin Belousov

unread,
Mar 9, 2014, 2:01:32 PM3/9/14
to
This is very similar to issue reported several time ago.
Try this patch. I never get a feedback.

diff --git a/sys/amd64/amd64/mem.c b/sys/amd64/amd64/mem.c
index abbbb21..fd9c5df 100644
--- a/sys/amd64/amd64/mem.c
+++ b/sys/amd64/amd64/mem.c
@@ -98,7 +98,13 @@ memrw(struct cdev *dev, struct uio *uio, int flags)
kmemphys:
o = v & PAGE_MASK;
c = min(uio->uio_resid, (u_int)(PAGE_SIZE - o));
- error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
+ v = PHYS_TO_DMAP(v);
+ if (v < DMAP_MIN_ADDRESS ||
+ (v > DMAP_MIN_ADDRESS + dmaplimit &&
+ v <= DMAP_MAX_ADDRESS) ||
+ pmap_kextract(v) == 0)
+ return (EFAULT);
+ error = uiomove((void *)v, (int)c, uio);
continue;

Glen Barber

unread,
Mar 9, 2014, 2:16:57 PM3/9/14
to
On Sun, Mar 09, 2014 at 08:01:32PM +0200, Konstantin Belousov wrote:
> On Sun, Mar 09, 2014 at 12:56:48PM -0400, Glen Barber wrote:
> > We are having regular panics on several machines in the cluster.
> >
> > Below follows the script from the kgdb(1) session, hopefully providing
> > enough information. This machine runs 11.0-CURRENT #2 r262892, from
> > 2 days ago.
> >
> > It uses tmpfs(5) for the port build workspace. I have an unconfirmed
> > suspicion that use of sysutils/lsof is involved somehow, but cannot be
> > sure. (In my experience with panics with port building, removing lsof
> > from the system did have an effect, but I may be going down the wrong
> > rabbit hole.)
> >
>
> This is very similar to issue reported several time ago.
> Try this patch. I never get a feedback.
>
> diff --git a/sys/amd64/amd64/mem.c b/sys/amd64/amd64/mem.c
> index abbbb21..fd9c5df 100644
> --- a/sys/amd64/amd64/mem.c
> +++ b/sys/amd64/amd64/mem.c
> @@ -98,7 +98,13 @@ memrw(struct cdev *dev, struct uio *uio, int flags)
> kmemphys:
> o = v & PAGE_MASK;
> c = min(uio->uio_resid, (u_int)(PAGE_SIZE - o));
> - error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
> + v = PHYS_TO_DMAP(v);
> + if (v < DMAP_MIN_ADDRESS ||
> + (v > DMAP_MIN_ADDRESS + dmaplimit &&
> + v <= DMAP_MAX_ADDRESS) ||
> + pmap_kextract(v) == 0)
> + return (EFAULT);
> + error = uiomove((void *)v, (int)c, uio);
> continue;
> }
> else if (dev2unit(dev) == CDEV_MINOR_KMEM) {

There is a very similar patch on one of these machines.

Index: sys/amd64/amd64/mem.c
===================================================================
--- sys/amd64/amd64/mem.c (revision 262298)
+++ sys/amd64/amd64/mem.c (working copy)
@@ -98,6 +98,12 @@
kmemphys:
o = v & PAGE_MASK;
c = min(uio->uio_resid, (u_int)(PAGE_SIZE - o));
+ v = PHYS_TO_DMAP(v);
+ if (v < DMAP_MIN_ADDRESS ||
+ (v > DMAP_MIN_ADDRESS + dmaplimit &&
+ v <= DMAP_MAX_ADDRESS) ||
+ pmap_kextract(v) == 0)
+ return (EFAULT);
error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
continue;
}
Index: sys/amd64/amd64/pmap.c
===================================================================
--- sys/amd64/amd64/pmap.c (revision 262298)
+++ sys/amd64/amd64/pmap.c (working copy)
@@ -321,7 +321,7 @@
"Number of kernel page table pages allocated on bootup");

static int ndmpdp;
-static vm_paddr_t dmaplimit;
+vm_paddr_t dmaplimit;
vm_offset_t kernel_vm_end = VM_MIN_KERNEL_ADDRESS;
pt_entry_t pg_nx;

Index: sys/amd64/include/pmap.h
===================================================================
--- sys/amd64/include/pmap.h (revision 262298)
+++ sys/amd64/include/pmap.h (working copy)
@@ -369,6 +369,7 @@
extern vm_paddr_t dump_avail[];
extern vm_offset_t virtual_avail;
extern vm_offset_t virtual_end;
+extern vm_paddr_t dmaplimit;

#define pmap_page_get_memattr(m) ((vm_memattr_t)(m)->md.pat_mode)
#define pmap_page_is_write_mapped(m) (((m)->aflags & PGA_WRITEABLE) != 0)

The machine this change is on paniced today as well. That machine runs
r262298M, and I have a vmcore from Feb 24 (there was not enough
available space to get a crash dump today.)

The backtrace from Feb 24 follows.

Script started on Sun Mar 9 18:14:41 2014
ro...@redbuild04.nyi:/usr/obj/usr/src/sys/REDBUILD # sh
# kgdb ./kernel.debug /var/crash/vmcore.3
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: vm_fault: fault on nofault entry, addr: fffffe03becbc000
cpuid = 23
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1838ec1180
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1838ec1230
panic() at panic+0x155/frame 0xfffffe1838ec12b0
vm_fault_hold() at vm_fault_hold+0x1e7a/frame 0xfffffe1838ec1500
vm_fault() at vm_fault+0x77/frame 0xfffffe1838ec1540
trap_pfault() at trap_pfault+0x199/frame 0xfffffe1838ec15e0
trap() at trap+0x4a0/frame 0xfffffe1838ec17f0
calltrap() at calltrap+0x8/frame 0xfffffe1838ec17f0
--- trap 0xc, rip = 0xffffffff80d971fb, rsp = 0xfffffe1838ec18b0, rbp = 0xfffffe1838ec1910 ---
copyout() at copyout+0x3b/frame 0xfffffe1838ec1910
memrw() at memrw+0x1ef/frame 0xfffffe1838ec1950
giant_read() at giant_read+0xa4/frame 0xfffffe1838ec1990
devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe1838ec19f0
dofileread() at dofileread+0x95/frame 0xfffffe1838ec1a40
kern_readv() at kern_readv+0x68/frame 0xfffffe1838ec1a90
sys_read() at sys_read+0x63/frame 0xfffffe1838ec1ae0
amd64_syscall() at amd64_syscall+0x3fb/frame 0xfffffe1838ec1bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1838ec1bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b8343a, rsp = 0x7fffffffcfe8, rbp = 0x7fffffffd030 ---
KDB: enter: panic

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/tmpfs.ko.symbols...done.
Loaded symbols for /boot/kernel/tmpfs.ko.symbols
Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
#0 doadump (textdump=-954994000) at pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0 doadump (textdump=-954994000) at pcpu.h:219
#1 0xffffffff8034a175 in db_fncall (dummy1=<value optimized out>,
dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>)
at /usr/src/sys/ddb/db_command.c:578
#2 0xffffffff80349e5d in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:449
#3 0xffffffff80349bd4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
#4 0xffffffff8034c630 in db_trap (type=<value optimized out>, code=0)
at /usr/src/sys/ddb/db_main.c:231
#5 0xffffffff80987329 in kdb_trap (type=3, code=0, tf=<value optimized out>)
at /usr/src/sys/kern/subr_kdb.c:656
#6 0xffffffff80d99009 in trap (frame=0xfffffe1838ec1160)
at /usr/src/sys/amd64/amd64/trap.c:571
#7 0xffffffff80d7dd12 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff80986a8e in kdb_enter (why=0xffffffff8100ed4f "panic", msg=<value optimized out>)
at cpufunc.h:63
#9 0xffffffff809462b5 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:752
#10 0xffffffff80c0981a in vm_fault_hold (map=<value optimized out>,
vaddr=<value optimized out>, fault_type=<value optimized out>,
fault_flags=<value optimized out>, m_hold=<value optimized out>)
at /usr/src/sys/vm/vm_fault.c:272
#11 0xffffffff80c07957 in vm_fault (map=0xfffff80002000000, vaddr=<value optimized out>,
fault_type=1 '\001', fault_flags=128) at /usr/src/sys/vm/vm_fault.c:217
#12 0xffffffff80d997f9 in trap_pfault (frame=0xfffffe1838ec1800, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:767
#13 0xffffffff80d99020 in trap (frame=0xfffffe1838ec1800)
at /usr/src/sys/amd64/amd64/trap.c:455
#14 0xffffffff80d7dd12 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#15 0xffffffff80d971fb in copyout () at /usr/src/sys/amd64/amd64/support.S:246
#16 0xffffffff8099bb35 in uiomove_faultflag (cp=<value optimized out>,
n=<value optimized out>, uio=0xfffffe1838ec1ab0, nofault=<value optimized out>)
at /usr/src/sys/kern/subr_uio.c:192
#17 0xffffffff80d8576f in memrw (dev=<value optimized out>, uio=<value optimized out>,
flags=<value optimized out>) at /usr/src/sys/amd64/amd64/mem.c:107
---Type <return> to continue, or q <return> to quit---
#18 0xffffffff808ec764 in giant_read (dev=0xfffff80011347c00, uio=0xfffffe1838ec1ab0, ioflag=0)
at /usr/src/sys/kern/kern_conf.c:442
#19 0xffffffff80817e2b in devfs_read_f (fp=0xfffff80854be3140, uio=0xfffffe1838ec1ab0,
cred=<value optimized out>, flags=0, td=0xfffff801f52c5490)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1193
#20 0xffffffff809a0e25 in dofileread (td=0xfffff801f52c5490, fd=4, fp=0xfffff80854be3140,
auio=0xfffffe1838ec1ab0, offset=<value optimized out>, flags=1172307968) at file.h:299
#21 0xffffffff809a0b48 in kern_readv (td=0xfffff801f52c5490, fd=4, auio=0xfffffe1838ec1ab0)
at /usr/src/sys/kern/sys_generic.c:256
#22 0xffffffff809a0ad3 in sys_read (td=<value optimized out>, uap=<value optimized out>)
at /usr/src/sys/kern/sys_generic.c:171
#23 0xffffffff80d9a04b in amd64_syscall (td=0xfffff801f52c5490, traced=0) at subr_syscall.c:133
#24 0xffffffff80d7dffb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
#25 0x0000000800b8343a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) quit

Script done on Sun Mar 9 18:14:59 2014

Glen

Sean Bruno

unread,
Mar 9, 2014, 6:10:20 PM3/9/14
to
Not sure I can add much here other than to say that redbuild machines
are now running -current as opposed to stable/10.

We are running redbuild01/02 unpatched and 03/04 with patch to compare
stability. We haven't seen much difference, so either I've screwed up
the patch or the bug report.

sean
signature.asc

Konstantin Belousov

unread,
Mar 10, 2014, 11:46:06 AM3/10/14
to
On Sun, Mar 09, 2014 at 02:16:57PM -0400, Glen Barber wrote:
> panic: vm_fault: fault on nofault entry, addr: fffffe03becbc000

I see, this panic is for access to the kernel map, not for the direct map.
I think that this is a race with other CPU unmapping some page in the
kernel map, which cannot be solved by access checks.

Please try the following. I booted with the patch and checked that
kgdb /boot/kernel/kernel /dev/mem works, but did not tried to reproduce
the issue.

diff --git a/sys/amd64/amd64/mem.c b/sys/amd64/amd64/mem.c
index abbbb21..5a4d8a9 100644
--- a/sys/amd64/amd64/mem.c
+++ b/sys/amd64/amd64/mem.c
@@ -76,14 +76,16 @@ MALLOC_DEFINE(M_MEMDESC, "memdesc", "memory range descriptors");
int
memrw(struct cdev *dev, struct uio *uio, int flags)
{
- int o;
- u_long c = 0, v;
struct iovec *iov;
- int error = 0;
+ u_long c, v;
+ int error, o, sflags;
vm_offset_t addr, eaddr;

GIANT_REQUIRED;

+ error = 0;
+ c = 0;
+ sflags = curthread_pflags_set(TDP_DEVMEMIO);
while (uio->uio_resid > 0 && error == 0) {
iov = uio->uio_iov;
if (iov->iov_len == 0) {
@@ -98,7 +100,15 @@ memrw(struct cdev *dev, struct uio *uio, int flags)
kmemphys:
o = v & PAGE_MASK;
c = min(uio->uio_resid, (u_int)(PAGE_SIZE - o));
- error = uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio);
+ v = PHYS_TO_DMAP(v);
+ if (v < DMAP_MIN_ADDRESS ||
+ (v > DMAP_MIN_ADDRESS + dmaplimit &&
+ v <= DMAP_MAX_ADDRESS) ||
+ pmap_kextract(v) == 0) {
+ error = EFAULT;
+ goto ret;
+ }
+ error = uiomove((void *)v, (int)c, uio);
continue;
}
else if (dev2unit(dev) == CDEV_MINOR_KMEM) {
@@ -119,22 +129,30 @@ kmemphys:
addr = trunc_page(v);
eaddr = round_page(v + c);

- if (addr < VM_MIN_KERNEL_ADDRESS)
- return (EFAULT);
- for (; addr < eaddr; addr += PAGE_SIZE)
- if (pmap_extract(kernel_pmap, addr) == 0)
- return (EFAULT);
-
+ if (addr < VM_MIN_KERNEL_ADDRESS) {
+ error = EFAULT;
+ goto ret;
+ }
+ for (; addr < eaddr; addr += PAGE_SIZE) {
+ if (pmap_extract(kernel_pmap, addr) == 0) {
+ error = EFAULT;
+ goto ret;
+ }
+ }
if (!kernacc((caddr_t)(long)v, c,
uio->uio_rw == UIO_READ ?
- VM_PROT_READ : VM_PROT_WRITE))
- return (EFAULT);
+ VM_PROT_READ : VM_PROT_WRITE)) {
+ error = EFAULT;
+ goto ret;
+ }

error = uiomove((caddr_t)(long)v, (int)c, uio);
continue;
}
/* else panic! */
}
+ret:
+ curthread_pflags_restore(sflags);
return (error);
}

diff --git a/sys/amd64/amd64/trap.c b/sys/amd64/amd64/trap.c
index f7d0afd..b1cbdbc 100644
--- a/sys/amd64/amd64/trap.c
+++ b/sys/amd64/amd64/trap.c
@@ -787,6 +787,12 @@ nogo:
frame->tf_rip = (long)curpcb->pcb_onfault;
return (0);
}
+ if ((td->td_pflags & TDP_DEVMEMIO) != 0) {
+ KASSERT(curpcb->pcb_onfault != NULL,
+ ("/dev/mem without pcb_onfault"));
+ frame->tf_rip = (long)curpcb->pcb_onfault;
+ return (0);
+ }
trap_fatal(frame, eva);
return (-1);
}
diff --git a/sys/sys/proc.h b/sys/sys/proc.h
index fce1f8a..e7cd022 100644
--- a/sys/sys/proc.h
+++ b/sys/sys/proc.h
@@ -424,6 +424,7 @@ do { \
#define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */
#define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */
#define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */
+#define TDP_DEVMEMIO 0x20000000 /* Accessing memory for /dev/mem */

/*
* Reasons that the current thread can not be run yet.
diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c
index 4a6495f..023860c 100644
--- a/sys/vm/vm_fault.c
+++ b/sys/vm/vm_fault.c
@@ -269,6 +269,8 @@ RetryFault:;
map_generation = fs.map->timestamp;

if (fs.entry->eflags & MAP_ENTRY_NOFAULT) {
+ if ((curthread->td_pflags & TDP_DEVMEMIO) != 0)
+ return (KERN_FAILURE);
panic("vm_fault: fault on nofault entry, addr: %lx",
(u_long)vaddr);
}

Glen Barber

unread,
Mar 10, 2014, 11:51:15 AM3/10/14
to
On Mon, Mar 10, 2014 at 05:46:06PM +0200, Konstantin Belousov wrote:
> On Sun, Mar 09, 2014 at 02:16:57PM -0400, Glen Barber wrote:
> > panic: vm_fault: fault on nofault entry, addr: fffffe03becbc000
>
> I see, this panic is for access to the kernel map, not for the direct map.
> I think that this is a race with other CPU unmapping some page in the
> kernel map, which cannot be solved by access checks.
>
> Please try the following. I booted with the patch and checked that
> kgdb /boot/kernel/kernel /dev/mem works, but did not tried to reproduce
> the issue.
>

Thank you for looking into this. I will report back.

Glen

Glen Barber

unread,
Mar 10, 2014, 2:05:08 PM3/10/14
to
The machine this was tested paniced again, but a bit differently.

This is the kgdb session from this vmcore:


Script started on Mon Mar 10 17:58:33 2014
command: /bin/sh
# kgdb ./kernel.debug /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
Sleeping thread (tid 100702, pid 24712) owns a non-sleepable lock
KDB: stack backtrace of thread 100702:
sched_switch() at sched_switch+0x29e/frame 0xfffffe18390b8820
mi_switch() at mi_switch+0xe1/frame 0xfffffe18390b8860
sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe18390b88e0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe18390b8910
_sleep() at _sleep+0x2a3/frame 0xfffffe18390b8990
pipe_read() at pipe_read+0x34a/frame 0xfffffe18390b89f0
dofileread() at dofileread+0x95/frame 0xfffffe18390b8a40
kern_readv() at kern_readv+0x68/frame 0xfffffe18390b8a90
sys_read() at sys_read+0x63/frame 0xfffffe18390b8ae0
amd64_syscall() at amd64_syscall+0x3fb/frame 0xfffffe18390b8bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe18390b8bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b8443a, rsp = 0x7fffffffac88, rbp = 0x7fffffffb500 ---
panic: sleeping thread
cpuid = 19
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe18392db010
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe18392db0c0
panic() at panic+0x155/frame 0xfffffe18392db140
propagate_priority() at propagate_priority+0x259/frame 0xfffffe18392db170
turnstile_wait() at turnstile_wait+0x3fe/frame 0xfffffe18392db1c0
__mtx_lock_sleep() at __mtx_lock_sleep+0x163/frame 0xfffffe18392db240
vm_map_lookup() at vm_map_lookup+0x38/frame 0xfffffe18392db2c0
vm_fault_hold() at vm_fault_hold+0xd1/frame 0xfffffe18392db510
vm_fault() at vm_fault+0x77/frame 0xfffffe18392db550
trap_pfault() at trap_pfault+0x199/frame 0xfffffe18392db5f0
trap() at trap+0x4a0/frame 0xfffffe18392db800
calltrap() at calltrap+0x8/frame 0xfffffe18392db800
--- trap 0xc, rip = 0xffffffff80d972cd, rsp = 0xfffffe18392db8c0, rbp = 0xfffffe18392db920 ---
copyin() at copyin+0x3d/frame 0xfffffe18392db920
pipe_write() at pipe_write+0x10ea/frame 0xfffffe18392db9f0
dofilewrite() at dofilewrite+0x87/frame 0xfffffe18392dba40
kern_writev() at kern_writev+0x68/frame 0xfffffe18392dba90
sys_write() at sys_write+0x63/frame 0xfffffe18392dbae0
amd64_syscall() at amd64_syscall+0x3fb/frame 0xfffffe18392dbbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe18392dbbf0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800b35afc, rsp = 0x7fffffffd3b8, rbp = 0x41 ---
KDB: enter: panic

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/tmpfs.ko.symbols...done.
Loaded symbols for /boot/kernel/tmpfs.ko.symbols
Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
#0 doadump (textdump=-959294432) at pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0 doadump (textdump=-959294432) at pcpu.h:219
#1 0xffffffff8034a175 in db_fncall (dummy1=<value optimized out>, dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>)
at /usr/src/sys/ddb/db_command.c:578
#2 0xffffffff80349e5d in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:449
#3 0xffffffff80349bd4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
#4 0xffffffff8034c630 in db_trap (type=<value optimized out>, code=0) at /usr/src/sys/ddb/db_main.c:231
#5 0xffffffff80987329 in kdb_trap (type=3, code=0, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:656
#6 0xffffffff80d99059 in trap (frame=0xfffffe18392daff0) at /usr/src/sys/amd64/amd64/trap.c:571
#7 0xffffffff80d7dd22 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff80986a8e in kdb_enter (why=0xffffffff8100edaf "panic", msg=<value optimized out>) at cpufunc.h:63
#9 0xffffffff809462b5 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:752
#10 0xffffffff80999949 in propagate_priority (td=<value optimized out>) at /usr/src/sys/kern/subr_turnstile.c:226
#11 0xffffffff8099a3ce in turnstile_wait (ts=<value optimized out>, owner=<value optimized out>, queue=<value optimized out>) at /usr/src/sys/kern/subr_turnstile.c:742
#12 0xffffffff8092f923 in __mtx_lock_sleep (c=0xfffff800020000b8, tid=18446735278394692752, opts=<value optimized out>, file=0x80 <Address 0x80 out of bounds>, line=-16843009)
at /usr/src/sys/kern/kern_mutex.c:508
#13 0xffffffff80c14138 in vm_map_lookup (var_map=0xfffffe18392db4a8, vaddr=18446741977052954624, fault_typea=2 '\002', out_entry=0xfffffe18392db4b0, object=0xfffffe18392db498,
pindex=0xfffffe18392db4a0) at /usr/src/sys/vm/vm_map.c:3843
#14 0xffffffff80c07a71 in vm_fault_hold (map=0xfffff80002000000, vaddr=18446741977052954624, fault_type=<value optimized out>, fault_flags=0, m_hold=0x0) at /usr/src/sys/vm/vm_fault.c:255
#15 0xffffffff80c07957 in vm_fault (map=0xfffff80002000000, vaddr=<value optimized out>, fault_type=2 '\002', fault_flags=128) at /usr/src/sys/vm/vm_fault.c:217
#16 0xffffffff80d99849 in trap_pfault (frame=0xfffffe18392db810, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:767
#17 0xffffffff80d99070 in trap (frame=0xfffffe18392db810) at /usr/src/sys/amd64/amd64/trap.c:455
#18 0xffffffff80d7dd22 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#19 0xffffffff80d972cd in copyin () at /usr/src/sys/amd64/amd64/support.S:292
#20 0xffffffff8099bb5f in uiomove_faultflag (cp=<value optimized out>, n=<value optimized out>, uio=0xfffffe18392dbab0, nofault=<value optimized out>) at /usr/src/sys/kern/subr_uio.c:194
#21 0xffffffff809a53ba in pipe_write (fp=0xfffff80adc4e2640, uio=0xfffffe18392dbab0, active_cred=<value optimized out>, flags=8, td=0x0) at /usr/src/sys/kern/sys_pipe.c:1215
#22 0xffffffff809a1297 in dofilewrite (td=0xfffff8002e61d490, fd=1, fp=0xfffff80adc4e2640, auio=0xfffffe18392dbab0, offset=<value optimized out>, flags=0) at file.h:307
#23 0xffffffff809a0fc8 in kern_writev (td=0xfffff8002e61d490, fd=1, auio=0xfffffe18392dbab0) at /usr/src/sys/kern/sys_generic.c:467
#24 0xffffffff809a0f53 in sys_write (td=<value optimized out>, uap=<value optimized out>) at /usr/src/sys/kern/sys_generic.c:382
#25 0xffffffff80d9a0bb in amd64_syscall (td=0xfffff8002e61d490, traced=0) at subr_syscall.c:133
#26 0xffffffff80d7e00b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
#27 0x0000000800b35afc in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) frame 10
#10 0xffffffff80999949 in propagate_priority (td=<value optimized out>) at /usr/src/sys/kern/subr_turnstile.c:226
226 panic("sleeping thread");
(kgdb) l
221 if (TD_IS_SLEEPING(td)) {
222 printf(
223 "Sleeping thread (tid %d, pid %d) owns a non-sleepable lock\n",
224 td->td_tid, td->td_proc->p_pid);
225 kdb_backtrace_thread(td);
226 panic("sleeping thread");
227 }
228
229 /*
230 * If this thread already has higher priority than the
(kgdb) tid 100702
[Switching to thread 624 (Thread 100702)]#0 sched_switch (td=0xfffff8001797a920, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1933
1933 cpuid = PCPU_GET(cpuid);
(kgdb) p cpuid
No symbol "cpuid" in current context.
(kgdb) quit
# exit
Script done on Mon Mar 10 17:59:07 2014

Glen

Konstantin Belousov

unread,
Mar 10, 2014, 3:01:12 PM3/10/14
to
On Mon, Mar 10, 2014 at 02:05:08PM -0400, Glen Barber wrote:
> Unread portion of the kernel message buffer:
> Sleeping thread (tid 100702, pid 24712) owns a non-sleepable lock

Would be nice to see the full message before and panic from the console.
From what I see, this is a lock leak, I forgot to unlock the map.
It is nice that it is so simple to reproduce the issue in your setup.

Try this update.
diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c
index 07d63f8..9633e34 100644
--- a/sys/kern/subr_trap.c
+++ b/sys/kern/subr_trap.c
@@ -157,6 +157,8 @@ userret(struct thread *td, struct trapframe *frame)
td->td_rw_rlocks));
KASSERT((td->td_pflags & TDP_NOFAULTING) == 0,
("userret: Returning with pagefaults disabled"));
+ KASSERT((td->td_pflags & TDP_DEVMEMIO) == 0,
+ ("userret: Returning with /dev/mem i/o leaked"));
KASSERT(td->td_no_sleeping == 0,
("userret: Returning with sleep disabled"));
KASSERT(td->td_pinned == 0 || (td->td_pflags & TDP_CALLCHAIN) != 0,
diff --git a/sys/sys/proc.h b/sys/sys/proc.h
index fce1f8a..e7cd022 100644
--- a/sys/sys/proc.h
+++ b/sys/sys/proc.h
@@ -424,6 +424,7 @@ do { \
#define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */
#define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */
#define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */
+#define TDP_DEVMEMIO 0x20000000 /* Accessing memory for /dev/mem */

/*
* Reasons that the current thread can not be run yet.
diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c
index 4a6495f..ab48462 100644
--- a/sys/vm/vm_fault.c
+++ b/sys/vm/vm_fault.c
@@ -269,6 +269,10 @@ RetryFault:;
map_generation = fs.map->timestamp;

if (fs.entry->eflags & MAP_ENTRY_NOFAULT) {
+ if ((curthread->td_pflags & TDP_DEVMEMIO) != 0) {
+ vm_map_unlock_read(fs.map);
+ return (KERN_FAILURE);
+ }
panic("vm_fault: fault on nofault entry, addr: %lx",
(u_long)vaddr);
}

Glen Barber

unread,
Mar 10, 2014, 3:10:23 PM3/10/14
to
On Mon, Mar 10, 2014 at 09:01:12PM +0200, Konstantin Belousov wrote:
> On Mon, Mar 10, 2014 at 02:05:08PM -0400, Glen Barber wrote:
> > Unread portion of the kernel message buffer:
> > Sleeping thread (tid 100702, pid 24712) owns a non-sleepable lock
>
> Would be nice to see the full message before and panic from the console.

I will include it in the future.

> From what I see, this is a lock leak, I forgot to unlock the map.
> It is nice that it is so simple to reproduce the issue in your setup.
>
> Try this update.
>

I will have the machine updated with this patch in the next few minutes.

Thank you.

Glen

0 new messages