kernel panic on boot with 4G RAM
Steve Shorter
I need an NFS server with 4G ram. When I boot a 4.5-PRE kernel
it panics during the boot process, not always at the same place though.
My first instinct is bad hardware because of the lack of consistency
in panic location, however I was wondering if there were some issues
with kernel tuning for machines with large memory that I should be aware of.
This machine boots OK with 3G or less.
Here is a trace and panic with kernel config etc...
So... hardware or kernel config or kernel bug???
Your help is greatly appreciated.
-steve
12: page fault while in kernel mode
fault virtual address = 0x45d3
fault code = supervisor write, page not present
instruction pointer = 0x8:0xc01f5688
stack pointer = 0x10:0xefaf7e84
frame pointer = 0x10:0xefaf7e88
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 6 (sh)
interrupt mask = net bio cam
kernel: type 12 trap, code=0
Stopped at vm_page_unqueue_nowakeup+0x40: decl 0(%eax)
db> trace
vm_page_unqueue_nowakeup(c179ce84,e77245e0,e7727e80,80bb000,efaf9000) at vm_page_unqueue_nowakeup+0x40
vm_fault(e7727e80,80bb000,3,8,e77245e0) at vm_fault+0x236
trap_pfault(efaf7fa8,1,80bbc4c,0,80bc1d8) at trap_pfault+0x96
trap(2f,2f,2f,80c1000,80bc1d8) at trap+0x1d3
calltrap() at calltrap+0x11
--- trap 0xc, eip = 0x8050dfa, esp = 0xbfbffbdc, ebp = 0xbfbffc04 ---
and another
panic: vm_page_remove(): page not found in hash
Debugger("panic")
Stopped at Debugger+0x34: movb $0,in_Debugger.426
db> trace
Debugger(c022419b) at Debugger+0x34
panic(c0237de0,c17a64c0,efb03e20,c01f5bf4,c17a64c0) at panic+0x70
vm_page_remove(c17a64c0,c17a64c0) at vm_page_remove+0x73
vm_page_free_toq(c17a64c0) at vm_page_free_toq+0x90
vm_object_collapse(c026cf00) at vm_object_collapse+0x198
vm_object_deallocate(c026c900) at vm_object_deallocate+0x170
vm_map_entry_delete(e7727d00,c0265d10,e7727d00,bfc00000,0) at vm_map_entry_delete+0x33
vm_map_delete(e7727d00,0,bfc00000,e7727d00,1) at vm_map_delete+0x171
vm_map_remove(e7727d00,0,bfc00000,e7727d6c,0,bfc00000) at vm_map_remove+0x6a
exit1(e77242a0,0,efb03fa0,c020e1da,e77242a0) at exit1+0x220
exit1(e77242a0,efb03f80,ffffffff,0,8072968) at exit1
syscall2(2f,2f,2f,8072968,0) at syscall2+0x166
Xint0x80_syscall() at Xint0x80_syscall+0x25
These are boot up messages
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.5-PRERELEASE #0: Mon Dec 31 13:33:03 EST 2001
ro...@homer.lets.net:/usr/local/4.5/src/sys/compile/NFS
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 797475848 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (797.48-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x686 Stepping = 6
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory = 4227858432 (4128768K bytes)
avail memory = 4120342528 (4023772K bytes)
Preloaded elf kernel "kernel.4.5.test" at 0xc02bc000.
Pentium Pro MTRR support enabled
Using $PIR table, 10 entries at 0xc00f51e0
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GR graphics accelerator> at 1.0 irq 0
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0xd400-0xd43f mem 0xfe900000-0xfe9fffff,0xfeafe000-0xfeafefff irq 10 at device 4.0 on pci0
fxp0: Ethernet address 00:e0:81:01:bf:ca
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel Pro 10/100B/100+ Ethernet> port 0xd000-0xd03f mem 0xfe700000-0xfe7fffff,0xfeafd000-0xfeafdfff irq 9 at device 5.0 on pci0
fxp1: Ethernet address 00:e0:81:01:bf:cb
inphy1: <i82555 10/100 media interface> on miibus1
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
pci0: <Unknown PCI ATA controller> at 15.1
pci0: <OHCI USB controller> at 15.2 irq 0
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
sym0: <1010-33> port 0xe400-0xe4ff mem 0xfebd8000-0xfebd9fff,0xfebe0000-0xfebe03ff irq 15 at device 6.0 on pci1
sym0: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking
sym0: open drain IRQ line driver, using on-chip SRAM
sym0: using LOAD/STORE-based firmware.
sym0: handling phase mismatch from SCRIPTS.
sym1: <1010-33> port 0xe800-0xe8ff mem 0xfebf0000-0xfebf1fff,0xfebf8000-0xfebf83ff irq 11 at device 6.1 on pci1
sym1: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking
sym1: open drain IRQ line driver, using on-chip SRAM
sym1: using LOAD/STORE-based firmware.
sym1: handling phase mismatch from SCRIPTS.
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drived on fdc0 drive 0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x30 on isa0
sio0: type 16550A, console
IP packet filtering initialized, divert disabled, rule-based forwarding disabled, default to accept, logging limited to 100 packets/entry by default
Waiting 15 seconds for SCSI devices to settle
(noperiph:sym0:0:-1:-1): SCSI BUS reset delivered.
(noperiph:sym1:0:-1:-1): SCSI BUS reset delivered.
da1 at sym1 bus 0 target 1 lun 0
da1: <JetStor II-LVD > Fixed Direct Access SCSI-2 device
da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da1: 211528MB (433209344 512 byte sectors: 255H 63S/T 26966C)
da0 at sym0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS_V__9_WLS 0230> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 62, 16bit), Tagged Queueing Enabled
da0: 8755MB (17930694 512 byte sectors: 255H 63S/T 1116C)
da2 at sym1 bus 0 target 1 lun 1
da2: <JetStor II-LVD > Fixed Direct Access SCSI-2 device
da2: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da2: 519424MB (1063780352 512 byte sectors: 255H 63S/T 681C)
da3 at sym1 bus 0 target 4 lun 0
da3: <JetStor II-LVD > Fixed Direct Access SCSI-2 device
da3: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da3: 211528MB (433209344 512 byte sectors: 255H 63S/T 26966C)
da4 at sym1 bus 0 target 4 lun 1
da4: <JetStor II-LVD > Fixed Direct Access SCSI-2 device
da4: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da4: 519424MB (1063780352 512 byte sectors: 255H 63S/T 681C)
Mounting root from ufs:/dev/da0s1a
Here is kernel config file
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.246.2.34 2001/08/12 13:13:46 joerg Exp $
machine i386
cpu I686_CPU
ident NFS
maxusers 0
#
# Enable the kernel debugger.
#
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
# Set the size of the mbuf KVM reservation, in clusters. This is scaled
# by approximately 2048 bytes. The system will auto-size the mbuf area
# if this options is not specified or set to 0.
#
#options NMBCLUSTERS=0
# Tune the kernel malloc area parameters. VM_KMEM_SIZE represents the
# minimum, in bytes, and is typically (12*1024*1024) (12MB).
# VM_KMEM_SIZE_MAX represents the maximum, typically 200 megabytes.
# VM_KMEM_SIZE_SCALE can be set to adjust the auto-tuning factor, which
# typically defaults to 4 (kernel malloc area size is physical memory
# divided by the scale factor).
#
#options VM_KMEM_SIZE="(10*1024*1024)"
#options VM_KMEM_SIZE_MAX="(100*1024*1024)"
#options VM_KMEM_SIZE_SCALE="4"
# Tune the buffer cache maximum KVA reservation, in bytes. The maximum is
# usually capped at 200 MB, effecting machines with > 1GB of ram. Note
# that the buffer cache only really governs write buffering and disk block
# translations. The VM page cache is our primary disk cache and is not
# effected by the size of the buffer cache.
#
#options VM_BCACHE_SIZE_MAX="(100*1024*1024)"
# Tune the swap zone KVA reservation, in bytes. The default is typically
# 70 MB, giving the system the ability to manage a maximum of 28GB worth
# of swapped out data.
#
#options VM_SWZONE_SIZE_MAX="(50*1024*1024)"
options INET #InterNETworking
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options SOFTUPDATES #Enable FFS soft updates support
#options MFS #Memory Filesystem
options NFS #Network Filesystem
#options NFS_ROOT #NFS usable as root device, NFS required
#options NFS_NOSERVER #Disable the NFS-server code.
options PROCFS #Process filesystem
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
#options KTRACE #ktrace(1) support
#options SYSVSHM #SYSV-style shared memory
#options SYSVMSG #SYSV-style message queues
#options SYSVSEM #SYSV-style semaphores
options ICMP_BANDLIM #Rate limit bad replies
#options NO_SWAPPING
options NSWAPDEV=1
options QUOTA #enable disk quotas
options PANIC_REBOOT_WAIT_TIME=16
options NMBCLUSTERS=1024
# IPFW firewall
options IPFIREWALL #firewall
options IPFIREWALL_VERBOSE #print information about
# dropped packets
#options IPFIREWALL_FORWARD #enable transparent proxy support
options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default
# ipfilter firewall
#options IPFILTER #ipfilter support
#options IPFILTER_LOG #ipfilter logging
#options IPFILTER_DEFAULT_BLOCK #block all packets by default
device isa
device pci
# Floppy drives
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
# SCSI Controllers
#device ahc # AHA2940 and onboard AIC7xxx devices
device sym # NCR/Symbios Logic (newer chipsets)
#options SYM_SETUP_LP_PROBE_MAP=0x40
# Allow ncr to attach legacy NCR devices when
# both sym and ncr are configured
# SCSI peripherals
device scbus # SCSI bus (required)
device da # Direct Access (disks)
# atkbdc0 controls both the keyboard and the PS/2 mouse
#device atkbdc0 at isa? port IO_KBD
#device atkbd0 at atkbdc? irq 1 flags 0x1
#device vga0 at isa?
# syscons is the default console driver, resembling an SCO console
#device sc0 at isa? flags 0x100
# Floating point support - do not disable.
device npx0 at nexus? port IO_NPX irq 13
# Serial (COM) ports
device sio0 at isa? port IO_COM1 flags 0x30 irq 4
options CONSPEED=57600 # speed for serial console
#device cy
#options CY_PCI_FASTINTR # Use with cy_pci unless irq is shared
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
# Pseudo devices - the number indicates how many units to allocate.
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
pseudo-device pty # Pseudo-ttys (telnet etc)
#pseudo-device vn #Vnode driver (turns a file into a device)
#pseudo-device md # Memory "disks"
To Unsubscribe: send mail to majo...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Terry Lambert
> > You really need to change the allocation of swap page descriptors
> > in /sys/i386/machdep.c (among other things).
>
> Is this a trivial change? I don't know much about
> FreeBSD kernel internals but can edit source with some guidance OR
> can these changes be effected in the kernel config?
No, but it was discussed in detail on the FreeBSD-current list
last month, so your best bet would be to look there.
> How about changing NKPT in sys/i386/include/pmap.h to
> something bigger? .. like 60?
No. The scaling would be too large. By default, the amount of
page mapings is ~200M. This can be grown automatically by the
kernel routines, so don't worry about it. The problem is that
with that much memory, you will suck up more page tables than
can possibly be used, since you can only map 4G, and the value
does not asymptotically approach that, if you use the unchanged
machdep.c. Matt Dillon posted a patch that makes the values
more reasonably autosized based on physical memory size, but is
not perfect (it can't be, for reasons discussed in the thread
that I've already referred you to, above).
> What "other things" should be done to tune a high memory
> machine, or to get it to boot.
Read the thread. We all spent a lot of time typing it. It's a
fairly exhastive listing of what should/can be tuned.
> > I personally suggest compiling your kernel with 3G KVA/1G User,
> > instead of the default of 1G KVA/3G User.
> >
>
> Ok. This can be done with
>
> options KVA_PAGES=768
Depends on what FreeBSD version you are running. You are posting
to the -hackers list and not the -current or -stable list, so you
will probably also have to adjust the kernbase address in the
/sys/conf/ldscript.i386, as well. As I said before, look at the
handbook, though it is somewhat incomplete.
> What about the other KVA stuff in kernel config
>
> options VM_BCACHE_SIZE_MAX=
>
> The comments in LINT mention that this is capped at 260M
> "effecting > 1G of ram". Should this be increased with machines with 4G?
Depends. You are probably more worried about having mbufs, not
FS bufs, which are allocated out of a totally seperate pool, not
the buffer cache.
-- Terry