Weird performance behaviour in 7.0
Bruce Albrecht
first AMD-64 system and my first ZFS base system, and I'm seeing
something really strange. Every now and then, processes run really
slowly, like at 1/600th (or worse) than it ought to, but it racks up
the CPU time as though it's running full tilt.
For example, I have this little test program that just does a tight
loop, and most times, it takes about 3 seconds to complete, but right
now, it's taking about 2000 seconds to complete. Right now, it's
consistently running slow, but sometimes it will run slow, but I can
terminate it and start another one which will run at normal speed.
Currently sysctl reports:
dev.cpu.0.freq: 2400
dev.cpu.0.temperature: 55
dev.cpu.1.temperature: 57
dev.cpu.2.temperature: 51
dev.cpu.3.temperature: 54
What might be causing this? What information is pertinent for
inclusion in a problem report?
Ivan Voras
> This is my first Q6600 based system, and my first 7.0 system, and my
> first AMD-64 system and my first ZFS base system, and I'm seeing
> something really strange. Every now and then, processes run really
> slowly, like at 1/600th (or worse) than it ought to, but it racks up
> the CPU time as though it's running full tilt.
Are you running powerd?
Also, cpufreq driver was included in the GENERIC kernel in 7.0 (it was
optional in 6.x), so it might also cause problems somehow. To verify
this, try building a kernel without cpufreq.
Also, does your BIOS do any power-saving gimmicks?
Richard Todd
> This is my first Q6600 based system, and my first 7.0 system, and my
> first AMD-64 system and my first ZFS base system, and I'm seeing
> something really strange. Every now and then, processes run really
> slowly, like at 1/600th (or worse) than it ought to, but it racks up
> the CPU time as though it's running full tilt.
>
> For example, I have this little test program that just does a tight
> loop, and most times, it takes about 3 seconds to complete, but right
> now, it's taking about 2000 seconds to complete. Right now, it's
> consistently running slow, but sometimes it will run slow, but I can
> terminate it and start another one which will run at normal speed.
This wouldn't by any chance be an Intel 965-chipset-based motherboard
with 4G or more of memory, would it? Because there's an interesting
little bug in the BIOS on some of those boards which causes the
cache-control registers to incorrectly declare a chunk of main memory
as uncacheable. This results in random slowdowns depending on whether
your process lands in the "bad" zone of memory or not. See
http://article.gmane.org/gmane.os.freebsd.stable/50135/ for more details.
br...@zuhause.mn.org
Bingo! This is a Intel DG965WH with 4 GB of memory. I don't think I
can downgrade to the 1669 firmware because of the processor I'm
using. The Fedora thread says that there's a hack to do the following
in linux to fix the "bad" zone
echo "base=0x1a8000000 size=0x4000000 type=write-back" >| /proc/mtrr
What would be the FreeBSD equivalent? The output of memcontrol list is
0x0/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x10000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x20000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x30000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x40000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x50000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x60000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x70000/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x80000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x84000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x88000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x8c000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x90000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x94000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x98000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0x9c000/0x4000 BIOS write-back fixed-base fixed-length set-by-firmware active
0xa0000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xa4000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xa8000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xac000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xb0000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xb4000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xb8000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xbc000/0x4000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc0000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc1000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc2000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc3000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc4000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc5000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc6000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc7000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc8000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xc9000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xca000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xcb000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xcc000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xcd000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xce000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xcf000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd0000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd1000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd2000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd3000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd4000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd5000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd6000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd7000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd8000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xd9000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xda000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xdb000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xdc000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xdd000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xde000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xdf000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe0000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe1000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe2000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe3000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe4000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe5000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe6000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe7000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe8000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xe9000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xea000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xeb000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xec000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xed000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xee000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xef000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf0000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf1000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf2000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf3000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf4000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf5000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf6000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf7000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf8000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xf9000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xfa000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xfb000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xfc000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xfd000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xfe000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0xff000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
Richard Todd
> Richard Todd writes:
> > This wouldn't by any chance be an Intel 965-chipset-based motherboard
> > with 4G or more of memory, would it? Because there's an interesting
> > little bug in the BIOS on some of those boards which causes the
> > cache-control registers to incorrectly declare a chunk of main memory
> > as uncacheable. This results in random slowdowns depending on whether
> > your process lands in the "bad" zone of memory or not. See
> > http://article.gmane.org/gmane.os.freebsd.stable/50135/ for more details.
>
> Bingo! This is a Intel DG965WH with 4 GB of memory. I don't think I
> can downgrade to the 1669 firmware because of the processor I'm
> using. The Fedora thread says that there's a hack to do the following
> in linux to fix the "bad" zone
> echo "base=0x1a8000000 size=0x4000000 type=write-back" >| /proc/mtrr
Yeah, but the exact numbers depend on what exactly the bad zone is, and
that seems to vary from system to system (depending on what memory is
installed and what rev of the BIOS. Let's see if we can figure it out.
It's gonna be one of the ranges in the list you posted below (the last few
ones are the important ones)
> 0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
> 0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
> 0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
> 0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
> 0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
These entries each specify a start address and a length of a range of
memory. Those ranges look a lot bigger than they really are thanks to that
leading "f" digit on all of them, which tells me that you're running in 64-bit
mode; apparently in 64-bit mode the Intel cache-control registers have fewer
active bits than the AMD64 equivalent for which the existing amd64 machdep
code in the kernel was written for, so we get 4 gratuitous extra high bits.
So, e.g., the first range is actually of length 0x80000000 (which is 2G)
starting at address zero; that one's "write-back" so cache is enabled on that
range. Ditto the next one which is length 0x40000000 (1G) at address 2G.
I'm guessing the bad ones are the two "uncachable" entries at the end, so
try doing
memcontrol clear -b 0xcf800000 -l 0xf000800000
memcontrol clear -b 0xcf700000 -l 0xf000100000
and see if that fixes things, and doesn't break anything.
Richard Todd
> > with 4G or more of memory, would it? Because there's an interesting
> > little bug in the BIOS on some of those boards which causes the
> > cache-control registers to incorrectly declare a chunk of main memory
> > as uncacheable. This results in random slowdowns depending on whether
> > your process lands in the "bad" zone of memory or not. See
> > http://article.gmane.org/gmane.os.freebsd.stable/50135/ for more details.
>
> Bingo! This is a Intel DG965WH with 4 GB of memory. I don't think I
> can downgrade to the 1669 firmware because of the processor I'm
> using. The Fedora thread says that there's a hack to do the following
> in linux to fix the "bad" zone
> echo "base=0x1a8000000 size=0x4000000 type=write-back" >| /proc/mtrr
Yeah, but the exact numbers depend on what exactly the bad zone is, and
that seems to vary from system to system (depending on what memory is
installed and what rev of the BIOS. Let's see if we can figure it out.
It's gonna be one of the ranges in the list you posted below (the last few
ones are the important ones)
> 0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
> 0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
> 0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
> 0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
> 0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
These entries each specify a start address and a length of a range of
memory. Those ranges look a lot bigger than they really are thanks to that
leading "f" digit on all of them, which tells me that you're running in 64-bit
mode; apparently in 64-bit mode the Intel cache-control registers have fewer
active bits than the AMD64 equivalent for which the existing amd64 machdep
code in the kernel was written for, so we get 4 gratuitous extra high bits.
So, e.g., the first range is actually of length 0x80000000 (which is 2G)
starting at address zero; that one's "write-back" so cache is enabled on that
range. Ditto the next one which is length 0x40000000 (1G) at address 2G.
I'm guessing the bad ones are the two "uncachable" entries at the end, so
you might try doing
memcontrol clear -b 0xcf800000 -l 0xf000800000
memcontrol clear -b 0xcf700000 -l 0xf000100000
and see if that fixes things, and doesn't break anything.
I am a bit puzzled that the writeback ranges listed in the above don't seem
to actually add up to enough to cover 4G of memory, though. This worries me,
in that you may need to add additional cache-control entries.
Before trying the memcontrol stuff above, it might help if you could boot
the system in verbose mode and record the SMAP entries that get printed
by getmemsize() in machdep.c as the system boots. If you have a serial
console, this is easy; if you don't it gets tricky, as those SMAP lines appear
before the console is fully initialized so that the SMAP lines don't show
up in dmesg later, so I had to resort to booting with -d and doing
cleverly-placed ddb breakpoints.
br...@zuhause.mn.org
>
> br...@zuhause.mn.org writes:
> > Richard Todd writes:
> > > This wouldn't by any chance be an Intel 965-chipset-based motherboard
> > > with 4G or more of memory, would it? Because there's an interesting
> > > little bug in the BIOS on some of those boards which causes the
> > > cache-control registers to incorrectly declare a chunk of main memory
> > > as uncacheable. This results in random slowdowns depending on whether
> > > your process lands in the "bad" zone of memory or not. See
> > > http://article.gmane.org/gmane.os.freebsd.stable/50135/ for more details.
> >
> > Bingo! This is a Intel DG965WH with 4 GB of memory. I don't think I
> > can downgrade to the 1669 firmware because of the processor I'm
> > using. The Fedora thread says that there's a hack to do the following
> > in linux to fix the "bad" zone
Embarrassingly, after I posted this, I realized that I was not at the
latest BIOS level, and not at a recommended level for my processor, so
I flashed it with the latest, version 1719. Although in the Fedora
thread Jussit said that it didn't fix the problem, it seems to have
fixed it for me. BTW, I was able to get the SMAP information at the
loader prompt.
I'm curious, though, should I be worried about the memcontrol list
entries that are listed as "set-by-firmware active bogus"?
Type '?' for a list of commands, 'help for wore detailed
OK smap
SMAP type=01 base=0000000000000000 len=000000000008f000
SMAP type=02 base=000000000008f000 len=0000000000011000
SMAP type=02 base=00000000000e0000 len=0000000000020000
SMAP type=01 base=0000000000100000 len=00000000cf461000
SMAP type=02 base=00000000cf561000 len=000000000000d000
SMAP type=0l base=00000000cf56e000 len=000000000009c000
SMAP type=04 base=00000000cf60a000 len=00000000000df000
SMAP type=01 base=00000000cf6e9000 len-0000000000009000
SMAP type=03 base=00000000cf6f2000 len=000000000000d000
SMAP type=01 base=00000000cf6ff000 len=0000000000001000
SMAP type=02 base=00000000cf700000 len=0000000000100000
SMAP type=02 base=00000000cf800000 len=0000000000800000
SMAP type=02 base=00000000fff00000 len=0000000000100000
SMAP type=01 base=0000000100000000 len=000000002c000000
OK
0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
0x100000000/0xf020000000 BIOS write-back set-by-firmware active bogus
0x120000000/0xf008000000 BIOS write-back set-by-firmware active bogus
br...@zuhause.mn.org
>
> br...@zuhause.mn.org writes:
> > Richard Todd writes:
> > > This wouldn't by any chance be an Intel 965-chipset-based motherboard
> > > with 4G or more of memory, would it? Because there's an interesting
> > > little bug in the BIOS on some of those boards which causes the
> > > cache-control registers to incorrectly declare a chunk of main memory
> > > as uncacheable. This results in random slowdowns depending on whether
> > > your process lands in the "bad" zone of memory or not. See
> > > http://article.gmane.org/gmane.os.freebsd.stable/50135/ for more details.
> >
> > Bingo! This is a Intel DG965WH with 4 GB of memory. I don't think I
> > can downgrade to the 1669 firmware because of the processor I'm
> > using. The Fedora thread says that there's a hack to do the following
> > in linux to fix the "bad" zone
Embarrassingly, after I posted this, I realized that I was not at the
0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
0x100000000/0xf020000000 BIOS write-back set-by-firmware active bogus
0x120000000/0xf008000000 BIOS write-back set-by-firmware active bogus
br...@zuhause.mn.org
> 0xff000/0x1000 BIOS uncacheable fixed-base fixed-length set-by-firmware active
> 0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
> 0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
> 0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
> 0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
> 0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
> 0x100000000/0xf020000000 BIOS write-back set-by-firmware active bogus
> 0x120000000/0xf008000000 BIOS write-back set-by-firmware active bogus
I hate to follow up to my own postings, but it seems to me that this
memcontrol list is missing 0x4000000 (64 MB). I'm using the onboard
video. I'm not sure if this means that it's never used by the OS, or
that I'm just lucky that I haven't yet experienced a slowdown. With
the old BIOS, I couldn't run a make release without hitting it, but as
far as I can tell, I haven't seen a problem yet even though I ran
another make release and ran the repeated tight CPU loops I was using
to test before.
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RC1 #2: Mon Jan 14 03:17:48 UTC 2008
to...@orca.zuhause.org:/usr/obj/usr/src/sys/ORCA
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2407.20-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
usable memory = 4202840064 (4008 MB)
avail memory = 4042080256 (3854 MB)
ACPI APIC Table: <INTEL DG965WH >
Richard Todd
> I'm curious, though, should I be worried about the memcontrol list
> entries that are listed as "set-by-firmware active bogus"?
>
> 0x0/0xf080000000 BIOS write-back set-by-firmware active bogus
> 0x80000000/0xf040000000 BIOS write-back set-by-firmware active bogus
> 0xc0000000/0xf010000000 BIOS write-back set-by-firmware active bogus
> 0xcf800000/0xf000800000 BIOS uncacheable set-by-firmware active bogus
> 0xcf700000/0xf000100000 BIOS uncacheable set-by-firmware active bogus
> 0x100000000/0xf020000000 BIOS write-back set-by-firmware active bogus
> 0x120000000/0xf008000000 BIOS write-back set-by-firmware active bogus
Worried that it says "bogus"? No, not really. This is, as I said earlier,
the result of an issue in the amd64 machdep.c code for handling MTRRs --
it was written for the original amd64 where the "size" field of the MTRR
was a certain width. On Core 2 Duo these fields are 4 bits shorter, so
the machdep.c code looks at 4 more bits from that register than it should and
so all the size fields have that leading "f" digit. This causes the sizes
to no longer be a power of two, which causes machdep.c to set the "bogus"
flag because those entries look funny. The "bogus" flag being set doesn't
seem to actually break anything, and as near as I can figure if you change
the memcontrol entries the correct data gets written back to the registers
even on Core2Duo, so the "bogus" warnings seem to be pretty harmless.
John Baldwin
Ahh, I will look at fixing this.
--
John Baldwin
John Baldwin
This should be fixed now in HEAD.
--
John Baldwin