Re: FreeBSD for serious performance?
Dieter BSD
> the last Alpha to be produced was shipped way back in 2004... eight years
> ago... with a top speed of 1.3 GHz. I now have a cheap little media player
> thingy sitting on my desk, and _each_ of its two cores runs faster than that.
> In short, Alphas hardly constitute high-end hardware in this day and age.
So clock rate is the only thing that matters in your world?
In my world, high-end means high quality. It doesn't necessarily
mean fast, or recent.
Warren writes:
> A. Use ahci(4):
> "The ada device driver takes full advantage of NCQ, when supported."
Achi and siis support NCQ, but neither attach to the nforce4-ultra,
which does support NCQ. I knew that NCQ would be required for acceptable
performance and gave up other useful features to get it. Silly me,
assuming that the performance orientated version of BSD would
support such an essential performance feature on a very popular chipset.
Linux has had NCQ on nforce4 since Oct 2006.
Without NCQ, writes are slower than USB2, even with purely sequential
writes (to the raw drive, no filesystem, so no seeking other then to
the next track):
device r/s w/s kr/s kw/s wait svc_t %b controller
ad6 1726.5 0.0 217535.0 0.0 1 0.6 96 ata3-master
ad6 0.0 109.5 0.0 13794.8 1 9.1 100 ata3-master
da0 249.9 0.0 15741.4 0.0 1 6.0 100 umass-sim0
ad6 is ST3000DM001-9YN166 on nforce4-ultra chipset
da0 is ST3000DM001-9YN166 USB3 disk on a USB2 port (from nforce)
Sorry no write performance data for da0, I yanked the USB-to-SATA bridge
off after doing some testing. The bridges they're currently shipping
are slightly less crappy than they used to be, but they are still crap.
> B. Use GPT, which does not have the CHS baggage. It is easier and more
> versatile. My systems with GPT disks don't complain about track
> alignment. Or maybe that's ahci(4)'s doing.
I never found a way to boot from different partitions, much less
different disks with GPT. So I use NetBSD's MBR for disks I want
to boot from. FreeBSD's and NetBSD's fdisk are both broken.
Building MBRs by hand is such fun. Non-boot disks with multiple
partitions use GPT. Big data disks get newfs-ed directly, no
partitioning.
The useless CHS baggage hangs around for decades, but useful
hardware loses all support 5 nanoseconds after the last machine
is sold. Other useful hardware waits years hoping to get support.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Adrian Chadd
ultra chipset."
So this has changed from "FreeBSD doesn't do NCQ" to "FreeBSD doesn't
do NCQ on my particular desktop-aimed motherboard chipset." They're
slightly different in scope, wouldn't you agree?
Please file a PR and see if that can get resolved. I personally have
no idea about the storage side of things so I don't know if it's a
device id or whether there's something more complicated than that.
Thanks,
Adrian
b.f.
>ultra chipset."
>
>So this has changed from "FreeBSD doesn't do NCQ" to "FreeBSD doesn't
>do NCQ on my particular desktop-aimed motherboard chipset." They're
>slightly different in scope, wouldn't you agree?
>Please file a PR and see if that can get resolved. I personally have
>no idea about the storage side of things so I don't know if it's a
>device id or whether there's something more complicated than that.
older Nvidia chipsets, but lack some SATA-specific features like NCQ.
Before AHCI was widely adopted, there were a variety of different
interfaces on early SATA controllers. When Alexander and Scott were
rewriting parts of the ATA and CAM code in FreeBSD, they had a limited
amount of time and resources, so they concentrated on the newer
ahci(4), mvs(4), and siis(4), not the many different interfaces of the
earlier SATA controllers, although they invited people to add some of
these on their own. The situation is also complicated by the fact that
some of these earlier controllers had bugs, so even in Linux, where a
larger number of the earlier controllers are more fully supported,
some of the drivers disable NCQ and some other features by default,
although the controllers are supposed to support them, e.g.:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=e2e031eb09760c36099ac127eeb175e06d257aef
Of course, it is now possible to buy relatively inexpensive add-in
SATA controller cards that are fully supported by one of the newer
drivers and perform better than most of the old controllers.
b.
Warren Block
>> B. Use GPT, which does not have the CHS baggage. It is easier and more
>> versatile. My systems with GPT disks don't complain about track
>> alignment. Or maybe that's ahci(4)'s doing.
>
> I never found a way to boot from different partitions, much less
> different disks with GPT.
systems I multiboot do not support GPT anyway. If you're booting
multiple versions of FreeBSD, see gpart(8) for some partition attributes
that may help. Or even consider ZFS boot environments.
Ronald F. Guilmette
In message <2012120909...@gmx.com>,
>Ronald writes:
>> the last Alpha to be produced was shipped way back in 2004... eight years
>> ago... with a top speed of 1.3 GHz. I now have a cheap little media player
>> thingy sitting on my desk, and _each_ of its two cores runs faster than that.
>> In short, Alphas hardly constitute high-end hardware in this day and age.
>
>So clock rate is the only thing that matters in your world?
As regards to reliability, except for the occasional low-level quirk (which
is usually taken care of for me by the kernel guys) I've never had a processor
fail on me. Once, about five or six years ago I accidentally burnt up an
Athlon XP (by not having the heatsink properly seated) but that was entirely
my fault.
>I never found a way to boot from different partitions, much less
>different disks with GPT.
most sincerly hope that you are either joking or mistaken about this.
>The useless CHS baggage hangs around for decades, but useful
>hardware loses all support 5 nanoseconds after the last machine
>is sold. Other useful hardware waits years hoping to get support.
Regards,
rfg
Adrian Chadd
support it any longer.
If someone wanted to stand up and resurrect it, support it, etc; I
doubt the FreeBSD project would complain.
The same thing is happening with ia64. Marcel still cares and he still
does a lot of ia64 heavy lifting.
Adrian
Dieter BSD
Adrian writes:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=e2e031eb09760c36099ac127eeb175e06d257aef
which is:
The mcp61 has bug with ncq.
- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA), SWNCQ },
- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA2), SWNCQ },
- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA3), SWNCQ },
+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA), GENERIC },
+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA2), GENERIC },
+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA3), GENERIC },
http://en.wikipedia.org/wiki/GeForce_6_Series
says: "MCP61 introduced a bug in the SATA NCQ implementation.
As a result, Nvidia employees have contributed code to disable NCQ operations under Linux."
I have not found a description of the bug.
But FreeBSD says I have:
<nVidia nForce CK804 SATA300 controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xcc00-0xcc0f
mem 0xfebfb000-0xfebfbfff irq 21 at device 7.0 on pci0
<nVidia nForce CK804 SATA300 controller> port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xb800-0xb80f
mem 0xfebfa000-0xfebfafff irq 22 at device 8.0 on pci0
The CK804 (a few lines higher in the source file) did not get changed to GENERIC.
As far as I can figure out, the bug (whatever it is) is limited to the MCP61.
b.f.
>
>
>
>which is:
>
>The mcp61 has bug with ncq.
>- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA), SWNCQ },
>- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA2), SWNCQ },
>- { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA3), SWNCQ },
>+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA), GENERIC },
>+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA2), GENERIC },
>+ { PCI_VDEVICE(NVIDIA, PCI_DEVICE_ID_NVIDIA_NFORCE_MCP61_SATA3), GENERIC },
>
>http://en.wikipedia.org/wiki/GeForce_6_Series
>says: "MCP61 introduced a bug in the SATA NCQ implementation.
>As a result, Nvidia employees have contributed code to disable NCQ operations under Linux."
>I have not found a description of the bug.
>
>But FreeBSD says I have:
>
><nVidia nForce CK804 SATA300 controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xcc00-0xcc0f
>mem 0xfebfb000-0xfebfbfff irq 21 at device 7.0 on pci0
><nVidia nForce CK804 SATA300 controller> port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xb800-0xb80f
>mem 0xfebfa000-0xfebfafff irq 22 at device 8.0 on pci0
>
>The CK804 (a few lines higher in the source file) did not get changed to GENERIC.
>As far as I can figure out, the bug (whatever it is) is limited to the MCP61.
-- I was just mentioning the bug as an example that some of the older
SATA controllers have problems with useful features like MSI, power
management, port multiplication, or NCQ that the controllers are
supposed to support -- so that if anyone were considering creating a
FreeBSD driver for one of these controllers based on our existing
FreeBSD drivers, the product datasheets, and the Linux drivers at:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=tree;f=drivers/ata;h=32325a5432d90c22dd1f19ee6c811af2145fc031;hb=HEAD
in order to get more than legacy mode support for the controller, then
they would be wise to look at the Linux drivers to see if the features
that they want actually work -- and weigh the investment of time and
effort against the alternative of buying a new storage card that works
well with one of our newer drivers.
b.
Dieter BSD
>>> the last Alpha to be produced was shipped way back in 2004... eight years
>>> thingy sitting on my desk, and _each_ of its two cores runs faster than tha\
t.
>>> In short, Alphas hardly constitute high-end hardware in this day and age.
>>
>> So clock rate is the only thing that matters in your world?
>
> Yea, pretty much.
>
> As regards to reliability, except for the occasional low-level quirk (which
r
> fail on me. Once, about five or six years ago I accidentally burnt up an
> Athlon XP (by not having the heatsink properly seated) but that was entirely
> my fault.
I care about data integrity, so things like ECC are on my must-have list.
I suspect that your "cheap little media player thingy" doesn't have ECC.
If you don't care about getting the correct answer you can have infinite
speed.
A high clock rate doesn't help when some device driver does
block_all_interrupts();
while(1)
DELAY(MIGHT_AS_WELL_BE_FOREVER);
At least four device drivers have caused me to lose data this way.
Not what I call high performance.
>> In my world, high-end means high quality. It doesn't necessarily
>> mean fast, or recent.
Data integrity, and yes, reliability, that sort of thing.
Performance-wise, in most cases I don't expect to get 99.9999%
of the theoretical best case, I'm usually happy with 90-95%.
But without NCQ I'm only getting ~6% of what I should be getting.
Pretty pathetic for an OS that claims to be all about performance.
All the more so when the crappy OSes do support NCQ on that chip.
It's not some rare, obscure chip. Lots of boxes have it.
>> I never found a way to boot from different partitions, much less
>> different disks with GPT.
>
> Having just been recently convinced to switch over to GPT (from MBR) I do
> most sincerly hope that you are either joking or mistaken about this.
I am neither joking nor mistaken. I looked but could not find a way.
I am not claiming that a way does not exist, merely that I couldn't
find it. Perhaps there wasn't a way when I looked (it was awhile ago)
but does exist now? I have never been a fan of MBRs, they are for
pee-cees with the expected ugly kludges and limitations, real machines
don't use them. GPT isn't perfect, but it seems much nicer than MBRs.
Warren writes:
> Grub (or grub2) can do it.
Back when I was triple-booting FreeBSD, NetBSD and Linux I used
grub (rev number forgotten). It was supposed to be able to boot
BSD from a partition but I never got that to work. I had to have it
boot the MBR of a different disk which then booted BSD. I wrote
3 little shell scripts that edited grub's menu to change the default.
So I could be running FreeBSD and type "boot_netbsd", and go have lunch
while it rebooted. Other than not booting BSD from a partition it
wasn't that bad for something that smells of penguin.
> If you're booting multiple versions of FreeBSD, see gpart(8)
> for some partition attributes that may help.
You mean the bootme bootonce stuff? That looks promising, assuming
I can manage to decode the man page, and figure out what it actually
does. Mostly multiple versions of FreeBSD. (for example 7.0 had a couple
of bugs preventing it from booting, so having 6 still available was
essential.) I no long need Linux (YEA!!!), and after a certain fubar
incident have declared a Linux Free Zone. It would be nice if I could
also boot Net/Open/Dragonfly. I don't see a way to boot multiple disks,
but GPT allows enough partitions that I probably won't care.
Not sure if the bootme bootonce stuff wasn't there yet when I looked,
or if I just missed it. Thanks for the pointer.
> Or even consider ZFS boot environments.
I plan to stick with FFS w/softdeps.
Adrian Chadd
* any references to driver code that exhibits that very broken behaviour;
* any patches from you to implement NCQ on your nforce chipset;
* any offer of incentive to any developer to add that support.
Now, (1) is definitely worrying but as you've not provided any actual
information, the level of helpfulness of this comment is 0.
(2) and (3) would go a long way to making FreeBSD "better".
FreeBSD is only as much as the community of contributors and
developers have the time and energy to make it be. If you want FreeBSD
to be different/better/something, you have to step up.
Adrian
Peter Jeremy
>I care about data integrity, so things like ECC are on my must-have list.
Intel Xeon, Oracle/Sun SPARC) and some desktop CPUs (most AMD x86 chips).
>A high clock rate doesn't help when some device driver does
>
>block_all_interrupts();
>while(1)
> DELAY(MIGHT_AS_WELL_BE_FOREVER);
>
>At least four device drivers have caused me to lose data this way.
>Data integrity, and yes, reliability, that sort of thing.
systems manage that.
>But without NCQ I'm only getting ~6% of what I should be getting.
complain that that FreeBSD doesn't support NCQ on an old, low-end
(consumer-grade) chipset that doesn't support ECC.
>It's not some rare, obscure chip. Lots of boxes have it.
>>> I never found a way to boot from different partitions, much less
>>> different disks with GPT.
written the code to support multiple boot partitions or disks. Note
that most BIOS's allow you to select the boot disk - which is a
workaround.
--
Peter Jeremy
Wojciech Puchar
>> So clock rate is the only thing that matters in your world?
>
> Yea, pretty much.
lose even at the same clock speed.
Stefan Esser
> .... I'm not seeing:
>
> * any references to driver code that exhibits that very broken behaviour;
> * any patches from you to implement NCQ on your nforce chipset;
> * any offer of incentive to any developer to add that support.
>
> Now, (1) is definitely worrying but as you've not provided any actual
> information, the level of helpfulness of this comment is 0.
> (2) and (3) would go a long way to making FreeBSD "better".
The cause of the low write performance is the disabled write cache.
Enabling the write cache is unsafe on SATA drives (with or without
NCQ), since they do not make any guarantees that nearby data is not
lost if power fails during a disk write. It never happened to me,
but there is a reason that SAS drives have less capacity, much lower
BER (one to two magnitudes) and are more expensive than SATA drives.
The solution to the performance problem is simple: Turn on the write
cache. If the data is valuable, then SAS is the solution to both the
performance and the inherent reliability problems. If SATA with NCQ
really provides acceptable reliability, then a cheap SATA controller
with NCQ support in FreeBSD might also be an option, at a cost that
would pay one developer hour. Asking Nvidia to release the confidential
documentation for their chip-set might help, but I doubt that there is
much interest to add support for NCQ to an obsolete chip-set, today,
unless you pay a developer (and even then ...).
If all these don't work for you, then you may really be better served
by Linux with the drivers donated by Nvidia.
And if you buy a new system, you may consider choosing components
from vendors that do not provide binary blobs for selected operating
systems, but publish the necessary documentation for driver writers
(or which develop open source drivers for FreeBSD, not only for Linux).
There are so many ways to solve your problem
Wojciech Puchar
> Enabling the write cache is unsafe on SATA drives (with or without
> NCQ), since they do not make any guarantees that nearby data is not
> lost if power fails during a disk write. It never happened to me,
> but there is a reason that SAS drives have less capacity, much lower
> BER (one to two magnitudes) and are more expensive than SATA drives.
then.
> The solution to the performance problem is simple: Turn on the write
> cache. If the data is valuable, then SAS is the solution to both the
solution.
> would pay one developer hour. Asking Nvidia to release the confidential
> documentation for their chip-set might help, but I doubt that there is
> much interest to add support for NCQ to an obsolete chip-set, today,
> unless you pay a developer (and even then ...).
Stefan Esser
>> The cause of the low write performance is the disabled write cache.
>> Enabling the write cache is unsafe on SATA drives (with or without
>> NCQ), since they do not make any guarantees that nearby data is not
>> lost if power fails during a disk write. It never happened to me,
>> but there is a reason that SAS drives have less capacity, much lower
>> BER (one to two magnitudes) and are more expensive than SATA drives.
>
> interface have nothing to do. Both allows you to force writes now and then.
drives, but with the exception of the WD Raptor they hardly ever are.
SATA drives are optimized for storage density at low cost. SAS drives
have completely different design targets, in general (or their higher
price could not be satisfied).
SATA drives have been reported to lie about the completion not only
of normal writes, but also of flush commands. And with "advanced
format" (4K sector) drives, data is at risk in (logical) sectors that
are not even being written to.
There is no technical reason that SATA drives are less reliable with
regard to hardware (bit density, BER, ...) and firmware (less strict
conformance testing compared to SAS drives), but there are market
forces that have this effect.
>> The solution to the performance problem is simple: Turn on the write
>> cache. If the data is valuable, then SAS is the solution to both the
>
> If data is valuable, regular and well done backup practice is the only
> solution.
for TCQ) of SAS drives. There were a number of consumer grade drives
that supported tags, but where data was at risk due to firmware bugs.
Backups are a way to recover from loss of data. Use of devices that
are magnitudes more reliable improves short-term availability of the
data and reduces the risk that a recovery will be needed. Don't mix
immediate availability (as required by processing requirements) with
availability of a recovery path.
>> would pay one developer hour. Asking Nvidia to release the confidential
>> documentation for their chip-set might help, but I doubt that there is
>> much interest to add support for NCQ to an obsolete chip-set, today,
>> unless you pay a developer (and even then ...).
>
> Even without this, i've never seen properly working NVidia hardware. ANY
other vendors' products. I guess this is due to the market segment
they target (not sure whether this still applies for their expensive
GPUs targeted at high performance computing).
Regards, STefan
Wojciech Puchar
> regard to hardware (bit density, BER, ...) and firmware (less strict
> conformance testing compared to SAS drives), but there are market
> forces that have this effect.
But actually it is mostly propaganda to sell nearly same thing at 5x
price.