FreeBSD 8.0 Performance (at Phoronix)
David Naylor
Phoronix recently published a comparative benchmark[1] for FreeBSD 7.2/8.0
against Linux and OpenSolaris. I would like to bring some of the good and bad
to light (in the hopes that the developers with the correct expertise will be
intrigued).
The tests were performed with a 'standard' installation of FreeBSD on a Lenovo
ThinkPad T61.
I've tried to eliminate tests who's performance is a result of compiler
differences and/or 3rd party applications and tests who's statistical
significance are not so strong (subjective guess).
Improvements for FreeBSD 8.0 vs 7.2:
- 7-Zip Compression (page 3)
- Timed MAFFT Alignment (page 5)
- GraphicsMagick (page 5)
- Threaded IO (64MB Random Write - 32 threads) (page 7)
- Threaded IO (64MB Read - 32 threads) (page 7)
Regressions for FreeBSD 8.0 vs 7.2:
- Gzip compressions of a 2GB file (page 3)
- C-Ray (page 4)
- Threaded IO (64MB Write - 4 threads) (page 7)
- Threaded IO (64MB Write - 32 threads) (page 7)
Poor performance relative to Linux and OpenSolaris
- Threaded IO (especially random writes) (page 7)
- OpenSSL (RSA 4096bit) (page 8)
- PostMark (disk transaction) (page 8)
It appears that threaded activity on UFS does not fair well against Linux/ext4
and OpenSolaris/ZFS. Phoronix intends to do a comparative test against
FreeBSD and OpenSolaris on ZFS.
Regards,
David
1)
http://www.phoronix.com/scan.php?page=article&item=freebsd8_benchmarks&num=1
Alexander Motin
Threaded I/O activity could get much benefit from NCQ-aware disk driver.
It is not included in default FreeBSD kernel, but it would be nice to
compare.
In power-consumption tests I believe FreeBSD could behave much better if
properly tuned. Unluckily our default installation doesn't use any
power-saving technologies. Six month ago I have shown on list how laptop
power consumption can be cut in half.
--
Alexander Motin
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
Matt Thyer
> HI,
>
> David
>
> 1)
> http://www.phoronix.com/scan.php?page=article&item=freebsd8_benchmarks&num=1
>
How much of the problem is due to FreeBSD staying with GCC 4.2.1 (due
to the GPL v3 issue) compared with Linux using the latest GCC ?
Is CLANG/LLVM expected to perform better ?
How much is due to FreeBSD using an older Xorg ? (or is this not
relevant to the benchmarks ?).
A recent Phoronix article showed that Ubuntu 9.04 (I think) performed
worse than Windows (7 I think) using the same open source software.
Interestingly though, Windows binaries generally ran faster under
Ubuntu using Wine than they did under Windows. This makes one think
that GCC still has a long way to go compared to MS Visual Studio (or
Wine has a lot less overhead than a full Windows system).
Matthew
Aragon Gouveia
> Quoting Alexander Motin <m...@FreeBSD.org>:
>> In power-consumption tests I believe FreeBSD could behave much better if
>> properly tuned. Unluckily our default installation doesn't use any
>> power-saving technologies. Six month ago I have shown on list how laptop
>> power consumption can be cut in half.
>
>
> Could you please add your power consumption optimizations into the wiki?
In case anyone is looking for it:
http://lists.freebsd.org/pipermail/freebsd-current/2009-May/006436.html
Regards,
Aragon
Robert Watson
On Mon, 30 Nov 2009, David Naylor wrote:
> Phoronix recently published a comparative benchmark[1] for FreeBSD 7.2/8.0
> against Linux and OpenSolaris. I would like to bring some of the good and
> bad to light (in the hopes that the developers with the correct expertise
> will be intrigued).
>
> The tests were performed with a 'standard' installation of FreeBSD on a
> Lenovo ThinkPad T61.
>
> I've tried to eliminate tests who's performance is a result of compiler
> differences and/or 3rd party applications and tests who's statistical
> significance are not so strong (subjective guess).
It would be nice to see the same tests rerun with more attention to
well-understood benchmarking methodology (i.e., multiple runs,
characterization of variance, statistical tests to decide if two results are
likely to be from the same set, and what statistically significant difference
exists). At the very least, three or five runs of each test so we can
consider spread would be very useful. However, I agree with your
characterizaitons that these are the likely areas of issue :-).
For tests that depend on file system namespace operations, seeing a
measurement comparing ZFS directly would be interesting; likewise, perhaps
using async file system mounts on FreeBSD without soft updates. Jeff's
forthcoming journaling changes would also be interesting to consider, but I'm
not sure to what extent he'll see performance improvements, or whether it's
just about the consistency improvements for his SU+J work.
Looking at compiler differences would also make a lot of sense for things like
compression/ray tracing/etc. Another area of comparison would simply be the
version and contents of gzip, particularly it might be interesting to try the
Linux gzip under FreeBSD's Linux emulation (and for a few other of the
CPU-centric tests: use a Linux chroot or Debian/kFreeBSD to do side-by-side
comparisons on the same kernel).
The threaded I/O is interesting and possiby the greatest concern. We should
try to reproduce these in "lab conditions" and attempt to understand the
workloads more. We do have known "strong" synchronization in the vnode write
path, such that writes to the same vnode may be serialized in a strong way --
it could be that these tests show that up.
Robert
>
> Improvements for FreeBSD 8.0 vs 7.2:
> - 7-Zip Compression (page 3)
> - Timed MAFFT Alignment (page 5)
> - GraphicsMagick (page 5)
> - Threaded IO (64MB Random Write - 32 threads) (page 7)
> - Threaded IO (64MB Read - 32 threads) (page 7)
>
> Regressions for FreeBSD 8.0 vs 7.2:
> - Gzip compressions of a 2GB file (page 3)
> - C-Ray (page 4)
> - Threaded IO (64MB Write - 4 threads) (page 7)
> - Threaded IO (64MB Write - 32 threads) (page 7)
>
> Poor performance relative to Linux and OpenSolaris
> - Threaded IO (especially random writes) (page 7)
> - OpenSSL (RSA 4096bit) (page 8)
> - PostMark (disk transaction) (page 8)
>
> It appears that threaded activity on UFS does not fair well against Linux/ext4
> and OpenSolaris/ZFS. Phoronix intends to do a comparative test against
> FreeBSD and OpenSolaris on ZFS.
>
> Regards,
>
> David
>
> 1)
> http://www.phoronix.com/scan.php?page=article&item=freebsd8_benchmarks&num=1
>
Ivan Voras
> Lars Engels wrote:
>> Quoting Alexander Motin <m...@FreeBSD.org>:
>>> In power-consumption tests I believe FreeBSD could behave much better if
>>> properly tuned. Unluckily our default installation doesn't use any
>>> power-saving technologies. Six month ago I have shown on list how laptop
>>> power consumption can be cut in half.
>>
>>
>> Could you please add your power consumption optimizations into the wiki?
>
> In case anyone is looking for it:
>
> http://lists.freebsd.org/pipermail/freebsd-current/2009-May/006436.html
There is some good advice in that message :)
I have added a preliminary, partially wikified version of it here:
http://wiki.freebsd.org/TuningPowerConsumption
Others may change or move the page if needed.
Andriy Gapon
> The threaded I/O is interesting and possiby the greatest concern. We
> should try to reproduce these in "lab conditions" and attempt to
> understand the workloads more. We do have known "strong"
> synchronization in the vnode write path, such that writes to the same
> vnode may be serialized in a strong way -- it could be that these tests
> show that up.
BTW, threaded I/O tests are done with tiobench:
http://sourceforge.net/projects/tiobench/
The tool compiles fine on FreeBSD.
I believe the tests were executed with options like the following:
tiotest -k3 -k2 -k1 -f 64 -t 4
As I understand the test number zero (sequential write) has always to be executed,
because it produces the files needed for other tests.
It seems that all phoronix tests were done using 4KB reads/writes. No syncing
and/or direct io was done.
So, for example, the first test essentially did the following:
- create 4 files
- create 4 threads
- each thread sequentially writes 4KB chunks to its own file
- measured value is delta between gettimeofday before and after each 4KB write(2)
call
For random write test the same files are reused. The threads do (pseudo-)random
seeks within the files before doing the writes.
There is fsync() call at the end of each write test, its time is not accounted.
For read tests the same files are re-used again.
So we should understand that this test depends on many things before disk writing
gets involved. E.g. VM/bufache strategy, fs tuning.
For instance, compare sequential write test with 32 threads and random write test
with the same number of threads. The article boasts:
"Ubuntu and Fedora did shockingly better, but for those unfamiliar with the
Phoronix Test Suite, tests are generally ran multiple times [...] for accuracy."
But notice that Ubuntu and Fedora results for random write test are *better* than
for the sequential test with all other setting being equal.
I think that that speaks of something about the test method.
The numbers are so ridiculously small that it looks like the writes never reached
disks, at least before fsync().
There are other systemic questions.
E.g. why sequential write test was performed with both 4 thread and 32 thread
configurations, but other tests only with 32 threads. In general, we don't see
how changing parameters affect results for the OSes, e.g. chunk size, total write
size, etc.
--
Andriy Gapon
Alexander Motin
> Threaded I/O activity could get much benefit from NCQ-aware disk driver.
> It is not included in default FreeBSD kernel, but it would be nice to
> compare.
To check possible NCQ effect, I've built test setup with new 320GB
7200RPM Seagate drive connected to Intel ICH10R controller. I've run
IMHO more reasonable benchmark/raidtest tool from ports on whole device,
to execute pregenerated random mix of 10000 random-sized (512B - 128KB)
read/write requests using default ata(4) driver and new ahci(4):
Number of READ requests: 5029.
Number of WRITE requests: 4971.
Number of bytes to transmit: 655986688.
Number of processes: 32.
The results:
ata(4) - no NCQ:
Bytes per second: 12455402
Requests per second: 189
ahci(4) - with NCQ:
Bytes per second: 19889778
Requests per second: 303
Results are repeatable up to the 4-th digit. Average time per request is
5.29ms and 3.3ms respectively, that seems realistic for this drive.
--
Alexander Motin
Ivan Voras
> Alexander Motin wrote:
>> Threaded I/O activity could get much benefit from NCQ-aware disk driver.
>> It is not included in default FreeBSD kernel, but it would be nice to
>> compare.
>
> To check possible NCQ effect, I've built test setup with new 320GB
> 7200RPM Seagate drive connected to Intel ICH10R controller. I've run
> IMHO more reasonable benchmark/raidtest tool from ports on whole device,
> to execute pregenerated random mix of 10000 random-sized (512B - 128KB)
> read/write requests using default ata(4) driver and new ahci(4):
> Number of READ requests: 5029.
> Number of WRITE requests: 4971.
> Number of bytes to transmit: 655986688.
> Number of processes: 32.
>
> The results:
> ata(4) - no NCQ:
> Bytes per second: 12455402
> Requests per second: 189
> ahci(4) - with NCQ:
> Bytes per second: 19889778
> Requests per second: 303
>
> Results are repeatable up to the 4-th digit. Average time per request is
> 5.29ms and 3.3ms respectively, that seems realistic for this drive.
If you have a drive to play with, could you also check UFS vs ZFS on
both ATA & AHCI? To try and see if the IO scheduling of ZFS plays nicely.
For benchmarks I suggest blogbench and bonnie++ (in ports) and if you
want to bother, randomio, http://arctic.org/~dean/randomio .
Nenhum_de_Nos
Alexander Motin <m...@FreeBSD.org> wrote:
> Alexander Motin wrote:
> > Threaded I/O activity could get much benefit from NCQ-aware disk driver.
> > It is not included in default FreeBSD kernel, but it would be nice to
> > compare.
>
> To check possible NCQ effect, I've built test setup with new 320GB
> 7200RPM Seagate drive connected to Intel ICH10R controller. I've run
> IMHO more reasonable benchmark/raidtest tool from ports on whole device,
> to execute pregenerated random mix of 10000 random-sized (512B - 128KB)
> read/write requests using default ata(4) driver and new ahci(4):
> Number of READ requests: 5029.
> Number of WRITE requests: 4971.
> Number of bytes to transmit: 655986688.
> Number of processes: 32.
>
> The results:
> ata(4) - no NCQ:
> Bytes per second: 12455402
> Requests per second: 189
> ahci(4) - with NCQ:
> Bytes per second: 19889778
> Requests per second: 303
>
> Results are repeatable up to the 4-th digit. Average time per request is
> 5.29ms and 3.3ms respectively, that seems realistic for this drive.
Alexander,
to make use of all this new stuff I need to run HEAD ? as former HEAD was born recently as 8.0R and the new stuff in it is big compared to 7.x, I'd like to use it from RELENG_8. is this possible ?
my hardware would be NVIDIA nForce 680i SLI motherboard and Seagate ES.2 disks.
thanks,
matheus
--
We will call you Cygnus,
The God of balance you shall be
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
http://en.wikipedia.org/wiki/Posting_style
Alexander Motin
> to make use of all this new stuff I need to run HEAD ? as former HEAD was born recently as 8.0R and the new stuff in it is big compared to 7.x, I'd like to use it from RELENG_8. is this possible ?
You need to run recent 8-STABLE and load ahci kernel module. It is not
included into default GENERIC kernel yet, due to work in progress.
8.0-RELEASE also includes some of that code, but on much earlier
development stage.
> my hardware would be NVIDIA nForce 680i SLI motherboard and Seagate ES.2 disks.
OK. Make sure that you have AHCI SATA mode enabled in BIOS.
--
Alexander Motin
Nenhum_de_Nos
Alexander Motin <m...@FreeBSD.org> wrote:
> Nenhum_de_Nos wrote:
> > to make use of all this new stuff I need to run HEAD ? as former HEAD was born recently as 8.0R and the new stuff in it is big compared to 7.x, I'd like to use it from RELENG_8. is this possible ?
>
> You need to run recent 8-STABLE and load ahci kernel module. It is not
> included into default GENERIC kernel yet, due to work in progress.
> 8.0-RELEASE also includes some of that code, but on much earlier
> development stage.
ok
> > my hardware would be NVIDIA nForce 680i SLI motherboard and Seagate ES.2 disks.
>
> OK. Make sure that you have AHCI SATA mode enabled in BIOS.
just to make sure, disks will be adaX now, right ?
another great reason to move to glabel ASAP :)
thanks,
matheus
--
We will call you Cygnus,
The God of balance you shall be
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
http://en.wikipedia.org/wiki/Posting_style
Alexander Motin
> If you have a drive to play with, could you also check UFS vs ZFS on
> both ATA & AHCI? To try and see if the IO scheduling of ZFS plays nicely.
I have even several drives to test.
> For benchmarks I suggest blogbench and bonnie++ (in ports) and if you
> want to bother, randomio, http://arctic.org/~dean/randomio .
Such tests completely depend on FS, caching and test parameters,
depending on what do you want to measure. Propose your schedule and I'll
run it. How much RAM should system have, what tool should I run, with
what parameters...?
--
Alexander Motin
Lars Engels
> I have added a preliminary, partially wikified version of it here:
>
> http://wiki.freebsd.org/TuningPowerConsumption
Thanks a lot! :)
>
> Others may change or move the page if needed.
That's why I'd like to have it in the wiki.
--
Lars
Antony Mawer
...
>
> For tests that depend on file system namespace operations, seeing a
> measurement comparing ZFS directly would be interesting; likewise, perhaps
> using async file system mounts on FreeBSD without soft updates. Jeff's
> forthcoming journaling changes would also be interesting to consider, but
> I'm not sure to what extent he'll see performance improvements, or whether
> it's just about the consistency improvements for his SU+J work.
Is there any information about what these journaling changes are or
what they involve? I would be interested to keep tabs on any ongoing
work being done on journalling for UFS...
--Antony
Dag-Erling Smørgrav
> The results:
> ata(4) - no NCQ:
> Bytes per second: 12455402
> Requests per second: 189
> ahci(4) - with NCQ:
> Bytes per second: 19889778
> Requests per second: 303
>
> Results are repeatable up to the 4-th digit. Average time per request is
> 5.29ms and 3.3ms respectively, that seems realistic for this drive.
/usr/bin/ministat
DES
--
Dag-Erling Smørgrav - d...@des.no
Ivan Voras
> Alexander Motin <m...@FreeBSD.org> writes:
>> The results:
>> ata(4) - no NCQ:
>> Bytes per second: 12455402
>> Requests per second: 189
>> ahci(4) - with NCQ:
>> Bytes per second: 19889778
>> Requests per second: 303
>>
>> Results are repeatable up to the 4-th digit. Average time per request is
>> 5.29ms and 3.3ms respectively, that seems realistic for this drive.
>
> /usr/bin/ministat
Wheee, it's built by default now :)
Julian Elischer
but it needs to have an option to make it print "bigger is better" or
something..
So often people just post the output and don't tell you the units,
or whether left or right is better.
Robert N. M. Watson
On 2 Dec 2009, at 04:01, Antony Mawer wrote:
> On Wed, Dec 2, 2009 at 12:25 AM, Robert Watson <rwa...@freebsd.org> wrote:
> ...
>>
>> For tests that depend on file system namespace operations, seeing a
>> measurement comparing ZFS directly would be interesting; likewise, perhaps
>> using async file system mounts on FreeBSD without soft updates. Jeff's
>> forthcoming journaling changes would also be interesting to consider, but
>> I'm not sure to what extent he'll see performance improvements, or whether
>> it's just about the consistency improvements for his SU+J work.
>
> Is there any information about what these journaling changes are or
> what they involve? I would be interested to keep tabs on any ongoing
> work being done on journalling for UFS...
I'm not aware of detailed public descriptions yet, but with any luck we'll hear more in the near future. I'm very much looking forward to seeing it make it into -current, with the usual understanding that these things take a while to shake out...
Robert_______________________________________________