sysbench / fileio - Linux vs. FreeBSD
Adam PAPAI
Hash: SHA1
Hi List,
A week ago I started to benchmark Linux vs. FreeBSD on a Dell Poweredge
1850.
CPU: 2 x 3.4Ghz Xeon (Dual Core)
Memory: 8GB (4x2)
Disk: 1 x SEAGATE ST373454LC D404 (SCSI)
FreeBSD kazoku 8.0-RELEASE-p3 FreeBSD 8.0-RELEASE-p3 #0: Tue May 25
20:54:11 UTC 2010
ro...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
The tests with seqrewr, seqrd, rndrd, and so on is still going on, so I
can only publish the seqwr result. (The PostgreSQL will be tested as well)
(soft-updates are on)
/dev/da0s1d on /usr (ufs, local, soft-updates)
Tested with:
sysbench --num-threads=$a --file-block-size=$bs --test=fileio
- --file-total-size=2G --file-fsync-all=no --file-test-mode=seqwr run
My first results (seqwr with 1,2,4,8,6,32 threads) can be found here.
http://tech-blog.wooh.hu/~wooh/fbsd_vs_debian_seqwr.html
Why FreeBSD is supreme with 1 and 2 thread. And why is it 2 and 3 times
slower with 4-8-16-32 threads compared to Debian? The first two tests (1
thread and 2 thread) showed me that FreeBSD is supreme in I/O, but later
tests showed me, that it can produce horrible I/O.
How can I tune my disk to make it faster? Is it possible? What is the
reason of the really slow I/O with more than 4 threads? What do you
recommend me to do? Why is it damn slow with 8K blocksize?
I have more than 15 FreeBSD servers in production environment and I
don't want to change operating system due to I/O issues. I changed my
OpenBSD servers to FreeBSD 3 years ago... :)
When all tests are ready I'll publish all the results, including the
postgresql benchmarks as well.
Best Regards,
- --
Adam PAPAI
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJMCZMrAAoJEGq0EWvh5uiI10MIAM1iZxFZ5xssKmawHl56Ruin
zHHgb4Nc15waTLdzFGfllAayDlZqvvpoSpOVbp8qDZYlkTbYPF6aMjkehqMvQUEo
nFs7WN2VaCSOhUUQSwjqfGdnMLW9H5uyW/ZkYvgoOjQjz/vewDV6Fi+ZfGmt5Zqw
gV1ZlXFdAUOUW6c90ODOPxn+7XCA5UC2sUMPB+1iNxrTiiS6C2YQ0Vy1fCXvrhU3
51n0ES/7JBF4sk5dH1VNEU/8AeQRBOoKPuAHhZKRZZ1x+1dMkDhwdD+KUHGrRGJd
fUAZmMhjE6fRG86FbwK5jrZizHZYpE3PfpZe6tI3SIvw7NbUNrRsCMSiel+0FBg=
=k3Sw
-----END PGP SIGNATURE-----
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Igor Mozolevsky
> How can I tune my disk to make it faster? Is it possible? What is the
> reason of the really slow I/O with more than 4 threads? What do you
> recommend me to do? Why is it damn slow with 8K blocksize?
Does linux still have async disk writes by default?
Igor
Bruce Cran
> Why FreeBSD is supreme with 1 and 2 thread. And why is it 2 and 3 times
> slower with 4-8-16-32 threads compared to Debian? The first two tests (1
> thread and 2 thread) showed me that FreeBSD is supreme in I/O, but later
> tests showed me, that it can produce horrible I/O.
>
> How can I tune my disk to make it faster? Is it possible? What is the
> reason of the really slow I/O with more than 4 threads? What do you
> recommend me to do? Why is it damn slow with 8K blocksize?
Some quick tests show that ufs does do rather poorly on my system too. I have
the following filesystems setup:
/var : ufs with softupdates
/usr/obj : zfs with checksums disabled
/usr/src : zfs with compression enabled
/home : zfs with compression disabled and checksums enabled
I ran a test with a blocksize of 8KB and 16 threads.
/var : 25.2MB/s
/usr/obj : 64.8MB/s
/usr/src : 386.3MB/s
/home : 60.3MB/s
--
Bruce Cran
Adam PAPAI
> Some quick tests show that ufs does do rather poorly on my system too. I have
> the following filesystems setup:
>
> /var : ufs with softupdates
> /usr/obj : zfs with checksums disabled
> /usr/src : zfs with compression enabled
> /home : zfs with compression disabled and checksums enabled
>
> I ran a test with a blocksize of 8KB and 16 threads.
>
> /var : 25.2MB/s
> /usr/obj : 64.8MB/s
> /usr/src : 386.3MB/s
> /home : 60.3MB/s
>
It seems I have to test it with zfs as well. Tomorrow I'm gonna test it.
--
Adam PAPAI
Bruce Cran
Stefan Miklosovic <miklosovi...@gmail.com> wrote:
> > /var : ufs with softupdates
> > /usr/obj : zfs with checksums disabled
> > /usr/src : zfs with compression enabled
> > /home : zfs with compression disabled and checksums enabled
> >
> > I ran a test with a blocksize of 8KB and 16 threads.
> >
> > /var : 25.2MB/s
> > /usr/obj : 64.8MB/s
> > /usr/src : 386.3MB/s
> > /home : 60.3MB/s
>
> Do I understand it well? It seems that zfs with compression enabled on
> /usr/src with 8KB block size and 16 threads performs 386.3MB/s which
> is about 6 times better than debian5? I am thinking about this image
> http://tech-blog.wooh.hu/~wooh/debian_vs_freebsd_io_16_seqwr.png
Yes - on one run it even hit 500MB/s. I suspect, however, that the
benchmark isn't accurate because it won't be writing typical data.
Instead it's probably using a buffer that compresses very well.
Adam PAPAI
Hash: SHA1
On 6/5/10 1:04 PM, Bruce Cran wrote:
> On Sat, 5 Jun 2010 12:50:15 +0200
> Stefan Miklosovic <miklosovi...@gmail.com> wrote:
>
>>> /var : ufs with softupdates
>>> /usr/obj : zfs with checksums disabled
>>> /usr/src : zfs with compression enabled
>>> /home : zfs with compression disabled and checksums enabled
>>>
>>> I ran a test with a blocksize of 8KB and 16 threads.
>>>
>>> /var : 25.2MB/s
>>> /usr/obj : 64.8MB/s
>>> /usr/src : 386.3MB/s
>>> /home : 60.3MB/s
>>
>> Do I understand it well? It seems that zfs with compression enabled on
>> /usr/src with 8KB block size and 16 threads performs 386.3MB/s which
>> is about 6 times better than debian5? I am thinking about this image
>> http://tech-blog.wooh.hu/~wooh/debian_vs_freebsd_io_16_seqwr.png
>
> Yes - on one run it even hit 500MB/s. I suspect, however, that the
> benchmark isn't accurate because it won't be writing typical data.
> Instead it's probably using a buffer that compresses very well.
Hm.. My ZFS tests showed me the same results. With compression it's
pretty fast. An application benchmark will give us typical data write,
so I'll run PgSQL benchmarks on the ZFS pool as well.
- --
Adam PAPAI
NETIDEA Informatikai Szolgáltató Kft.
http://www.netidea.hu
E-mail: wo...@wooh.hu
Phone: +36 30 33-55-735 (Hungary)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJMCjJiAAoJEGq0EWvh5uiIKFMH/1dP4OZGAMiBNSoRqGFfnZ5B
/vtf5do2t3JRbjfYi2HyNn8gXss4xRDPouVmftl2OglIXA77hMIyIcjyoWnHGTBc
M1WnnNDz1wIb8EYSl9MYKAjQA1wGsYd4UImd1MqOtZfSuOht6hTLoSiAnC1xMLtk
9vgFUtMok8XclPqL08J/dWs39+HwhSaooRnLEx7IYLSgFis7vQtJjOaWWG3LUADw
QsivcCSjBBoQ7LD9WXN5prmlwt+CMBU/F1yyMaJXa0bNI7AM+hh5Mix03P4HAKEz
4Z92lcmLXzSVnllA0tAJvAwEPtk4laP6yzM9egStDNvxONLueQVLXfY8gvukQ2k=
=MVI2
-----END PGP SIGNATURE-----
Stefan Miklosovic
> /usr/obj : zfs with checksums disabled
> /usr/src : zfs with compression enabled
> /home : zfs with compression disabled and checksums enabled
>
> I ran a test with a blocksize of 8KB and 16 threads.
>
> /var : 25.2MB/s
> /usr/obj : 64.8MB/s
> /usr/src : 386.3MB/s
> /home : 60.3MB/s
Do I understand it well? It seems that zfs with compression enabled on
/usr/src with 8KB block size and 16 threads performs 386.3MB/s which
is about 6 times better than debian5? I am thinking about this image
http://tech-blog.wooh.hu/~wooh/debian_vs_freebsd_io_16_seqwr.png
what is your system specs?
Igor Mozolevsky
>>> Do I understand it well? It seems that zfs with compression enabled on
>>> /usr/src with 8KB block size and 16 threads performs 386.3MB/s which
>>> is about 6 times better than debian5? I am thinking about this image
>>> http://tech-blog.wooh.hu/~wooh/debian_vs_freebsd_io_16_seqwr.png
>>
>> Yes - on one run it even hit 500MB/s. I suspect, however, that the
>> benchmark isn't accurate because it won't be writing typical data.
>> Instead it's probably using a buffer that compresses very well.
>
> Hm.. My ZFS tests showed me the same results. With compression it's
> pretty fast.
That's hardly a surprise - you take the source code, compress it into
virtual non-existence leaving hardly anything to be written to the
disk... Obviously if compression speed >> IO speed and the result of
the compression is a significant reduction in size, you have a massive
gain in writing that data to the disk.
--
Igor
Julian Elischer
> On 6/5/10 3:36 AM, Bruce Cran wrote:
>> Some quick tests show that ufs does do rather poorly on my system too. I have
>> the following filesystems setup:
>>
>> /var : ufs with softupdates
>> /usr/obj : zfs with checksums disabled
>> /usr/src : zfs with compression enabled
>> /home : zfs with compression disabled and checksums enabled
>>
>> I ran a test with a blocksize of 8KB and 16 threads.
>>
>> /var : 25.2MB/s
>> /usr/obj : 64.8MB/s
>> /usr/src : 386.3MB/s
>> /home : 60.3MB/s
>>
>
> It seems I have to test it with zfs as well. Tomorrow I'm gonna test it.
>
>
Then the linux people will insist you use btrfs to compare apples to
apples ... etc... etc..
Matthew Jacob
The following seems to be true, though:
a) FreeBSD sequential write performance in UFS has always been less than
optimal.
b) Linux sequential write performance in just about any filesystem has
always been "impressive". But that "impressive" has come at some not so
obvious costs. First of all, Linux is probably the most aggressive
cluster/write-behind OS I've even seen. You can suck down all available
memory with writebehind using dd. This means that some stats are
"impressive", and others are "painful". A desktop that becomes
completely unresponsive while you're doing this dd is one personal outcome.
Also, you have to be careful what you're asking for in comparing the two
platforms, or any platforms for that matter. What do you want to
optimize for? Apparent responsiveness as a desktop? A specific workload
(nfs, cifs) that completes N quatloos per fortnight?
Attilio Rao
>
> All of these tests have been apples vs. oranges for years.
>
> The following seems to be true, though:
>
> a) FreeBSD sequential write performance in UFS has always been less than optimal.
>
> b) Linux sequential write performance in just about any filesystem has always been "impressive". But that "impressive" has come at some not so obvious costs. First of all, Linux is probably the most aggressive cluster/write-behind OS I've even seen. You can suck down all available memory with writebehind using dd. This means that some stats are "impressive", and others are "painful". A desktop that becomes completely unresponsive while you're doing this dd is one personal outcome.
>
> Also, you have to be careful what you're asking for in comparing the two platforms, or any platforms for that matter. What do you want to optimize for? Apparent responsiveness as a desktop? A specific workload (nfs, cifs) that completes N quatloos per fortnight?
Besides anything, I'm much more concerned about the loss of
performance within FreeBSD itself. I wouldn't expect a so high
pessimization when the number of threads increases (without
considering the big performance loss with the 8k blocksize, pretty
much reproducible). I'm trying to drive, privately, the tester to
pmc/lock profiling analysis in order to start collecting some useful
datas.
While I think that we might pay a lot of attention to ZFS, I think we
might not leave alone FFS. Having a fast, well supported, native
filesystem might be a great thing for us.
Comparing with other operating systems, as you smartly point out,
might not be got as 'undefeatable truths' but have cons and prons that
needs to be fully understood before to make false claims.
Thanks,
Attilio
--
Peace can only be achieved by understanding - A. Einstein
Kostik Belousov
> 2010/6/5 Matthew Jacob <m...@feral.com>
> >
> > All of these tests have been apples vs. oranges for years.
> >
> > The following seems to be true, though:
> >
> > a) FreeBSD sequential write performance in UFS has always been less than optimal.
> >
> > b) Linux sequential write performance in just about any filesystem has always been "impressive". But that "impressive" has come at some not so obvious costs. First of all, Linux is probably the most aggressive cluster/write-behind OS I've even seen. You can suck down all available memory with writebehind using dd. This means that some stats are "impressive", and others are "painful". A desktop that becomes completely unresponsive while you're doing this dd is one personal outcome.
> >
> > Also, you have to be careful what you're asking for in comparing the two platforms, or any platforms for that matter. What do you want to optimize for? Apparent responsiveness as a desktop? A specific workload (nfs, cifs) that completes N quatloos per fortnight?
>
> Besides anything, I'm much more concerned about the loss of
> performance within FreeBSD itself. I wouldn't expect a so high
> pessimization when the number of threads increases (without
> considering the big performance loss with the 8k blocksize, pretty
> much reproducible). I'm trying to drive, privately, the tester to
> pmc/lock profiling analysis in order to start collecting some useful
> datas.
If yes, then this behaviour is well understood.
Attilio Rao
>
> On Sat, Jun 05, 2010 at 07:41:23PM +0200, Attilio Rao wrote:
> > 2010/6/5 Matthew Jacob <m...@feral.com>
> > >
> > > All of these tests have been apples vs. oranges for years.
> > >
> > > The following seems to be true, though:
> > >
> > > a) FreeBSD sequential write performance in UFS has always been less than optimal.
> > >
> > > b) Linux sequential write performance in just about any filesystem has always been "impressive". But that "impressive" has come at some not so obvious costs. First of all, Linux is probably the most aggressive cluster/write-behind OS I've even seen. You can suck down all available memory with writebehind using dd. This means that some stats are "impressive", and others are "painful". A desktop that becomes completely unresponsive while you're doing this dd is one personal outcome.
> > >
> > > Also, you have to be careful what you're asking for in comparing the two platforms, or any platforms for that matter. What do you want to optimize for? Apparent responsiveness as a desktop? A specific workload (nfs, cifs) that completes N quatloos per fortnight?
> >
> > Besides anything, I'm much more concerned about the loss of
> > performance within FreeBSD itself. I wouldn't expect a so high
> > pessimization when the number of threads increases (without
> > considering the big performance loss with the 8k blocksize, pretty
> > much reproducible). I'm trying to drive, privately, the tester to
> > pmc/lock profiling analysis in order to start collecting some useful
> > datas.
> Are the benchmarks create threads that write to the same file ?
> If yes, then this behaviour is well understood.
Actually I still don't know as I just sent an e-mail to the tester and
he didn't followup still.
However I'm not entirely sure this is a full bottleneck which may be
reconduit to missing of byte-range locking.
I want to dig more and better understand what's going on exactly.
Adam PAPAI
Hash: SHA1
On 6/5/10 2:43 AM, Igor Mozolevsky wrote:
> On 5 June 2010 00:58, Adam PAPAI <wo...@wooh.hu> wrote:
>
>> How can I tune my disk to make it faster? Is it possible? What is the
>> reason of the really slow I/O with more than 4 threads? What do you
>> recommend me to do? Why is it damn slow with 8K blocksize?
>
> Does linux still have async disk writes by default?
Anyway, I looked after the default ext3 values:
Debian mounts the ext3 with "defaults" option.
This means: rw, suid, dev, exec, auto, nouser, and async.
Well it means I have to test it with UFS (async) and Debian (sync).
These test will take some time but I hope it worth the effort.
Hm...
- --
Adam PAPAI
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJMC31LAAoJEGq0EWvh5uiIu3MH/i7KWfcYj2zXSsqbUK2W4dKi
B0+pD861FBtxmS+O4c4jzR5vJYeVVyVfZ4DLpHs0tqr6u2QZWgTD5c9GXxRNn9Hg
pVIL8/iL9BGtjNZdbjKU2RlE+QOb4LUuxqTWtz3poH4e6CQlAMOzvBcmbK41eWVn
nr2/jlS8n7TFk74ewAH9NXABrhIaOtCjBf5YWWA9AnKhqjdlAM7gxC6QcbsGTLlR
5zvq6UfGuAMECOV98FDlm3k20LydLT0/Mdw9jth9+50v1NMnAddYjfZ/7Ci2KzZo
uUN1VRcOhxmw6oliMPu/+Z324d6Xrp1vXpDQN8tSzME1d3O3CswPDfs3ocpjmkU=
=VEsH
-----END PGP SIGNATURE-----
Matthew Dillon
:All of these tests have been apples vs. oranges for years.
:
:The following seems to be true, though:
:
:a) FreeBSD sequential write performance in UFS has always been less than
:optimal.
If there's no read activity sequential write performance should be
maximal with UFS. The keyphrase here is "no read activity".
UFS's main problem, easily demonstrated by running something like
blogbench --iterations=100, is that read I/O is given such a huge
precedence over write I/O it can cause the write I/O to come to
a complete grinding halt once the system caches are blown out and
the reads start having to go to disk.
Another big issue with filesystem benchmarks is the data footprint
size of the benchmark. Many benchmarks do not have a sufficiently large
data footprint and wind up simply testing how much memory the kernel
is willing to give over to cache the benchmark's tests, instead of
testing disk performance. Bonnie++ is a really good example of the
latter problem.
That said, all the BSDs have stall issues with parallel read & write
activity on the same file. It essentially comes down to the vnode
lock held during writes which can cause reads on the same file to
stall even when those reads could be satisfied from the VM/BUF cache.
Linux might appear to work better in such benchmarks because Linux
essentially allows infininte write buffering, up to the point where
system memory is exhausted, and the BSDs do not. Infinite write
buffering might make a benchmark look good but it creates horrible
stalls and inconsistencies on production systems.
I noticed that FreeBSD's ZFS implementation issues VOP_WRITE's
with a shared lock instead of an exclusive lock, thus avoiding
this particular problem. It would be possible to do this with UFS
too with some work to prevent file size changes from colliding during
concurrent writes, or even using a separate serializer for
modifying/write operations so read operations can continue to run
concurrently.
blogbench is a good way to test read/write interference during the
system-cache phase of blogbench's operation (that would be the first
500-800 or so blogs on a 4G system). If working properly both read and
write operations should be maximal during this phase. That is, the
disk should be 100% saturated with writes while all reads are still
fully satisfiable from the buffer cache / VM system, and at the same
time the read rate should not suffer (not be seen to stall).
It would be interesting to see a blogbench comparison between UFS
and ZFS on the same hw/disk.
-Matt
Adam PAPAI
Hash: SHA1
On 6/6/10 7:20 PM, Matthew Dillon wrote:
> It would be interesting to see a blogbench comparison between UFS
> and ZFS on the same hw/disk.
I'll do it, just tell me how do you want to run the tests.
The system params are:
8GB Memory
2x72GB SCSI HDD
2x3.4Ghz Xeon
Overall: Dell Poweredge 1850. With no raid installed.
I'm waiting the benchmark options to run.
- --
Adam PAPAI
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJMDAkBAAoJEGq0EWvh5uiIPs8H+wUm/AMo7KBOViRG+OMZKl65
RgeJRBco/FvAjZc7Qu8EwfL3QqkZzZcDZmYODwvfhu4Hmw5QFrbu/QjUG3OFFBjH
6Kgyod2VsIRNo74LDTcdHIDwqCKfWYpH1sSkzr73ewgVAMtwen/6ob1hqW9i7jp8
4Fg5emWcJxJ6vh5t/5cgNhrpFdlZ1g4Zcd4yOPdgC04RoRC5Pla4QK6if6KdN4Qe
V9MmiJBdekdCwyaoL7sgXh6QLeXqEHd6D2BDYxS40BCH1OxTVnSb+AedqAeIO+OE
0470hgJnHb4500lu4UfdZtvj4DOUkDn/DiA7VU1we6iZCSitFQNxGi6fK2WyXaw=
=sERi
-----END PGP SIGNATURE-----
Matthew Dillon
:> It would be interesting to see a blogbench comparison between UFS
:> and ZFS on the same hw/disk.
:
:
:I'll do it, just tell me how do you want to run the tests.
:
:The system params are:
:
:8GB Memory
:2x72GB SCSI HDD
:2x3.4Ghz Xeon
:Overall: Dell Poweredge 1850. With no raid installed.
:
:I'm waiting the benchmark options to run.
:
:- --
:Adam PAPAI
With 8G of ram blogbench should blow out the system caches at around
blog 1000-1600, though it also depends on the maximum number of
vnodes supported by the system. One of the two (VM pages or vnode
limit) will be hit.
All you need to do is run blogbench with enough iterations to ensure
that the run eventually blows out the system caches. 200 or 300 should
do the job. It's easy to tell when the system cache gets blown out from
looking at the output.
Run something like the following script for a few hours. You want to
get at least four full runs under your belt for each filesystem to
factor out edge cases.
For the filesystem setup it would be cool to test both the single-drive
case and a simple non-redundant interleaved or mirrored setup (double
the read bandwidth). With UFS use default parameters with softupdates
turned on (I'd say also without SUJ). With ZFS I don't know how best
to tune it, try to find a ZFS setup that performs decently.
However, be sure to turn off compression and dedup (if those fs options
are available), because blogbench basically just writes all-zeros which
is highly compressable/collapsable and would skew the results badly.
-Matt
#!/bin/csh
#
# /build is the filesystem under test.
set i = 0
while(1)
set name = `printf "bench%05d" $i`
echo $name
if ( ! -d /build/blogs/$name ) then
mkdir -p /build/blogs/$name
blogbench --iterations=200 -d /build/blogs/$name
sleep 120
endif
@ i = $i + 1
end