Re: Dell PowerEdge 2950 performance
Bucky Jordan
Is the PERC 5/i dual channel? If so, are 1/2 the drives on one channel and the other half on the other channel? I find this helps RAID10 performance when the mirrored pairs are on separate channels.
...
With the SAS controller (PERC 5/i), every drive gets it's own 3 GB/s port.
...
Your transfer rate seems pretty good for Dell hardware, but I'm not experienced with SAS drives to know if those numbers are good in an absolute sense.
Also, which driver picked up the SAS controller? amr(4) or aac(4) or some other? That makes a big difference too. I think the amr driver is "better" than the aac driver.
..
The internals of the current SAS drives are similar to the U320's they replaced in terms of read/write/seek performance, however the benefit is the SAS bus, which helps eliminate some of the U320 limitations (e.g. with Perc4, you only get 160 MB/s per channel as you mentioned). It's using the mfi driver...
Here's some simplistic performance numbers:
time bash -c "(dd if=/dev/zero of=bigfile count=125000 bs=8k && sync)"
Raid0 x 2 (2 spindles) ~138 MB/s on BSD
Raid5 x 4 ~160 MB/s BSD, ~274 MB/s Knoppix (ext2)
Raid5 x 6 ~255 MB/s BSD, 265 MB/s Knoppix (ext3)
Raid10 x 4 ~25 MB/s BSD
Raid50 x 6 ~144 MB/s BSD, 271 MB/s Knoppix
* BSD is 6.1-RELEASE amd64 with UFS + Soft updates, Knoppix is 5.1 (ext2 didn't like the > 1TB partition for the 6 disk RAID 5, hence ext3)
Seems to me the PERC5 has issues with layered raid (10, 50) as others have suggested on this list is a common problem with lower end raid cards. For now, I'm going with the RAID 5 option, however if I have time, I would like to test having the hardware do raid 0 and doing raid 1 in the os, or vice versa, as proposed in other posts.
Also, I ran a pgbench -s 50 -c 10 -t 1000 on a completely default BSD 6.1 and PG 8.1.4 install with RAID5 x 6 disks, and got 442 tps on a fresh run (the numbers climb very rapidly due to caching after running simultaneous tests without reinitializing the test db. I'm guessing this is due to OS caching since the default postgresql.conf is pretty limited in terms of resource use). I probably need to up the scaling factor significantly so the whole data set doesn't get cached in RAM if I want realistic results from simultaneous tests, but it seems quicker to just reinit each time at this point.
On to some kernel tweaks and some adjustments to postgresql.conf...
- Bucky
---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster
Bucky Jordan
Of more interest would be a test which involved large files with lots
of seeks all around (something like bonnie++ should do that).
...
Here's the bonnie++ numbers for the RAID 5 x 6 disks. I believe this was
with write-through and 64k striping. I plan to run a few others with
different block sizes and larger files- I'd be happy to send out a link
to the list when I get a chance to post them somewhere. I've also been
running some basic tests with pgbench just to help jumpstart customizing
postgresql.conf, so that might be of interest too.
bash-2.05b$ bonnie++ -d bonnie -s 1000:8k
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
1000M 587 99 246900 71 225124 76 1000 99 585723 99
8573 955
Latency 14367us 50829us 410ms 57965us 1656us
432ms
Version 1.93c ------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 28192 91 +++++ +++ +++++ +++ 26076 89 +++++ +++
+++++ +++
Latency 25988us 75us 37us 24756us 36us
41us
1.93c,1.93c,
,1,1155223901,1000M,,587,99,246900,71,225124,76,1000,99,585723,99,8573,9
55,16,,,,,28192,91,+++++,+++,+++++,+++,26076,89,+++++,+++,+++++,+++,1436
7us,50829us,410ms,57965us,1656us,432ms,25988us,75us,37us,24756us,36us,41
us
...
Thanks for sharing your numbers.
...
You're welcome- I prefer to see actual numbers rather than people simply
stating that RAID controller X is better, so hopefully more people will
do the same.
- Bucky
---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend
Luke Lonergan
I see you are running bonnie++ version 1.93c. The numbers it reports are
very different from version 1.03a, which is the one everyone runs - can you
post your 1.03a numbers from bonnie++?
- Luke
---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?
Bucky Jordan
I see you are running bonnie++ version 1.93c. The numbers it reports are
very different from version 1.03a, which is the one everyone runs - can
you
post your 1.03a numbers from bonnie++?
Luke,
Thanks for the pointer. Here's the 1.03 numbers, but at the moment I'm
only able to run them on the 6 disk RAID 5 setup (128k stripe, writeback
enabled since the Perc5 does have a battery backed cache).
bonnie++ -d bonnie -s 1000:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
1000M 155274 95 265359 44 232958 52 166884 99
1054455 99 +++++ +++
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 +++++ +++ +++++ +++ +++++ +++ 30550 88 +++++ +++
+++++ +++
,1000M,155274,95,265359,44,232958,52,166884,99,1054455,99,+++++,+++,16,+
++++,+++,+++++,+++,+++++,+++,30550,88,+++++,+++,+++++,+++
- Bucky
---------------------------(end of broadcast)---------------------------
Luke Lonergan
I don't know why I missed this the first time - you need to let bonnie++
pick the file size - it needs to be 2x memory or the results you get will
not be accurate.
In this case you've got a 1GB file, which nicely fits in RAM.
- Luke
---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majo...@postgresql.org so that your
message can get through to the mailing list cleanly
Bucky Jordan
For some reason it looks like bonnie is picking a 300M file.
> bonnie++ -d bonnie
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
300M 179028 99 265358 41 270175 57 167989 99 +++++ +++
+++++ +++
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
,300M,179028,99,265358,41,270175,57,167989,99,+++++,+++,+++++,+++,16,+++
++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
So here's results when I force it to use a 16GB file, which is twice the
amount of physical ram in the system:
> bonnie++ -d bonnie -s 16000:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
16000M 158539 99 244430 50 58647 29 83252 61 144240 21
789.8 7
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 7203 54 +++++ +++ +++++ +++ 24555 42 +++++ +++
+++++ +++
,16000M,158539,99,244430,50,58647,29,83252,61,144240,21,789.8,7,16,7203,
54,+++++,+++,+++++,+++,24555,42,+++++,+++,+++++,+++
... from Vivek...
which is an issue with freebsd and bonnie++ since it doesn't know
that freebsd can use large files natively (ie, no large file hacks
necessary). the freebsd port of bonnie takes care of this, if you
use that instead of compiling your own.
...
Unfortunately I had to download and build by hand, since only bonnie++
1.9x is available in BSD 6.1 ports when I checked.
One other question- would the following also be mostly a test of RAM? I
wouldn't think so since it should force it to sync to disk...
time bash -c "(dd if=/dev/zero of=/data/bigfile count=125000 bs=8k &&
sync)"
Oh, and while I'm thinking about it, I believe Postgres uses 8k data
pages correct? On the RAID, I'm using 128k stripes. I know there's been
posts on this before, but is there any way to tell postgres to use this
in an effective way?
Thanks,
Bucky
-----Original Message-----
From: pgsql-perfo...@postgresql.org
[mailto:pgsql-perfo...@postgresql.org] On Behalf Of Vivek Khera
Sent: Tuesday, August 15, 2006 3:18 PM
To: Pgsql-Performance ((E-mail))
Subject: Re: [PERFORM] Dell PowerEdge 2950 performance
On Aug 15, 2006, at 2:50 PM, Luke Lonergan wrote:
> I don't know why I missed this the first time - you need to let
> bonnie++
> pick the file size - it needs to be 2x memory or the results you
> get will
> not be accurate.
which is an issue with freebsd and bonnie++ since it doesn't know
that freebsd can use large files natively (ie, no large file hacks
necessary). the freebsd port of bonnie takes care of this, if you
use that instead of compiling your own.
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Luke Lonergan
you posted, but you got the 16GB/8k parts right.
Now we're looking at realistic numbers - 790 seeks/second, 244MB/s
sequential write, but only 144MB/s sequential reads, perhaps 60% of what it
should be.
Seems like a pretty good performer in general - if it was Linux I'd play
with the max readahead in the I/O scheduler to improve the sequential reads.
- Luke
---------------------------(end of broadcast)---------------------------
Bucky Jordan
Thanks for the tips. I'm running FreeBSD 6.1 amd64, but, I can also
enable readahead on the raid controller, and also adaptive readahead.
Here's tests:
Readahead & writeback enabled:
bash-2.05b$ bonnie++ -d bonnie -s 16000:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
16000M 156512 98 247520 47 59560 27 83138 60 143588 21
792.8 7
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 +++++ +++ +++++ +++ 27789 99 +++++ +++ +++++ +++
+++++ +++
,16000M,156512,98,247520,47,59560,27,83138,60,143588,21,792.8,7,16,+++++
,+++,+++++,+++,27789,99,+++++,+++,+++++,+++,+++++,+++
Writeback and Adaptive Readahead:
bash-2.05b$ bonnie++ -d bonnie -s 16000:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
16000M 155542 97 246910 47 60356 26 82798 60 143321 21 787.3 6
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 6329 49 +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
,16000M,155542,97,246910,47,60356,26,82798,60,143321,21,787.3,6,16,6329,
49,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
(As a side note- according to the controller docs, Adaptive read ahead
reads ahead sequentially if there are two reads from sequential sectors,
otherwise it doesn't).
So, I'm thinking that the RAID controller doesn't really help with this
too much- I'd think the OS could do a better job deciding when to read
ahead. So, I've set it back to no readahead and the next step is to look
at OS level file system tuning. Also, if I have time, I'll try doing
RAID 1 on the controller, and RAID 0 on the OS (or vice versa). Since I
have 6 disks, I could do a stripe of 3 mirrored pairs (raid 10) or a
mirror of two striped sets of 3 (0+1). I suppose theoretically speaking,
they should have the same performance characteristics, however I doubt
they will in practice.
Thanks,
Bucky
- Luke
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?