Problem of performance of I/O in AIX 6.1
MPC
First of all, sorry for my english.
We have a performance problem about I/O over AIX 6.1
Our system in production, a Pseries 9117-570 (a power 5 host) , is
faster than new one 9117-MMA (a power 6 host) when doing I/O working
processes. The storage subsystems is the same in both hosts (Symmetrix
EMC), and we tested same LUN in both systems (export vg from one and
imported in the other one). In addition, first one (the faster) has
2GB fiber channel cards, and the lowest has 4GB fiber cards.....we
can't find any plausible explanation to this behaviour.
Any ideas about what can we do???
PD: checked params about vmtune (minperm %, maxperm% & maxclient%) ,
Disk pacing and AIO servers.
We would very appreciated.
Regards
Montse
Niel Lambrechts
> Hello:
> First of all, sorry for my english.
> We have a performance problem about I/O over AIX 6.1
> Our system in production, a Pseries 9117-570 (a power 5 host) , is
> faster than new one 9117-MMA (a power 6 host) when doing I/O working
> processes. The storage subsystems is the same in both hosts (Symmetrix
> EMC), and we tested same LUN in both systems (export vg from one and
> imported in the other one). In addition, first one (the faster) has
> 2GB fiber channel cards, and the lowest has 4GB fiber cards.....we
> can't find any plausible explanation to this behaviour.
>
> Any ideas about what can we do???
>
> Disk pacing and AIO servers.
Hi,
What tests did you do to compare the performance?
Guessing about some common issues as a starting point:
Any differences in 'powermt display dev=all'?
Do you have exactly the same amount of HBA devices?
How does fcstat fcX compare between the hosts, are the 4Gb cards connected at 4GB?
Any differences in output of 'ioo -a'?
Any differences in output of 'oslevel -s'?
Is the syncd frequency the same on both (ps -ef|grep sync[d])?
Please post some output if possible.
Regards,
Niel
MPC
Hi Neil,
Thanks for your quickly response.
> Do you have exactly the same amount of HBA devices?
No in the faster host we have 2 HBA (2Gb) and in the lowest we have 4
(4Gb)
> How does fcstat fcX compare between the hosts, are the 4Gb cards connected at 4GB?
Yes. The cards support 8Gb (IBM FC5735) but the switch only support
4Gb. They negotiated at 4GB
> Any differences in output of 'ioo -a'?
In AIX 5.3 (faster)
j2_atimeUpdateSymlink = 0
j2_dynamicBufferPreallocation = 16
j2_inodeCacheSize = 400
j2_maxPageReadAhead = 128
j2_maxRandomWrite = 0
j2_maxUsableMaxTransfer = 512
j2_metadataCacheSize = 400
j2_minPageReadAhead = 2
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_nRandomCluster = 0
j2_nonFatalCrashesSystem = 0
j2_syncModifiedMapped = 1
j2_syncdLogSyncInterval = 1
jfs_clread_enabled = 0
jfs_use_read_lock = 1
lvm_bufcnt = 9
maxpgahead = 8
maxrandwrt = 0
memory_frames = 11534272
minpgahead = 2
numclust = 1
numfsbufs = 196
pd_npages = 65536
pgahd_scale_thresh = 0
pv_min_pbuf = 512
sync_release_ilock = 0
In AIX 6.1 (lowest)
aio_active = 0
aio_maxreqs = 65536
aio_maxservers = 30
aio_minservers = 3
aio_server_inactivity = 300
j2_atimeUpdateSymlink = 0
j2_dynamicBufferPreallocation = 16
j2_inodeCacheSize = 400
j2_maxPageReadAhead = 128
j2_maxRandomWrite = 0
j2_metadataCacheSize = 400
j2_minPageReadAhead = 2
j2_nPagesPerWriteBehindCluster = 32
j2_nRandomCluster = 0
j2_syncPageCount = 0
j2_syncPageLimit = 16
lvm_bufcnt = 9
pd_npages = 65536
posix_aio_active = 0
posix_aio_maxreqs = 65536
posix_aio_maxservers = 30
posix_aio_minservers = 3
posix_aio_server_inactivity = 300
> Is the syncd frequency the same on both (ps -ef|grep sync[d])?
Yes it's the same
Any more ideas??? we are so desperate......
Hajo Ehlers
> Hello:
> First of all, sorry for my english.
> We have a performance problem about I/O over AIX 6.1
> Our system in production, a Pseries 9117-570 (a power 5 host) , is
> faster than new one 9117-MMA (a power 6 host) when doing I/O working
> processes. The storage subsystems is the same in both hosts (Symmetrix
> EMC), and we tested same LUN in both systems (export vg from one and
> imported in the other one). In addition, first one (the faster) has
> 2GB fiber channel cards, and the lowest has 4GB fiber cards.....we
> can't find any plausible explanation to this behaviour.
>
> Any ideas about what can we do???
What kind of problem do you have ?
Its is related to IOPS or bandwidth ?
OS
1) AIX 6.1 is 64bit where AIX 5.3 can be 32 or 64
2) AIX 6.1 needs more memory leaving less for caching and other
( nmon and svmon are your friends )
2.1) So check the page fault rate and memory distribution
IBM/HBC
1) Compare the HBA settings. ( lsattr -El fcsX -a num_cmd_elems ) - In
case the amount is low you might suffer in case of a high IOPS rate.
"iostat -a 1" is a nice tools to get some numbers.
EMC
0) Is powerpath used at all on the new machine. ( VERIFY )
1) EMC Symetric is an active/active array thus both SP can be used
simultaneously . In case powerpath is used - is the license active ?
2) Powerpath - are any LUN trespass or not. Are all pathes used ?
Also do some basic testing with at least using dd and with different
block sizes. ( between 1k - 1M ) .
cheers
Hajo
Niel Lambrechts
> On 15 jun, 21:18, Niel Lambrechts<n...@devnull.org> wrote:
>> On 06/15/2009 08:27 PM, MPC wrote:
>>
>>> Hello:
>>> First of all, sorry for my english.
>>> We have a performance problem about I/O over AIX 6.1
>>> Our system in production, a Pseries 9117-570 (a power 5 host) , is
>>> faster than new one 9117-MMA (a power 6 host) when doing I/O working
>>> processes. The storage subsystems is the same in both hosts (Symmetrix
>>> EMC), and we tested same LUN in both systems (export vg from one and
>>> imported in the other one). In addition, first one (the faster) has
>>> 2GB fiber channel cards, and the lowest has 4GB fiber cards.....we
>>> can't find any plausible explanation to this behaviour.
>>> Any ideas about what can we do???
>>> PD: checked params about vmtune (minperm %, maxperm%& maxclient%) ,
>>> Disk pacing and AIO servers.
>> Hi,
>>
>> What tests did you do to compare the performance?
Please try and answer this. What do you do to test your I/O performance?
If possible, include a few lines of output of 'vmstat 1' and 'iostat 1' on the slow system while you do a performance test.
>> Any differences in output of 'ioo -a'?
> In AIX 5.3 (faster)
<snip>
> In AIX 6.1 (lowest)
> aio_active = 0
AIO is not active in your AIX 6.1 environment.
If you are using Oracle or some other product that requires AIO, this could make a difference.
Regards,
Niel
Andreas Schulze
>[...]
> We have a performance problem about I/O over AIX 6.1
> Our system in production, a Pseries 9117-570 (a power 5 host) , is
> faster than new one 9117-MMA (a power 6 host) when doing I/O working
> processes. The storage subsystems is the same in both hosts (Symmetrix
>
> Regards
> Montse
Depending on what workload you have on your server you might be hit by
a bug in the Power6 CPU. That processor appears to have performance
problems with certain workloads such as batch processing. Should that
resemble your problem you might need a software patch (i.e. new
binary) that works around that problem.
MPC
Hi Andreas,
You put us under a great way to solve it. Latest test that we did is
to create a LPAR with AIX 5.3 in the POWER6 server, and ooops!!! the
results are the same....poor performance. We called IBM but they
didn't tell us that were a bug with POWER6 CPUs. Andreas,
we will be very appreciated if you can tell us the bug number or some
link to read more about this bug.
Thanks in advanced.
Regards.
MPC
Hi Neil,
we saw this behaviour and ask to IBM techs, they told us that
aio_server in 6.1 are completely auto. When the system needs to start
the aio daemon
it does, then the parameters max_aio_server, min_aio_servers..etc does
not apply in 6.1
Isn't it?
Regards
Mark Taylor
How are you generating the I/O ?
Are you using shared CPU ?
Have you tested dedicated CPU ?
Do you have SMT on ?
Turn SMT off and test again .. what are the results ?
Collect perfpmr for **exactly** the same workload on each system and
send it to IBM for analysis ..
Rgds
Mark Taylor
Niel Lambrechts
> On 17 jun, 01:48, Niel Lambrechts<n...@devnull.org> wrote:
>> On 06/16/2009 04:37 PM, MPC wrote:
>>> On 15 jun, 21:18, Niel Lambrechts<n...@devnull.org> wrote:
>> AIO is not active in your AIX 6.1 environment.
>>
>> If you are using Oracle or some other product that requires AIO, this could make a difference.
> we saw this behaviour and ask to IBM techs, they told us that
> aio_server in 6.1 are completely auto.
That is not entirely correct, the aio_minservers and aio_maxservers settings still apply.
You do not seem to have to activate it explicitly as I though when I first posted, aio_active seems to be a static tunable that just indicates whether AIO is in
use or not.
So perhaps you should at least match your AIX 6.1 minservers/maxservers to AIX 5.3 to rule out AIO.
Regards,
Niel
Mark Taylor
benefit out of power6 you have to recompile your app on power6 because
there were some significant changes between the p5 and p6 chips (out
of order execution etc) ..
Obviously if you are driving your workload via a Database (Oracle)
then this is not going to be so easy if at all possible ..
Rgds
Mark Taylor
MPC
Andreas,
Please , can you tell us more about the bug in the power6 cpu.
Regards
Montse