Kernel Asynchronous IO on HP-UX
EsquivelR
I have two Unix servers which run Online 7.23.UC7 and Online 7.14.UD6. Both
servers are running HP-UX 10.20. Has anyone out there had any experience using
Kernel Asynchronous IO? I'm looking for any known "gotchas" or testimonials on
increased performance.
Douglas Agnew
but not well. At 7.23, I don't know what the situation is, but it may require
changes from HP to resolve.
David Andrieu
Hello,
we are using KAIO on HP-UX 10.01, Online 7.23.UC1 on
2 servers, in a production environment with no problem for
more than 1 year. We are testing HP-UX 10.20 with IDS 7.30.UC2
and have experienced some problems when restoring a tape
(we have to unset KAIOON before ontape -r, because, apparently,
of a shared memory problem. It is a problem with HP-UX, NOT informix)
David...
-----Message d'origine-----
De : EsquivelR <esqu...@aol.com>
Ą : inform...@iiug.org <inform...@iiug.org>
Date : jeudi 1 octobre 1998 09:10
Objet : Kernel Asynchronous IO on HP-UX
Tom Grenier
KAIO -- had a machine that was not in production and benchmarked it. When I did
NOT use KAIO my test ran about 20% faster. It was not a long/exhaustive set of
tests -- I just had a process that ran for several hours and tried it both ways.
It may have been the specific test I ran and may have been another tuning
parameter I should have set with the KAIO -- but decided not to use KAIO.
We're about to move to 7.30UC2 -- I'd be interested if anyone has any experience
with KAIO w/7.30.
Tom Grenier
DBA, Biztravel.com
Chuck Renaud
> We're about to move to 7.30UC2 -- I'd be interested if anyone has any
> experience with KAIO w/7.30.
My experience is approximately a 15% improvement using KAIO,
regardless of the version of IDS. Machines include K2xx, D3xx
and V2250 and OS's include 10.10, 10.20 and 11.00. I have never
seen a performance degradation using KAIO, but I have run into
people who thought it was enabled when it wasn't. I haven't
personally run into any of the reported problems with KAIO,
though...I wouldn't be surprised to find out some quirks under 10.10
or 11.
--Chuck
Pete Smith
It reads thus:
HP KAIO HAS A SIGNIFICANT I/O BOTTLENECK WHICH CAN CAUSE 30% TO 75% IN
PERFORMANCE
The general symptom of the problem is that HP KAIO (native kernel
async I/O) is *much* slower than Informix AIO. The impact is
anywhere from 30% to >100% slowdown in disk I/O.
The problem lies in the server's I/O polling mechanism. When the
server issues an I/O request through KAIO, it must poll the HP-UX
asynchronous driver to determine when the request has been
fulfilled. If a request has not yet been filled, the I/O thread may
choose to go to sleep for 10 milliseconds before checking the request
status again. Up through ODS 7.1, the I/O thread would sleep using a
select() system call. This was changed in 7.2 to use the newer and
lighter nanosleep() system call. Performance tests showed this was
lighter on the CPU, and still gave us the desired sleep behavior.
Sometime after 7.2 was released (we believe), nanosleep() was changed
to correctly conform to its IEEE standard. One of the impacts is
that we can no longer sleep for just 10 milliseconds. On average,
the I/O thread is now sleeping for almost twice that long (when it
does not need to) and this is why we believe KAIO performance has
gotten so much worse.
I know of one of customer's running into this when upgrading from 7.13 to
7.24.UC5.
We don't use KAIO on our HP kit because of the performance hit.
I'm not aware of any release which has fixed this, either
HTH
Pete Smith
DBA
Aethos Communication Systems
Chuck Renaud
> Defect 92132 is one such reported problem:
It appears a coding fix was put in for 7.3 and 9.1...
--Chuck
John F. Miller III
so those of you that make it to the end I have include some HP Kaio tuning
tips.
There have been three different version of HP KAIO over the years which are
basically the same except their polling medthods. You can tell which version of KAIO
by the line put in the online.log at startup
Version 1 "HPUX Version XXX -> Using select based KAIO"
Version 2 "HPUX Version XXX -> Using flag style KAIO"
Version 3 "HPUX Version XXX ->Using flag/select style KAIO"
Version 1
It uses the select() system call to poll the OS for IO completions
status. This causes the kernel to use excess amounts of system
time due to the expensive nature of select() system call on HP.
Advantages:
Very fast
Disadvantages:
As the system gets busy the select() system call gets
very expensive.
Online Versions
7.1, 7.2, 7.3
Version 2
It use a memory address registered with the kernel to poll the OS
for IO completions status.
When the server issues an I/O request through KAIO, it must
poll the HP-UX asynchronous driver to determine when the
request has been fulfilled. If a request has not yet been
filled, the I/O thread may choose to go to sleep for 10
milliseconds before checking the request status again. Up
through ODS 7.1, the I/O thread would sleep using a select()
system call. This was changed in 7.2 to use the newer and
lighter nanosleep() system call. Performance tests showed this
was lighter on the CPU, and still gave us the desired sleep
behavior. Sometime after 7.2 was released (we believe),
nanosleep() was changed to correctly conform to its IEEE
standard. One of the impacts is that we can no longer sleep
for just 10 milliseconds. On average, the I/O thread is now
sleeping for almost twice that long (when it does not need to)
and this is why we believe KAIO performance has gotten so much
worse.
Advantages:
Signigicant reduction in resources
Disadvantages:
When the system is not busy the kaio poll thread will
wait for I/O an extra 6 to 8 ms (Defect #92132)
Not supported prior to HPUX 10.10
Online Versions
7.2, 7.3
Version 3
It use the memory address registered with the kernel to poll the
OS for I/O completions status when the system is busy, but when
the system goes idle and waits on I/O completions the system
will not use nanosleep as in version 2, but will use the select()
system call to wait.
Advantages:
Signigicant reduction in resources
Very fast.
In direct response to Defect #92132
Disadvantages:
None found to date.
Online Versions
7.30.UC3 and above and 9.14.UC1 and above
Question:
Using the select() system call won't this cause the
same performance issues as kaio version 1?
Answer:
No it will not. The problem with version one was that
as the system became busier the amount of I/O inceased
which caused an increase in select() calls to
be made. In the new method (version 3) we only use the
select system call if the system is idle and is wait on
I/O. In addition there is a benefit of using the
select system call which is the ability to interupt the
timer if and I/O completes before the timer expires.
This will give early notification of the I/O.
Tune Tips for HP KAIO
RESIDENCY
In version 7.30 for best KAIO performance set RESIDENT flag
in the onconfig to -1. In versions prior to version 7.30 set the
resident flag to 1. This will reduce a significant amount of
kernel locking that must be done on non-resident segments.
NOTE:
Not setting the RESIDENT flag in the onconfig will
cause an additional 8KB of memory to be allocated because the
online system will require assurance that this flag
is on its on OS memory page. If the RESIDENCY flag is
set this assurance does not have to be done.
IFMX_HPKAIO_NUM_REQ
This specifies the maximum number of simultaneous KAIO request
online can process at one time. This can currently be
set between HPKAIO_MAX_CONCURRENT (5000) and
HPKAIO_MIN_CONCURRENT (10) with a default value of
HPKAIO_DEF_CONCURRENT 1000. The only draw back to
setting this higher is memory. Below is the formula
to determine how much memory will be used:
bytes of memory = (# of cpuvps)*((12*IFMX_HPKAIO_NUM_REQ)+4)
The default setting should take about 12KB per cpu vp.
If you see the error
"KAIO: out of OS resources, errno = %d, pid = %d"
then you should consider raising IFMX_HPKAIO_NUM_REQ.
Hope this helps and answers alot of your questions,
---jmiller
John Miller
Pete Smith wrote:
> Defect 92132 is one such reported problem:
> It reads thus:
>
> HP KAIO HAS A SIGNIFICANT I/O BOTTLENECK WHICH CAN CAUSE 30% TO 75% IN
> PERFORMANCE
> The general symptom of the problem is that HP KAIO (native kernel
> async I/O) is *much* slower than Informix AIO. The impact is
> anywhere from 30% to >100% slowdown in disk I/O.
> The problem lies in the server's I/O polling mechanism. When the
> server issues an I/O request through KAIO, it must poll the HP-UX
> asynchronous driver to determine when the request has been
> fulfilled. If a request has not yet been filled, the I/O thread may
> choose to go to sleep for 10 milliseconds before checking the request
> status again. Up through ODS 7.1, the I/O thread would sleep using a
> select() system call. This was changed in 7.2 to use the newer and
> lighter nanosleep() system call. Performance tests showed this was
> lighter on the CPU, and still gave us the desired sleep behavior.
> Sometime after 7.2 was released (we believe), nanosleep() was changed
> to correctly conform to its IEEE standard. One of the impacts is
> that we can no longer sleep for just 10 milliseconds. On average,
> the I/O thread is now sleeping for almost twice that long (when it
> does not need to) and this is why we believe KAIO performance has
> gotten so much worse.
>
> I know of one of customer's running into this when upgrading from 7.13 to
> 7.24.UC5.
> We don't use KAIO on our HP kit because of the performance hit.
> I'm not aware of any release which has fixed this, either
>
Clem Akins
KAIO on HP-UX can bring a good performance increase (~10%) or, under
certain conditions, can result in a severe (30-100%) decrease in
disk IO speed. Informix and HP continually tweak their respective
code lines to get the most out of each system, which sometimes
results in issues like this. For example, Informix changed the way
the "select ()" and the "nanosleep ()" calls worked from version
7.1 to 7.2 to 7.3 at the same time that HP changed the way these
calls are implemented in the o/s.
Your best plan is to read the release notes and machine notes for
Informix and HP, use the very latest versions of each that you can,
and then to verify for yourself how the feature runs in your
environment. Be cautious about general statements and anecdotes.
One of the nice things about KAIO is that it is pretty easy to turn
on and off.
HTH,
Clem
At 02:37 PM 01-10-98 GMT, Douglas Agnew wrote:
>We were using 7.21 on HPUX 10.20 and were told NOT to use KAIO. Seems it
works,
>but not well. At 7.23, I don't know what the situation is, but it may
require
>changes from HP to resolve.
>
>EsquivelR wrote:
>
>> I have two Unix servers which run Online 7.23.UC7 and Online 7.14.UD6. Both
>> servers are running HP-UX 10.20. Has anyone out there had any experience
using
>> Kernel Asynchronous IO? I'm looking for any known "gotchas" or
testimonials on
>> increased performance.
>>
>> esqu...@aol.com
>
>
>
>
>
_____________________________________________________________
Clem Akins (aka cl...@informix.com) Menlo Park x5937
Customer Course Development Informix Software, Inc
Last seen: Pondering weak and weary over many quaint and
curious volume of Informix lore...