Interpreting WCHAN in a meaningful way
J.Smith
We currently are experiencing some problems on one of our AIX hosts,
where the system is spending a lot of time in iowait. Since the disks
are only at about half of their maximum throughput, it seems unlikely
that the throughput of the disk subsystem itself is the bottleneck
here. Also, the memory is only about half way used so that seems like
an unlikely candidate as well. I get the impression that there is some
sort of lock in the kernel that multiple processes need to access
simultaneously, but what is it ?
When we run ps, we see a lot of processes that have some value in the
WCHAN field. From the documentation I get the impression that this
field represents some variable in the kernel that represents the event
that the process is waiting on. However, the values displayed don't
mean all that much to me as a sysadmin. Is there some that doesn't
require having asource code access or using a kernel debugger to
determine in a more meaningful way what the process is actually
waiting on ?
Sincerely,
John Smith
J.Smith
>
> Is the WCHAN field displaying an address? If so, you can probably
> just use nm against /unix to find the address closest to the one
> in the ps output. Not sure if that'll help, but it kind of answers
> your question.
Well when I use the BSD style ps command (without the '-' switch), I
get the meaningless name of 'EVENT' in the WCHAN field. When I use the
SYSV style ps command (including the '-' switch) I do get values
instead of names. But these values (nor anything in the same range)
are nowhere to be found in the kernel. I guess that that means that
these are not symbolic constants, but rather dynamically generated
values ?
By the way, all processes that have values in the WCHAN field are
actually listed as 'A' active instead of 'S' sleeping by the ps
command. I always thought that processes waiting on an event were put
to sleep by the kernel until the event actually occurred ? Or does
this mean that I am looking at spin locks here, which continuously
keep polling the lock and therefore remain Active instead of Sleeping
?
>
> Sorry. I didn't notice the massive cross-post. Setting Followup-To
> to just comp.unix.aix, where it's most likely to be answered.
>
The reason I cross posted is because - believe it or not - most people
who actually know how a kernel works appear to be hanging out on the
*bsd* newsgroups these days , and not on the mostly sysadmin related
comp.unix.aix. Then again, the sysadmins that know aix hang out on
comp.unix.aix ...
Anyways, thanks for the pointers.
Sincerely,
John Smith
mark taylor
> In comp.unix.aix J.Smith <lbal...@hotmail.com> wrote:
> JS> Hi.
>
> [ snip ]
>
> Sorry. I didn't notice the massive cross-post. Setting Followup-To
> to just comp.unix.aix, where it's most likely to be answered.
>
>
> Nicholas
man "ps"
/WCHAN
WCHAN
(-l flag) The event for which the process or kernel thread is
waiting or
sleeping. For a kernel thread, this field is blank if the kernel
thread is
running. For a process, the wait channel is defined as the wait
channel of the
sleeping kernel thread if only one kernel thread is sleeping;
otherwise a star
is displayed.
ok, so that does not mean a lot to you, but you are interested in wait
i/o threads right ?
use nmon, then subcommand "t", then "5" to order processes in
character throughput, this will give you your rogue threads....
an altrernative would be.
echo "th -w WPGIN" | kdb
Then use trace and curt / splat (included in the perfpmr package) and
tprof to see what those threads are doing and if they are indeed
spinning on locks or stuck in a routine / driver loop or any kind.. if
there is a problems you will be able to see the kernel ticks will
stick out like a sore thumb..
You should also use filemon to see which disks are being hit and their
throughput and work from there on.... you dont mention your AIX level,
disk subsystem type, LVM layout, vmtune settings.. Is this a database
node ? is there a SAN ? raw or jfs or jfs2 ? how many adapters, ESS ?
level of SDD ?
Are you hitting FS buffers ?
vmtune -a | grep cnt
Any high numbers ? then increase you FS buffers with vmtune -b -B and
if you have raw LVs then look into vmtune -u for LVM raw buffers.
actually, check out the tuning docs before playing around -->
http://publibn.boulder.ibm.com/doc_link/en_US/a_doc_lib/aixbman/prftungd/prftungdtfrm.htm
If it is a db node, then check out your tablespace layout, you may be
hitting table / index contention problems.
That give you something to get you going...
Rgds
Mark Taylor
Tao Chen
> Hi.
> We currently are experiencing some problems on one of our AIX hosts,
> where the system is spending a lot of time in iowait.
Do you have some *real* performance problem, like slow response time
from application or lower_than_expected data throughput, or, it's
just those high iowait value in vmsttat output?
> I get the impression that there is some
> sort of lock in the kernel that multiple processes need to access
> simultaneously, but what is it ?
lock wait/contention doesn't contribute to iowait.
iowait is strictly related to I/O.
> When we run ps, we see a lot of processes that have some value in the
> WCHAN field. From the documentation I get the impression that this
> field represents some variable in the kernel that represents the event
> that the process is waiting on.
> However, the values displayed don't
> mean all that much to me as a sysadmin. Is there some that doesn't
> require having asource code access or using a kernel debugger to
> determine in a more meaningful way what the process is actually
> waiting on ?
If you're familiar with kdb, you can check a thread's wait type
with 'th -w <wtype>' :
(0)> th -w
Missing wtype:
NOWAIT
WEVENT
WLOCK
WTIMER
WCPU
WPGIN
WPGOUT
WPLOCK
WFREEF
WMEM
WLOCKREAD
WUEXCEPT
WZOMB
WLOCK_WX
VMMWAIT
The total number of threads with WPGIN type is the
number showed up as 'b' column in vmstat.
A waiting thread has only one wtype, it may be
waiting for I/O (WPGIN), or for a lock (WLOCK).
If it's waiting for a lock, the wait time won't
be charged into iowait.
Keep in mind that
1. The I/O wait time are charged into vmstat's iowait
*ONLY* when the CPU is idle, i.e., no matter how slow/busy
you disks are, as long as the CPUs are 100% busy (occupied
by other threads that don't have to wait for I/O),
you won't see *any* iowait in vmstat.
2. On the other hand, if the CPU is idle, iowait time is charged
if there's *any* pending I/O, i.e. one thread waiting on
one I/O does the same to iowait as thousands of threads
waiting on millions of I/Os.
Conclusion? iowait doesn't really tell anything about your I/O
performance. What really matters: I/O patterns ( size/location )
and the response time for each single I/O.
'wchan', among other event structures you can see in kdb is mostly
used for analyzing event synchronization problems in AIX kernel.
Other posts have mentioned some common tools to do I/O analyzing.
Hope this helps a little bit.
Regards,
Tao
mark taylor
> 'wchan', among other event structures you can see in kdb is mostly
> used for analyzing event synchronization problems in AIX kernel.
> Other posts have mentioned some common tools to do I/O analyzing.
>
> Hope this helps a little bit.
>
> Regards,
> Tao
the moment I may pay a visit this year :)
Mark Taylor
Tao Chen
> Hey Tao, hows Austin ? with the Pound so good against the dollar at
> the moment I may pay a visit this year :)
>
> Mark Taylor
Hi Mark!
So you are *the* Mark Taylor I know of!
I didn't know you're so into AIX performance tuning though. LOL.
Come visit us, now is the best season!
Tao
J.Smith
>
> lock wait/contention doesn't contribute to iowait.
> iowait is strictly related to I/O.
>
Ok, thansk for the info. By the way, does that strictly mean *disk*
I/O, or network I/O as well ?
>
> If you're familiar with kdb, you can check a thread's wait type
> with 'th -w <wtype>' :
> (0)> th -w
> Missing wtype:
> NOWAIT
> WEVENT
>
Well when I use the BSD style ps command (without the '-' switch), I
get the name of 'EVENT' in the WCHAN field. I guess that this should
correspond to the 'WEVENT' you mention for kdb. But what does this
mean then exactly ? Obviously it means "waiting-for-an-event", but
what kind of event exactly ? How do I get more info on this ?
>
> The total number of threads with WPGIN type is the
> number showed up as 'b' column in vmstat.
>
Ok, so if my wait type is 'WEVENT', then this is not the reason for
the iowait we see ?
>
> Hope this helps a little bit.
>
A bit
;)
Sincerely,
John Smith.
J.Smith
>
> lock wait/contention doesn't contribute to iowait.
> iowait is strictly related to I/O.
>
Ok, thansk for the info. By the way, does that strictly mean *disk*
I/O, or network I/O as well ?
>
> If you're familiar with kdb, you can check a thread's wait type
> with 'th -w <wtype>' :
> (0)> th -w
> Missing wtype:
> NOWAIT
> WEVENT
>
Well when I use the BSD style ps command (without the '-' switch), I
get the name of 'EVENT' in the WCHAN field. I guess that this should
correspond to the 'WEVENT' you mention for kdb. But what does this
mean then exactly ? Obviously it means "waiting-for-an-event", but
what kind of event exactly ? How do I get more info on this ?
>
> The total number of threads with WPGIN type is the
> number showed up as 'b' column in vmstat.
>
Ok, so if my wait type is 'WEVENT', then this is not the reason for
the iowait we see ?
>
> Hope this helps a little bit.
>
mark taylor
mark taylor
Working with Grover and Tom had a lasting effect on me.....
Tomas
Użytkownik "Nicholas Dronen" <ndr...@io.frii.com> napisał w wiadomości
news:402d88be$0$197$7586...@news.frii.net...
> In comp.unix.aix J.Smith <lbal...@hotmail.com> wrote:
> JS> Hi.
>
> [ snip ]
>
> Sorry. I didn't notice the massive cross-post. Setting Followup-To
> to just comp.unix.aix, where it's most likely to be answered.
>
> Regards,
>
> Nicholas
>
> --
> http://www.faqs.org/rfcs/rfc1855.html
> 3.1.1 General Guidelines for mailing lists and NetNews
> 3.1.3 NetNews Guidelines