vhand runaway on Quad PIII
Stuart J. Browne
process go crazy and consume excessive amounts of processor time, and start
causing strange disruptions to TCP stacks (namely JetDirect printers stop
working!).
TA109419 doesn't seem to be relevant in this case (not 504, and it states
this issue was fixed in 505).
Does anybody have any idea's as to what would be causing this to go haywire,
and any suggestions as to how to fix it?
--------------------------
HP NetServer LH6000
QUad PIII 600 Xeon
512 MB RAM
6 x 18Gig RAID array (HP NetRAID)
--------------------------
435# ps -elf | grep vhand
61 O root 2 0 79 95 20 fb1172b0 0 - Jul-18
? 04:52:02 vhand
436# w |head -1
1:39pm up 48 days, 3:05, 105 users, load average: 1.75, 1.67, 1.36
554# uname -X
System = SCO_SV
Node = skglh
Release = 3.2v5.0.5
KernelID = 98/07/02
Machine = i80386
BusType = ISA
Serial = 2HD012703
Users = 105-user
OEM# = 0
Origin# = 1
NumCPU = 4
558# customquery listpatches | head
SCO:Unix::5.0.5Eb rs505a.Unix505.1.0a oss497c.Unix505.1.0a
oss600a.Unix505 OSS621A.505.SCO.Unix.RTS
--------------------------
bkx
Bela Lubkin
> I have a Quad PIII machine of which (every 48 days or so) has a VHand
> process go crazy and consume excessive amounts of processor time, and start
> causing strange disruptions to TCP stacks (namely JetDirect printers stop
> working!).
>
> TA109419 doesn't seem to be relevant in this case (not 504, and it states
> this issue was fixed in 505).
>
> Does anybody have any idea's as to what would be causing this to go haywire,
> and any suggestions as to how to fix it?
>
> --------------------------
> HP NetServer LH6000
>
> QUad PIII 600 Xeon
> 512 MB RAM
> 6 x 18Gig RAID array (HP NetRAID)
> --------------------------
> 558# customquery listpatches | head
> SCO:Unix::5.0.5Eb rs505a.Unix505.1.0a oss497c.Unix505.1.0a
> oss600a.Unix505 OSS621A.505.SCO.Unix.RTS
Well, TA #109419 has a few of the details. Its supposed fix of
disabling HPPS is rather peculiar...(*)
The most typical cause of vhand spinning is running out of DMAable
memory. OpenServer's kernel memory allocator distinguishes between
"DMAable" memory (meaning ISA-DMAable, that is, memory below 16MB) and
non-DMAable. In most modern systems, the only device that could
possibly need ISA DMAable memory is a floppy drive. Other rarer users
would include: ISA sound cards, Adaptec 154x host adapters, and old ISA
QIC02/QIC36 tape adapters.
Unfortunately, it's _really_ easy for kernel code to mistakenly request
DMAable memory. You have to explicitly request non-DMAable, or else
your request is understood to be for DMAable memory. There is a lot of
code in the kernel and in third party drivers which mistakenly requests
DMAable. This doesn't show up in testing because it's a
"no-consequence" bug. Using DMAable memory doesn't hurt the driver at
all, it's just a waste.
Except... as overall system memory gets larger over time, people are
doing more with systems. The same drivers that used to mistakenly
allocate 512K of DMAable may now allocate 1MB, or 2MB... Pretty soon
you're completely out of that tiny 16MB window!
Each release of OpenServer has corrected some amount of code that
mistakenly requests DMAable. There was a major push to fix these
problems in OSR507, and it is almost completely clean of such mistaken
requests. (Some are rather hard to root out because multiple drivers
use a single memory allocation service that doesn't give a way to
specify DMA requirements, and _one_ of those drivers actually does need
DMAable, so they all have to accept it.)
When DMAable memory is exhausted and someone requests more of it, vhand
starts spinning, looking for memory to use.
As an administrator, you don't have control over most uses of DMAable
memory. However, there is one large user that you _do_ control: the
buffer cache. The kernel parameter PLOWBUFS controls how many 1K disk
buffers are allocated in DMAable address space. You can see how many
are currently being used by running `grep bufs /usr/adm/messages` (this
might produce a lot of output). On a 507 system that I checked, I get:
kernel: Hz = 100, i/o bufs = 12752k (high bufs = 11728k)
This system has about 13000 I/O buffers, 1024 of which are DMAable
(total minus "high"). You want almost all of your buffers to be "high",
which are fine for use by PCI host adapters and IDE drives.
PLOWBUFS sets what percentage of total buffers should be allocate below
16MB. If you have 20000 total buffers, setting PLOWBUFS to 1 (its
lowest setting) gets you 200 DMAable buffers, which is only 1/80 of the
total 16MB space. If you have default parameters for 505, you probably
have 6652 total buffers and about 2000 DMAable buffers, so you could
save 1.8MB right there. That might be enough to _never_ hit the
problem, or might only push the problem horizon out from 48 days to a
year or so...
If you have non-default parameters then buffers might be consuming much
more of the low 16MB, and you could improve things much more.
Starting with OpenServer 506, the PLOWBUFS parameter has a dual meaning.
Values <= 100 mean to allocate that percentage of total buffers from
DMAable memory. Values > 100 mean to allocate exactly that many
buffers. The machine I was looking at has PLOWBUFS=1024, which is why
it got exactly that many DMAable buffers. That's the default setting in
507, chosen to allow floppy drives and old ISA host adapters to work.
In a system where the only user is the floppy drive, it could probably
be set to the minimum, 101, without negative consequences. Setting it
lower than that wouldn't be useful -- if you're going to run out of
15.9MB then you're going to run out of 16.0MB moments later.
Other things that you can change (including PLOWBUFS, so this is a
comprehensive remedy list):
- reduce PLOWBUFS to 101 (on 506 or later), or 1 (any release) to
sharply decrease the number of DMAable disk buffers
- edit /etc/conf/pack.d/str/space.c, change the value of `str_pool_mem'
from MEM_BUF to MEM_KVMAPPED (504 or 505: won't work on 500 or 502;
already changed on 506)
- remove NFS from the kernel if you are not using it -- it is a sloppy
waster of DMAable memory (fixed in 507)
- if you _are_ using NFS, edit /etc/conf/cf.d/mdevice, find all "nfsd"
entries, and make sure they have characteristics 'd' and 'P'. Be
careful: the mdevice file cannot be reconstructed from other files.
If there is more than one "nfsd" line, they will look different from
each other; they're _supposed_ to look different. Just add 'd' and
'P' to the third field of each entry, if they're missing.
- if you're on 504 or earlier and nothing else has worked, disable
HPPS
Note that PCI devices can DMA to any address in the system. The word
"DMAable" dates back to the ISA days. Reducing DMAable buffers will not
harm your modern I/O devices.
>Bela<
(*)Well, not all _that_ peculiar. HPPS was one of those drivers which
mistakenly requested DMAable memory. That was fixed starting with 505,
so disabling it would not help you.
Tony Lawrence
Re: vhand runaway on Quad PIII [possible FAQ material]
I squirrel away almost everything Bela writes anyway; they end up in my
http://aplawrence.com/Bofcusm/ archives and I add indexing keys to
(hopefully) make them easier to find.
However, I will pay close attention to messages with this suggestion in
them and WILL add them to the FAQ also.
--
Please note new phone number: (781) 784-7547
Tony Lawrence
Unix/Linux Support Tips, How-To's, Tests and more: http://aplawrence.com
Free Unix/Linux Consultants list: http://aplawrence.com/consultants.html
Stuart J. Browne
*blink* WOOHOO!! YES! YES!
:P
On systems of this size, we normally up the NBUF (and a few other) values,
and drop PLOWBUF to 20 (although this machine is set to 26 for some reason,
go figure). *Updates intranet documentation about buffer tuning* .. (Hey! I
didn't write 'em!).
This WILL be the cause!
The tell-tale sign I never saw before (or atleast didn't think overly much
of):
kernel: Hz = 100, i/o bufs = 150604k (high bufs = 141980k)CONFIG: Low
buffers unavailable, converted to high buffers
At 26% of 150MB, there most definately isn't enough, although going by the
number differences, there should still be some left assigned. But given the
errors, I'll drop PLOWBUF down to 1%, and wait a few months to see if the
problem re-occurs.
Thank you Bela.
bkx
"Bela Lubkin" <be...@caldera.com> wrote in message
news:2002110510...@mammoth.ca.caldera.com...
Bela Lubkin
> (please excuse top post):
>
> *blink* WOOHOO!! YES! YES!
>
> :P
>
> On systems of this size, we normally up the NBUF (and a few other) values,
> and drop PLOWBUF to 20 (although this machine is set to 26 for some reason,
> go figure). *Updates intranet documentation about buffer tuning* .. (Hey! I
> didn't write 'em!).
>
> This WILL be the cause!
>
> The tell-tale sign I never saw before (or atleast didn't think overly much
> of):
>
> kernel: Hz = 100, i/o bufs = 150604k (high bufs = 141980k)CONFIG: Low
> buffers unavailable, converted to high buffers
>
> At 26% of 150MB, there most definately isn't enough, although going by the
> number differences, there should still be some left assigned. But given the
> errors, I'll drop PLOWBUF down to 1%, and wait a few months to see if the
> problem re-occurs.
Setting PLOWBUFS to 1 will get you 1506 DMAable buffers vs. the
8624 you currently have, a savings of 7118K. Total DMAable space is
nominally 16MB but has usually shrunk to about 12MB by the time the
kernel is loaded and initialized. So this gets your initial DMAable
pool from about 3.5MB to about 10.5MB. Depending on what and why you're
needing even that much DMAable, the tripled pool should last at least 3x
as long and possibly "forever".
In private mail I asked you to run some other diagnostics. Can you run
those for at least a couple of weeks before changing PLOWBUFS and
rebooting? I really want to answer the question of how to determine
current usage...
>Bela<
Tony Lawrence
hundred or more at http://aplawrence.com/Bela/
Bela Lubkin
> In private mail I asked you to run some other diagnostics. Can you run
> those for at least a couple of weeks before changing PLOWBUFS and
> rebooting? I really want to answer the question of how to determine
> current usage...
Well, Stuart can call off the dogs, I found what I was looking for. It
was right there all along.
crash(ADM) has a command, "pfdfreeq", that does just what we need:
> pfdfreeq
pfdat free queues:
DMAABLE:
Max 2719 Limit 60 Gets 10227760 Next 1479
Free 1233 Low 60 Heads 8035190 Prev 2700
Reqd 0 High 180 Tails 2191084
DIRECT MAPPED:
Max 65791 Limit 1000 Gets 237072682 Next 22210
Free 28962 Low 1000 Heads 155227270 Prev 60127
Reqd 0 High 3000 Tails 81808583
UNMAPPED:
Max 64980 Limit 980 Gets 251869438 Next 75421
Free 33223 Low 980 Heads 195598941 Prev 124774
Reqd 0 High 2940 Tails 56238740
What is all this junk? It's really quite useful, actually. This is a
scorecard for free RAM in the system. Let's analyze it a bit...
First, it shows three kinds of memory, "DMAABLE", "DIRECT MAPPED" and
"UNMAPPED".
DMAABLE is memory that's reachable by the ISA DMA controller on the
motherboard -- memory below address 16MB. Although very little hardware
needs this sort of memory any more, some drivers ask for it by mistake,
so this small pool can be a bottleneck.
The distinction between DIRECT MAPPED and UNMAPPED is not a hardware
difference, but a difference in how the OpenServer kernel handles the
memory. DIRECT MAPPED pages can be mapped into kernel virtual address
space by simply adding C0000000 to their physical address. This makes
certain kinds of physical address-based I/O easier, so some drivers
prefer this sort of memory. UNMAPPED pages are more difficult for a
driver, and especially I/O hardware, to access; the kernel prefers to
use these for user process pages. Systems with less than 768MB of RAM
generally do not have any UNMAPPED pages.
Now, what are all those stats?
"Max" shows how many of each type of page were present when the memor
subsystem was initialized. This happens _after_ a few major
allocations, in particular the loading of the kernel itself and the
allocation of the buffer cache. If we add the three Max values here we
can see that we started out with 2719 + 65791 + 64980 = 133490 4K pages,
or about 520MB of RAM. The machine I'm looking at actually has 1GB RAM,
but it also has 450000 I/O buffers. Buffers + buf headers consume
478MB.
"Free" shows how many of each type are currently available. For
instance, 1233 4K DMAABLE pages, or about 5MB total, are available.
This system is not about to start thrashing over DMAABLE memory.
"Reqd" would be nonzero if no pages of a given type were available, and
someone was waiting for some.
"Low" and "High" are hints used by the paging/swapping subsystem. If
the number of free pages of a particular type goes below its Low point,
the kernel looks for pages it can write out to swap (and thus free).
This continues until the free count hits the High value for that page
type. The Low/High values come from the kernel parameters GPGSLO and
GPGSHI, split among the different page types according to their relative
abundance. (The system I'm looking at has GPGSLO = 2000, GPGSHI = 6000,
which are unusually high values.)
"Limit" just shows which of Low or High the kernel is currently working
towards. It is either actively paging out process pages because it hit
the Low value and is working towards High; or it isn't doing that, in
which case it's nominally working from High towards Low (though the true
direction of growth depends on system activity).
"Gets" shows how many pages of that type have ever been allocated.
"Heads" and "Tails" show how many times pages of that type have been
freed. Some types of page are freed to the head of the queue (to be
reused immediately); some, particularly pages that were mapped from a
disk files, are put at the tail of the queue. That's done so that if
the page is needed again before it gets used for something else, it
doesn't have to be reread from disk.
Finally, "Next" shows us the Page Frame Number of the next page that
will be allocated from each pool, and "Prev" shows the PFN of the page
that was most recently freed to the tail of that pool. PFNs are
addresses divided by 4096: a PFN of 2700, for instance, represents
address 11059200, or 0xA8C000. These stats probably aren't very useful
(and change rapidly, so what you see is wrong by the time you see it).
What good is it?
Most administrators will probably only be concerned with the "Free"
numbers. For most purposes it doesn't even matter what kinds of pages
are free, as long as some free pages exist. The exception would be if
some driver is trying to allocate a specific kind of page, and none are
available for a long time. This would show up as a persistent nonzero
value in "Reqd" (and the "vhand" process would spin, consuming CPU).
Such problems are almost always with the DMAABLE pool, which is limited
to a total of 16MB minus the size of kernel .data + .bss, minus other
low-memory allocations set up by the hardware, BIOS and /boot, minus
DMAABLE I/O buffers. Few systems are likely to have more than about
13MB in this pool, and it could be much less. The "Max" field can help
you determine whether tuning the kernel in various ways is actually
helping to grow the DMAABLE pool.
>Bela<
Stuart J. Browne
> I wrote (to Stuart J. Browne):
>
> > In private mail I asked you to run some other diagnostics. Can you run
> > those for at least a couple of weeks before changing PLOWBUFS and
> > rebooting? I really want to answer the question of how to determine
> > current usage...
>
> Well, Stuart can call off the dogs, I found what I was looking for. It
> was right there all along.
>
> crash(ADM) has a command, "pfdfreeq", that does just what we need:
>
> > pfdfreeq
wow.. nice.. and it almost makes sense too me..
after re-tuning PLOWBUF down to 1, and relink/reboot, it currently reads:
DMAABLE:
Max 2959 Limit 15 Gets 84214 Next
373
Free 434 Low 15 Heads 62641 Prev
2689
Reqd 0 High 24 Tails 19048
DIRECT MAPPED:
Max 83767 Limit 25 Gets 2752945 Next
9287
Free 15405 Low 25 Heads 1695508 Prev
85312
Reqd 0 High 39 Tails 989075
UNMAPPED:
Max 0 Limit 0 Gets 0 Next
Nomapq
Free 0 Low 0 Heads 0 Prev
Nomapq
Reqd 0 High 0 Tails 0
will keep an eye on it.
Thanks Bela.
bkx