read cache hit rates at 0%?
matth...@my-deja.com
I'm one of the the sys admins and DB admins for several SCO OpenServer
systems running Progress DB. The systems are all at 5.0.4 or 5.0.5.
Progress is version 8.2 on at least three of them. One system has,
unfortunately, a RAID 5 array for the DB filesystem (all the others are
simple mirrors/RAID 1). We did this a while ago before we realized
that RAID 5 is bad for DB's with a lot of write activity. On these
systems, we collect sar data. In addition, we average that sar data
for every weekday for read cache hit rates, CPU utilization, WIO, run
queue length, and paging.
On the system with RAID 5, the read cache hit rates are almost flat
0%. The only time the daily average goes much above 0% is on holidays
when they're closed. This would imply that it has something to do with
DB activity, perhaps. This server also has more RAM than any of the
others. One, for example, has 384 MB of RAM, another has 512 MB, and
this one has 896 MB. The DB's are all between 1 and 2 GB in size.
None of the systems are paging. I'm manually setting NHBUF and NBUF on
all of them. On the problem server, NBUF is 131072, and NHBUF is
32768. They used to be the same as the other servers at 65536 and
16384, but I recently changed them, hoping read cache hits would
increase. It did not. The other servers tend to be between about 70%
and 90%.
Why is this one server at 0%? I can't find any other kernel parameter
that I missed changing on that server that might cause this. The only
differences between this server and the others are the increased RAM
and the RAID 5 array (Compaq hardware RAID).
Any help would be greatly appreciated.
Sent via Deja.com
http://www.deja.com/
Tony Lawrence
That's puzzling. The RAID certainly is going to affect overall
performance, but I can't think of any reason that it would affect
the Unix buffer cache, at least not directly. However, the
mention of Compaq always raises a red flag because they put out
their own little daemon agents and maybe (pure wild assed
speculation) one of their little dooliewhoppers is stomping on
cache by reading stuff nilly-willy here and there- if it's not
their fault, then it sure sounds like SOMETHING is reading from
all over the place.
Have you compared blks/sec (sar -d) against the other servers?
All other things being equal, a high amount of disk activity that
was trashing cache would show up here and in breads/sec on sar
-b.
If you do see it there, finding the culprit is the next fun task
:-)
--
Tony Lawrence (to...@aplawrence.com)
SCO/Linux articles, help, book reviews, tests,
job listings and more : http://www.pcunix.com
Matt Harrell
Thanks. I checked into those things. sar -b and sar -d don't show any
big differences. Also, all of our servers are Compaqs, and they all
have more or less the same daemons running (varying with the version of
Smart Start). I'm baffled. It's acting like maybe Progress is
bypassing the OS I/O buffers. There is a startup parameter in progress
for this, but it's not set on this server. I am really stumped. Right
now, there is not a performance problem on the server (from the users'
perspectives), so I guess I'll have to let it go. It drives me nuts
not knowing why the server is doing this, though.
--
------------------
Matt Harrell
Plexus Systems
mh...@plex-sys.com
Dirk Hart
Here's my own speculation:
I wonder if the raid controller has its own memory? Perhaps the drivers
are such that they go direct to the controller?
I wonder if the boot drive is separate from the raid array - which might
explain the miniscule activity recorded.
I wonder if the BIOS setup for write through etc. are setup correctly?
Bill Vermillion
Dirk Hart <dh...@ultranet.com> wrote:
>matth...@my-deja.com wrote:
>> Why is this one server at 0%? I can't find any other kernel
>> parameter that I missed changing on that server that might cause
>> this. The only differences between this server and the others are
>> the increased RAM and the RAID 5 array (Compaq hardware RAID).
>Here's my own speculation:
>I wonder if the raid controller has its own memory? Perhaps the drivers
>are such that they go direct to the controller?
>I wonder if the boot drive is separate from the raid array - which
>might explain the miniscule activity recorded.
>I wonder if the BIOS setup for write through etc. are setup correctly?
ISTR that he did say the program ran in sync mode. That can severly
impact performance. OTOH if buffers are awfully large maybe it's
taking too long to flush them. I would think that sync on a RAID 5
could possible be a big performance hit. That's just speculation.
I wonder if the ap can be made run in non-sync mode. Running sync
is like running an MS application where everything waits.
--
Bill Vermillion - bv @ wjv . com
Matt Harrell
Yes, the boot drive is separate from the RAID 5 array. This RAID
controller should be very similar to the ones used on our other
servers, except for that it's setup as RAID 5. Thanks for the response.
--
Matt Harrell
Plexus Systems
mh...@plex-sys.com
Matt Harrell
I don't understand what you mean by "sync mode". I verified that the
DB is not set to bypass the OS buffers, so the system is using both the
Progress DB buffers and the OS buffers. This is the setting on all of
our UNIX DB servers, however, and the others don't have 0% read cache
hit rates.
--
Matt Harrell
Plexus Systems
mh...@plex-sys.com
Bill Vermillion
Matt Harrell <mh...@plex-sys.com> wrote:
>In article <G70t...@wjv.com>,
> b...@wjv.com wrote:
>> In article <3A5E23D5...@ultranet.com>,
>> Dirk Hart <dh...@ultranet.com> wrote:
>> >matth...@my-deja.com wrote:
>> >> Why is this one server at 0%? I can't find any other kernel
>> >> parameter that I missed changing on that server that might cause
>> >> this. The only differences between this server and the others are
>> >> the increased RAM and the RAID 5 array (Compaq hardware RAID).
>>
>> >Here's my own speculation:
>>
>> >I wonder if the raid controller has its own memory? Perhaps the
>drivers
>> >are such that they go direct to the controller?
>>
>> >I wonder if the boot drive is separate from the raid array - which
>> >might explain the miniscule activity recorded.
>>
>> >I wonder if the BIOS setup for write through etc. are setup
>correctly?
>>
>> ISTR that he did say the program ran in sync mode. That can severly
>> impact performance. OTOH if buffers are awfully large maybe it's
>> taking too long to flush them. I would think that sync on a RAID 5
>> could possible be a big performance hit. That's just speculation.
>>
>> I wonder if the ap can be made run in non-sync mode. Running sync
>> is like running an MS application where everything waits.
>I don't understand what you mean by "sync mode". I verified that the
>DB is not set to bypass the OS buffers, so the system is using both the
>Progress DB buffers and the OS buffers. This is the setting on all of
>our UNIX DB servers, however, and the others don't have 0% read cache
>hit rates.
I went back and re-read the first full article. I swore I read
that the database was in sync mode. It must have been some other
other post, and since that part of the article was not quoted, I
pulled it from my memory, which appears to be failing rapidly.
[Dave - Dave - my mind is going]
'sync' Synchronous file systems. Found in many MS operating
systems. The system is asked to write to disk and that operation
must finish before any other processing can continue.
'async' - Asynchronous file system. Unix style. You tell it to
write, OS waits a moment, and gives you back the prompt so you can
continue. The file has not been written at that point but is
scheduled to be written.
Disk controllers can have write-through, which goes directly to
disk, or write-back, where it is in cache and then will be written
as appropriate. This is the more efficient model, and the
controller can then take many schedule writes are perform then
in one long seek across the disk instead of hopping back and forth
as happens in a syncchronous mode.
I really thougt I had seen 'sync' earlier in the thread. Sorry
about that. Some DB's I have seen force all write to be
synchronous - for some design reason I assume - but most current
OSes are robust enough.
I'll bow out now.
Bill Vermillion
>In article <93n9cc$h0j$1...@nnrp1.deja.com>,
>Matt Harrell <mh...@plex-sys.com> wrote:
>>In article <G70t...@wjv.com>,
[mucho deletia - wjv]
>>I don't understand what you mean by "sync mode". I verified that the
>>DB is not set to bypass the OS buffers, so the system is using both the
>>Progress DB buffers and the OS buffers. This is the setting on all of
>>our UNIX DB servers, however, and the others don't have 0% read cache
>>hit rates.
>I went back and re-read the first full article. I swore I read
>that the database was in sync mode. It must have been some other
>other post, ....
I grep'ed the article storage. It was from a Mark who had i/o
probles on a database, but it was a cobol ap, which he said
was in sync mode. I guess I confused the Matt and Mark.
Jeff Poulin
5.0.4) for years and consistently had over 90% cache read hits. In my case,
I had 512MB RAM and 256MB of it was for buffers (and hash buffers was set to
131072). You don't have some background process that is constantly flushing
the cache do you? A background "find" would make a nice culprit. So would
a large DB file that is so fragmented, parts of it keep getting swapped in
and out of the cache.
Aside from sar, have you tested the cache any other way? Try this:
time cat some_big_file > /dev/null && time cat some_big_file > /dev/null
some_big_file should be big, but not bigger than your cache (15MB would be a
good test size). Assuming some_big_file is not already in your cache, the
first one should be slow, the 2nd fast. If the time on your 2nd run is
about the same as your first run, then your system is not caching. Let us
know what happens.
-- Jeff Poulin
"Matt Harrell" <mh...@plex-sys.com> wrote in message
news:93n96g$gub$1...@nnrp1.deja.com...
Jeff Poulin
"Jeff Poulin" <je...@nbhc.org> wrote in message
news:t5v6e83...@corp.supernews.com...
> The RAID 5 should not be a problem. I ran a RAID 5 on a DB server (SCO
> 5.0.4) for years and consistently had over 90% cache read hits. In my
case,
> I had 512MB RAM and 256MB of it was for buffers (and hash buffers was set
to
> 131072). You don't have some background process that is constantly
flushing
> the cache do you? A background "find" would make a nice culprit. So
would
> a large DB file that is so fragmented, parts of it keep getting swapped in
> and out of the cache.
>
> Aside from sar, have you tested the cache any other way? Try this:
>
> time cat some_big_file > /dev/null && time cat some_big_file > /dev/null
>
> some_big_file should be big, but not bigger than your cache (15MB would be
a
> good test size). Assuming some_big_file is not already in your cache, the
> first one should be slow, the 2nd fast. If the time on your 2nd run is
> about the same as your first run, then your system is not caching. Let us
> know what happens.
>
> -- Jeff Poulin
For comparison, on my system (SCO OS 5.0.4), it takes 1.93 seconds of real
time for the first cat (using a 15MB test file), and 0.00 seconds of real
time for the 2nd cat (again, I have 512MB RAM with half of it dedicated to
the buffer.)
Also, you might want to repeat the command after about 5 minutes to see if
your 15MB test file is still in the cache. Obviously, if both times are
close to 0.00, then it is. If one time is several seconds and the other is
0.00 then it isn't. You might want to try this throughout the day to see
about how long (on average) you can go before the test file gets swapped
out. If the system keeps the test file in the cache for a long time (say,
an 30 minutes or more between cat commands), then I would guess your
implementation of sar is not working correctly.
-- Jeff Poulin
Matt Harrell
Thanks for the help. I tried this with a 15 MB ASCII file. On the
first run, it took 1.37 seconds and on the second run it took 0.27
seconds. Faster, but the second time was not the 0 seconds you got,
either. Also, it only seems to take a few minutes (no more than 7 in
my last test) for it to be cleared out of cache.
Tony Lawrence
Then something is running around reading files large enough to
overwrite cache..
Find that something and that's the answer.