vmstat sr question - whether or not to add memory
Rob Patton
I've been reading through the book Sun Performance and Tuning, and
I'm hoping for some clarification on what to do when vmstat's scan
rate gets to 200-1000+ etc. On one page the authors say that if sr is
high then adding more memory is likely to help. Two pages later they
say that when paging starts the reaction is to add RAM but that this
usually does not help the problem. I think I understand their
explanation of why it doesn't help, but now I'm left where I was
before reading any of this - how do I tell from vmstat whether I need
more memory?
Thanks!
Scott Howard
> I've been reading through the book Sun Performance and Tuning, and
>I'm hoping for some clarification on what to do when vmstat's scan
>rate gets to 200-1000+ etc. On one page the authors say that if sr is
>high then adding more memory is likely to help. Two pages later they
>say that when paging starts the reaction is to add RAM but that this
>usually does not help the problem. I think I understand their
It sounds like you're confusing scan rate and paging, which are two
(relatively) unreleated things. Due to the way things work in
Solaris, paging is a normal fact of life, and can (generally) be
ignored. Scanning may or may not be a problem - more details below.
>explanation of why it doesn't help, but now I'm left where I was
>before reading any of this - how do I tell from vmstat whether I need
>more memory?
To a large extent, it depends on what version of Solaris you're
running, and whether you're using Priority Paging or not.
Solaris pre-8.
For Solaris < 8, you'll need to first work out if you have priority
paging turned on. To find out what priority paging is, why you should
use it, and how to turn it on, have a read of
http://www.sun.com/sun-on-net/performance/priority_paging.html
(Do it now - come back here when you're done!)
Now, if you dont have priority paging enabled, go and enable it.
Depending on your workload it will normally make some difference to
preformance, and often will make a huge difference.
There's only one problem with turning Priority Paging on - it makes it
a little more difficult to work out if you're running out of memory or
not. Without PP, the "scan rate" is a fairly good indication of
memory usage. If it's high, you're short on memory. If it's 0, you're
not. With PP, the meaning of scan rate changes, and it basically
becomes useless.
So how do you monitor memory shortages when PP is trned on? The same
way you do in Solaris 8.
Solaris 8
Solaris 8 has a whole new memory management system. It's not actually
Priority Paging (although it was devised by the same person), but it
also makes the "scan rate" measurement useless (in fact, it almost
always makes it 0).
So how do you monitor memory shortages? There's a few ways, and
unfortunately none of them are overly scientific.
The first is mentioned in the URL above - memstat - which is an
unsupported tool for pulling some swapping details out of the kernel.
The second will scare you, but it's actually very reliable. When you
run out of memory, you swap out to disk, so you can monitor how much
you're swapping by monitoring your swap partition(s) using iostat.
First, work out where swap is. ie :
# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t3d0s1 85,31 8 1025992 759664
Now, you can watch what this disk is doing using iostat :
# iostat -xnp 5 | egrep 'device|c0t3d0s1'
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 35.1 0 0 c0t3d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t3d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t3d0s1
(trimmed a bit to keep it clean - and remember to always ignore the
first line as it's an average-since-boot value)
So in this case, our swap disk is doing nothing, so we're not running
out of memory. If we start to run out of memory, we'll see some
action on the disk :
# iostat -xnp 5 | egrep 'device|c0t3d0s1'
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 35.1 0 0 c0t3d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t3d0s1
0.0 5.4 0.0 485.6 0.0 0.6 0.0 109.1 0 14 c0t3d0s1
0.0 31.9 0.0 3553.9 0.0 2.8 0.0 87.6 0 82 c0t3d0s1
0.8 28.8 3.2 3472.0 0.0 2.8 0.0 94.7 0 83 c0t3d0s1
2.5 26.4 9.9 3157.7 0.0 2.8 0.0 95.4 0 82 c0t3d0s1
0.0 19.9 0.0 2437.8 0.0 1.6 0.0 82.4 0 56 c0t3d0s1
1.6 0.0 6.4 0.0 0.0 0.0 0.0 14.4 0 2 c0t3d0s1
As you can see, we started doing some serious swapping out there
(around 3 MB/sec), and our swap slice was getting very busy (up to 83%
busy in fact).
Often the most difficult thing is knowing exactly where you draw the
line in the sand as to whats good performance and whats not. A swap
disk being 83% busy for a few seconds probably isn't a big issue, but
one running like this for hours on end would be. At the end of the
day most forms of performance tuning are half black magic, half
experience, and half luck. The best way to get better at it is to
understand how your system work, and what their baseline is. If a
machine normally doesn't swap at all (based on iostat as above), and
today the disk is 30% busy, then you're probably got something wrong.
And now, given that I have completely forgotten what the original
question was, I think I'll stop.
Ohh.. and you might want to go and have a read of "Solaris Memory
Sizing White Paper" by Richard McDougall - it's at
http://www.sun.com/sun-on-net/performance/vmsizing.pdf
Scott.
Greg Andrews
>Hi,
> I've been reading through the book Sun Performance and Tuning, and
>I'm hoping for some clarification on what to do when vmstat's scan
>rate gets to 200-1000+ etc. On one page the authors say that if sr is
>high then adding more memory is likely to help. Two pages later they
>say that when paging starts the reaction is to add RAM but that this
>usually does not help the problem.
>
Those two statements are talking about different fields in the
vmstat output.
The one they say can be helped by adding RAM is the sr field.
The one they say won't be helped by adding RAM are the re, mf,
pi, and po fields.
-Greg
--
Do NOT reply via e-mail.
Reply in the newsgroup.
Rob Patton
misunderstand the relationship between the scan rate and paging, I had
figured (without a loaded system to watch) that during heavy paging
the scan rate would go up.
Darren Dunham
"paging" (moving of data to/from RAM a page at a time) is associated not
only with low memory situations. page ins occur normally whenever
programs are run because it is efficient to do so.
The scan rate goes up whenever the available RAM is below LOTSFREE.
Unfortunately, prior to Solaris 8, you can be below LOTSFREE because of
lack of RAM for running programs, or just because you're doing I/O.
It's almost impossible to separate the two cases.
With Solaris 8, I/O uses a separately manage list of RAM, so that RAM
not used for programs shows up clearly.
--
Darren Dunham ddu...@taos.com
Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
Rob Patton
> With Solaris 8, I/O uses a separately manage list of RAM, so that RAM
> not used for programs shows up clearly.
Thanks for the info, and I have a question about the last part of it.
How do I make the distinction between the paging for programs vs. I/O?
If I look at vmstat -p, is this the difference between the executable
and the anonymous/filesystem paging fields? Thanks!