high memory usage on Linux hosts
askzaman askzaman
looks like some little amount of swapping also ( %swpused ) but free
is not reporting any swapping.
=====
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached
kbswpfree kbswpused %swpused kbswpcad
08:40:01 AM 224796 7944056 97.25 178364
5898568 8121928 432 0.01 432
08:50:01 AM 224724 7944128 97.25 178364
5898500 8121928 432 0.01 432
09:00:01 AM 224572 7944280 97.25 178364
5898500 8121928 432 0.01 432
Average: 227441 7941411 97.22 178339
5898181 8121928 432 0.01 432
------
-bash-3.00$ free -m
total used free shared buffers
cached
Mem: 7977 7752 224 0 174
5760
-/+ buffers/cache: 1818 6159
Swap: 7931 0 7931
=====
The host has apache , java running and one our custom application .
But these applications are just running and are not serving any
traffic .
====
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2% us, 0.1% sy, 0.0% ni, 99.2% id, 0.0% wa, 0.0% hi,
0.5% si
Mem: 8168852k total, 7942712k used, 226140k free, 178372k
buffers
Swap: 8122360k total, 432k used, 8121928k free, 5898628k
cached
PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S
COMMAND
19657 nobody 16 0 0 1:30.18 1.5 719m 122m 9.9m S
java_daemon
6545 nobody 15 0 1 198:18.07 1.0 527m 83m 5508 S
tcs_server
15732 nobody 16 0 0 2:23.94 0.2 31964 18m 5008 S
apache
15716 nobody 16 0 0 2:08.11 0.2 32016 18m 4952 S
apache
15733 nobody 16 0 0 2:09.58 0.2 31980 18m 4944 S
apache
15718 nobody 16 0 0 2:27.99 0.2 31976 18m 4936 S
apache
15735 nobody 16 0 0 2:28.53 0.2 31952 18m 4952 S
apache
15738 nobody 16 0 0 2:39.02 0.2 31952 18m 4928 S
apache
15727 nobody 16 0 0 2:16.02 0.2 31948 18m 4920 S
apache
15725 nobody 16 0 0 2:28.89 0.2 32024 18m 4928 S
apache
15724 nobody 16 0 0 2:25.61 0.2 31856 18m 4944 S
apache
7076 nobody 16 0 0 2:01.68 0.2 31400 18m 4912 S
apache
7091 nobody 16 0 0 2:03.89 0.2 31316 17m 4936 S
apache
====
The above output is in terms of high memory usage.
Can somebody explain why sar is reporting high memory usage on host .
The OS is RHEL 4 .
Any help will be highly appreciated.
Pascal Hambourg
askzaman askzaman a écrit :
> In one of Linux host , sar is showing always high memory usage .
> looks like some little amount of swapping also ( %swpused ) but free
> is not reporting any swapping.
What's wrong with this ? You wouln't want your system not to use all the
available memory, would you ? As expected, 'free' shows that most of the
available memory is used by pagecache (disk cache), to avoid disk access
and speed up file access. That memory can be reclaimed anytime if a
process needs it.
unruh
> In one of Linux host , sar is showing always high memory usage .
> looks like some little amount of swapping also ( %swpused ) but free
> is not reporting any swapping.
>
If there is memory to spare, linux uses it for caching. It retains old
programs in memory under the assupption they may be used again. Thus the
first time you open ooffice it takes longer than the second.
If that memory is needed by a running process, it is released from
cache.
Worry if it starts swapping.
Doug Freyburger
> askzaman askzaman <askq...@gmail.com> wrote:
>
>> In one of Linux host , sar is showing always high memory usage .
>> looks like some little amount of swapping also ( %swpused ) but free
>> is not reporting any swapping.
>
> If there is memory to spare, linux uses it for caching. It retains old
> programs in memory under the assupption they may be used again. Thus the
> first time you open ooffice it takes longer than the second.
> If that memory is needed by a running process, it is released from
> cache.
This is a topic of endless discussion between IT support folks and
clients who come from other operating systems. Using nearly all of the
physical memory is a design feature and it is intended behavior. On any
UNIX-like system if you don't see 90+% of your memory in use then
something is very wrong.
Binary images from the loader are cached in memory so a program can be
jumped to instead of loaded again from disk. Also file system blocks
stay in cache in case they get read again. Directories in particular
benefit from staying in memory.
> Worry if it starts swapping.
There are two separate concepts when it comes to swap.
One is page faults to disk, listed as swaps in Linux. They indicate
that demand for memory is too high and the system needed to go virtual.
Disk is several orders of magnitude slower than memory so going virtual
means slowing down. On the other hand if you tune your system so it
never sees a page fault to disk then you might be under utilizing both
memory and disk. In the end you might actually want a few page faults
to disk per hour on a system running a very large application. I tend
to tune Oracle or other databases to stay memory resident all of the
time and often the result is additional applications page fault to disk
every now and then.
The other is swap reservations by images and allocations. When a
program starts it reserves enough space in swap to copy its instructions
but that image is not always copied to disk. It isn't even usually
copied to disk. When a program allocates a large block of memory it
reserves enough space in swap to handle it but that memory is not always
copied to disk.
Together it's common to end up with a system that shows a large
percentage of its swap space "in use" and yet almost never see a page
fault to disk aka swap.
unruh
> unruh wrote:
>> askzaman askzaman <askq...@gmail.com> wrote:
>>
>>> In one of Linux host , sar is showing always high memory usage .
>>> looks like some little amount of swapping also ( %swpused ) but free
>>> is not reporting any swapping.
>>
>> If there is memory to spare, linux uses it for caching. It retains old
>> programs in memory under the assupption they may be used again. Thus the
>> first time you open ooffice it takes longer than the second.
>> If that memory is needed by a running process, it is released from
>> cache.
>
> One is page faults to disk, listed as swaps in Linux. They indicate
> that demand for memory is too high and the system needed to go virtual.
> Disk is several orders of magnitude slower than memory so going virtual
> means slowing down. On the other hand if you tune your system so it
> never sees a page fault to disk then you might be under utilizing both
> memory and disk. In the end you might actually want a few page faults
> to disk per hour on a system running a very large application. I tend
> to tune Oracle or other databases to stay memory resident all of the
> time and often the result is additional applications page fault to disk
> every now and then.
Actually I would tend to disagree, because usually if you get one page
swap, you get lots. And once a program starts swapping, you might as
well kill it cause it aint going anywhere. Memeory is cheap, and the
cost of not having what you call underutilisation of memory can be so
severe ( a long job essentially dying) that the the cost of extra memory
is trivial If memory still cost $10000/MB you would have a point. At
pennies per MB it is silly to have page swaps, especially since the
extra memory is put to good use as you and I point out in the erased
part of this message.
...
The Natural Philosopher
swapped out programs that are rarely used...inetd?
I agree that having swapping occur in an active well utilised daemon is
a total disaster.
Jean-David Beyer
> is trivial If memory still cost $10000/MB you would have a point.
Young whippersnappers.
I remember in the early 1950s when memory was $40/bit! It took 10 of
those to store a digit (in ENIAC).
Then they invented Williams Tube memories that were cheaper.
http://inventors.about.com/library/weekly/aa060998.htm
Magnetic core memory came next. (I skip delay line memories and rotating
magnetic drum memories as main memory.) Core could be about $1/byte.
That is when semiconductor memory came out. The last 4 memory modules I
bought were in the order of $50/Gigabyte.
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 21:15:01 up 10 days, 5:20, 3 users, load average: 4.69, 4.59, 4.57
Doug Freyburger
> unruh wrote:
>> Doug Freyburger <dfre...@yahoo.com> wrote:
>
>>> One is page faults to disk, listed as swaps in Linux. They indicate
>>> that demand for memory is too high and the system needed to go virtual.
>>> Disk is several orders of magnitude slower than memory so going virtual
>>> means slowing down. On the other hand if you tune your system so it
>>> never sees a page fault to disk then you might be under utilizing both
>>> memory and disk. In the end you might actually want a few page faults
>>> to disk per hour on a system running a very large application. I tend
>>> to tune Oracle or other databases to stay memory resident all of the
>>> time and often the result is additional applications page fault to disk
>>> every now and then.
>>
>> Actually I would tend to disagree, because usually if you get one page
>> swap, you get lots. And once a program starts swapping, you might as
>> well kill it cause it aint going anywhere. Memeory is cheap, and the
>> cost of not having what you call underutilisation of memory can be so
>> severe ( a long job essentially dying) that the the cost of extra memory
>> is trivial If memory still cost $10000/MB you would have a point. At
>> pennies per MB it is silly to have page swaps, especially since the
>> extra memory is put to good use as you and I point out in the erased
>> part of this message.
>
> swapped out programs that are rarely used...inetd?
That's the deal. It should be a goal to push your hardware towards an
optimal. Pushing it until it does hundreds or thousands of page swaps
per second will kill performance. Running it with low memory
utilization means having hardware you don't use - Great for at the home
but not the goal for a small business. An ad hoc compromise in their
somewhere is for crontab jobs to page a little every so often. It's
like trying to get a race car to spend as much of the race as possible
at yellow line to stay ahead of the pack and have a chance of winning
the race.
That and it's a techno-game to squeeze out that little bit of extra
processing even if you need to add seti-at-home or similar to have
extra load. ;^)
unruh
False. Linux always uses memory. It uses it for caching. Memory is so
cheap now that even in a small business it's cost is completely
negligible compared to the people cost. If the extra memory saves a few
minutes per job it has paid for itself. This is penny wise and pound
foolish as they say.
> somewhere is for crontab jobs to page a little every so often. It's
> like trying to get a race car to spend as much of the race as possible
> at yellow line to stay ahead of the pack and have a chance of winning
> the race.
No it is not like that at all. It's like you want to buy retreads for your racing
car because usually they will not delaminate and they will save you a
few bucks. Of course occasionally they will blow up and crash the car,
but that's the cost of living on the edge.
>
> That and it's a techno-game to squeeze out that little bit of extra
> processing even if you need to add seti-at-home or similar to have
> extra load. ;^)
????? The similies are wandering off into the heavens.
The Natural Philosopher
> Memory is so
> cheap now that even in a small business it's cost is completely
> negligible compared to the people cost. If the extra memory saves a few
> minutes per job it has paid for itself. This is penny wise and pound
> foolish as they say.
well yes, IF the chassis will take the extra RAM...
I agree however, that not stuffing the motherbaord with all the RAM it
can take, is a bit pointless if you have any significant swapping at all.
David Brown
Memory is not necessarily cheap - as you get beyond the current standard
for your dimms (2GB at the moment, IIRC), it costs a lot more. And if
you have to get a new motherboard with more space, it can be very
expensive. Also adding memory means a hardware upgrade - adding swap
space is just a couple of commands that can be executed remotely, and is
basically free.
It's always a matter of balance. Maybe adding more memory will speed up
a couple of jobs - but does that matter? How does that translate into
dollars earned or saved? If the job in question is something people
wait for, then it makes a big difference. But if the job that takes
extra memory is a table re-index run at off-peak hours, then it doesn't
matter if swap is used.
And while it is true that extra memory can always be used for caching,
that does not necessarily mean more speed. There comes a point when you
have already cached most of the useful disk data - more memory then has
negligible effect.
There is another use of memory and swap space that hasn't been mentioned
here - tmpfs mounts. I set up my desktops with a large amount of swap
space, and regularly use tmpfs mounts for temporary files. For some
kinds of work, such as large compilations, tmpfs is much faster than
files on disk, even if it overflows the memory and uses swap (either for
the tmpfs files or for current programs).
Doug Freyburger
>
> Memory is not necessarily cheap - as you get beyond the current standard
> for your dimms (2GB at the moment, IIRC), it costs a lot more. And if
> you have to get a new motherboard with more space, it can be very
> expensive. Also adding memory means a hardware upgrade - adding swap
> space is just a couple of commands that can be executed remotely, and is
> basically free.
There are times I've gotten more performance benefit from adding more
swap than from adding more memory. It's not as common a result as
adding more memory but it does happen.
Lately I've encountered systems built by someone else that had 48 GB of
RAM but only 2-4 GB of swap. If it has enough RAM it won't need swap,
right? Wrong. It needs swap because of the reservation issue even if
it doesn't do any page swaps. I've added 20 GB of swap space to such
systems and gotten 50+% performance improvements. The initial
installer being long gone there was no way to turn it into a teaching
event. Folks on my team call me to discuss sizing when they do builds.
> And while it is true that extra memory can always be used for caching,
> that does not necessarily mean more speed. There comes a point when you
> have already cached most of the useful disk data - more memory then has
> negligible effect.
In the case of running Oracle, caching filesystem blocks to disk hurts
performance. It's worth extra effort to activate any directio or
quickio async option possible on any mount point with DBF files. The
database performance improvements are often high enough it's worth doing
for every mount point on that host. This makes it a very good idea to
separate out the database layer of any pplication to a different
machine than the other layers so those other layers can have cached
filesystem access.
Also for Oracle, adding more memory gives declining returns. Above some
point it no longer helps performance. Equally not true for the other
layer types of a multilayer application.
> There is another use of memory and swap space that hasn't been mentioned
> here - tmpfs mounts. I set up my desktops with a large amount of swap
> space, and regularly use tmpfs mounts for temporary files. For some
> kinds of work, such as large compilations, tmpfs is much faster than
> files on disk, even if it overflows the memory and uses swap (either for
> the tmpfs files or for current programs).
I do tmpfs on my Solaris hosts because it comes that way out of the box.
There are a few application that make heavy use of temp files but most
of the applications I use aren't among them. I happen to see so little
gain from going to tmpfs on my Linux boxes it has not been worth it.
Hand me some boxes with one of those applications that does make heavy
use of temp files and my song will change -
"When you ask a SysAdmin a performance question what will be his
answer?" "It depends."