swappiness, caches

[johny...@sigaint.org]

It always seemed a bit "off" to me that there should be any swap usage or
significant buffers/caches inside VM's.

dom0 already caches the virtual .img files, so having the kernel inside
each VM also buffering/caching files and metadata is really just a waste
of CPU and disk space.

More importantly, swap activity is a major performance killer on bare
hardware; even more so on a virtual machine. There's a case to let some
stuff swap out (unused x-servers in sys-net/sys-firewall, etc.) but the
default is way too aggressive for good performance, IMHO.

The kernel has a "vm.swappiness" (0 to 100) parameter that controls the
balance between grabbing a new free page from the cache or swapping a page
out to re-use. The default is 60, which leans towards swapping instead of
cache use. Not ideal.

In dom0 and all my VM's, I tried changing the swappiness factor to see
what the effect would be:

# echo 0 >/proc/sys/vm/swappiness
or
$ sudo sysctl -w vm.swappiness=0

To empty existing swap space, I did a "swapoff -av && swapon -av"

Performance is noticeably better, with no swapping happening in any of the
VM's, nor in dom0.

And memory usage reported by the Vm's seems to be smaller; a heavy
browsing session used to crank a VM up to 1.5G; now it seems to be more
like 800M. So I can run more VM's, get more done.

(I'm not sure why this is, but firefox seems to be less of a memory pig
with this change. Maybe with the former aggressive swapping out, Firefox
thought free memory was a bit cheaper than it is, and was more aggressive
in its own use, or more lax in freeing things up. With a more realistic
picture of the memory situation, by changing vm.swappiness to 0, it
doesn't seem to be quite the pig it was.)

You can set the parameter permanently by adding "vm.swappiness=0" to
/etc/sysctl.conf in dom0 and your templates.

Poking around 'net, I see a few comments where 0 swappiness is best for
guest VM's. I wonder if a little higher might not be bad, for the case of
an unused X server or whatever, being able to swap out. I might play a
bit with different settings in different VM's.

It would be nice to disable caching in the VM's, but that seems to be a
difficult thing to do in Linux.

So far I've found that the system is pretty good about keeping the VM size
to the minimum/startup size, and giving up buffers/cache when needed.

(buffers/cache aren't included in the 'memory-used' calculation when
mem-balancing between VM's, which keeps the buffers/cache under control a
bit, I think. Excessive cache use is not given weight and rewarded by
additional memory in the balancing. Any real memory needs from an
existing or new VM would effectively take priority over buffer space, in
the memory balancing allocations.)

Real quick and dirty way of checking your swap usage across VM's (I might
add this info to the VM manager for fun, since it's pretty critical
performance information; will submit any changes):

$ sudo xenstore-ls -f /local/domain | awk '/meminfo/ { print $12-$14, $1 }'

The # reported in the path is the domid, which you can see with "qvm-ls -i"

I'd be interested in hearing others' thoughts on this.

Related reading:

https://www.xenproject.org/directory/directory/research/118-geiger-monitoring-the-buffer-cache-in-a-virtual-machine-envrionement.html

http://www.sersc.org/journals/IJCA/vol6_no1/12.pdf

Cheers

JJ

[David Hobach]

Interesting, sounds reasonable.

Running with absolutely 0 swap however can lead to unexpected problems
from my experience:
Some applications (e.g. firefox downloads the last time I tested it) use
memory-mapped files of arbitrary size assuming swapping is enabled, i.e.
you'll store the entire files in memory without swapping. If however the
files are getting too large (say you download a multi Gig file via
firefox), your memory will run out and the application will report an
error (firefox will cancel the download due to a write error).

Most applications work though and reducing the swap size to an absolute
minimum certainly helps to improve speed.

[johny...@sigaint.org]

> Interesting, sounds reasonable.
>
> Running with absolutely 0 swap however can lead to unexpected problems
> from my experience:

Interesting that the Wiki page for swappiness (this kernel parameter is
officially more famous and I am) recommends setting it to at least 1.

https://en.wikipedia.org/wiki/Swappiness

More on the vm.swapiness parameter. It's a bit more subtle than I thought.

Some references:

1) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

This page describes it as being how prone Linux is to start swapping out
processes. At 0, nothing will be swapped until memory is fully exhausted;
at 100, swapping out of processes will begin almost immediately. It
indicates the default setting of 60 means swapping will start when memory
is around "half" full.

So a setting of zero doesn't prevent swapping, it just puts it off until
there is no memory available. This is the old-school Unix behaviour I'm
used to, and probably best for VM.

2)
http://unix.stackexchange.com/questions/88693/why-is-swappiness-set-to-60-by-default

This page talks about it in relation to choosing pages with backing store
(programs, libraries, cached/mem-mapped data files, already-swapped data)
vs. "anonymous" pages of allocated memory. Cached files have a weight of
200-vm.swappiness, and anonymous pages a weight of vm.swapiness.

That may be saying the same thing as #1 but in a different; possibly more
prescise way, since a setting of 100 gives a 50/50 weight for new page
acquisition betreen swap and cache.

3) http://www.cloudibee.com/linux-performance-tuning-vm-swappiness/

This one talks about it as a balance between "swapping and freeing cache,"
which is the same, I think.

-----

Any anonymous page is going to need to be written to swap before being
given to the VM needing memory. (As well as a read when the page is used
in the future.) And writes are usually more expensive than reads to start
with.

A cached file/program/library doesn't need to be written, the page can be
discarded and re-used immediately since it can be retrieved from the
backing file/program/library when needed in the future.

Having a swap/anon page swapped and retrieved has a cost of 1w+1r

Having a file/prog page discarded and later retrieved has a cost of 1r.

So swapping has a r/w cost of at least 2x that of stealing from the
file-backed cache. (Writes are usually a bit more costly than reads, as
well.) Obviously the nature of your machine (server/desktop) affects
things.

That 60 default setting means file-backed cached pages have a weight of
200-60, or 140, while the anonymous/to-be-swapped pages have a weight of
60, a 70%/30% balance in favour of resuing file-backed cached pages versus
swapping something out to get free pages.

Not a bad compromise for running on bare hardware, or a server; but not
appropriate/necessary for a VM.

With vm.swappiness set to 0 and the same swap space as before, swap can
get used when needed; and as much as before, but not until memory is
exhausted.

And when the free memory is exhausted, that also implies that all of the
cache has also been re-allocated as assigned memory as well. Since the
VM's really shouldn't be caching in the first place (double-caching in
both dom0 and the VM has to be slower than just one level of cache).

I'm still looking around for options to disable file caching, but having
vm.swappiness low at least gives any running program priority over the
memory being used as cache.

The qmemman load balancer won't consider the memory used for a VM's cache
as part of it's "required" memory (but it does include swapped out stuff,
giving it reasonable chance of getting back into memory without
thrashing), so with low vm.swapiness a VM will not be given extra memory
to maintain any significant level of cache, unless there's free memory
around to be doled out between VM's overall.

I can't help but think the original intent was for vm.swappiness=0 behaviour.

Once vm.swappiness is >0, then some level swapping will occur resulting in
free pages for the VM, and Linux will then go and use these pages for
additional (and unnecessary) cache space; an all-around waste of
disk-access, CPU time, memory.

Running with vm.swappiness=0 seems to work in practice so far. I'm still
amazed at the difference in memory/performance I'm seeing.

Zero swap on a system that used to have 40M-ish swapped on all VM's and in
dom0. And smaller VM's, allowing more to be started. I'm surprised that
this hasn't been a default, or at least some similar tuning done by
default.

In the source code, the 350M "dom0 memory boost" is mentioned as being
specifically to give dom0 free memory (that will inherently be used as
cache) beyond its actual needs (used memory+used swap).

So there is intent to let dom0 do the file caching. But not a similar
effort to prevent unnecessary caching in the VM's.

Also, it's worth verifying the benefitting from a low vm.swappiness in
dom0 itself. Swapping can just kill performance so badly.

(I was going to post a report that the "dom0 memory boost" value in the QM
Global Settings seems to be ignored in the source code and hard-coded to
350M. But as I was looking at the code, an update fininshed, the file
reloaded, and the fix was in place. Instant telepathic bug fix updates.
Impressive, ITL. :) )

JJ

[johny...@sigaint.org]

> Interesting that the Wiki page for swappiness (this kernel parameter is

> officially more famous than I am) recommends setting it to at least 1.
>
> https://en.wikipedia.org/wiki/Swappiness

I'm going to stick with vm.swappiness=0 for a few days just to see if any
reliability problems or app failures do pop up, out of curiosity.

I think a setting of 1 (or at least <10) is probably best for the long
run, letting some very infrequently used stuff creep off to swap.

[Chris Laprise]

On 10/19/2016 11:54 AM, johny...@sigaint.org wrote:
> It always seemed a bit "off" to me that there should be any swap usage or
> significant buffers/caches inside VM's.
>
> dom0 already caches the virtual .img files, so having the kernel inside
> each VM also buffering/caching files and metadata is really just a waste
> of CPU and disk space.

I don't quite agree.

Disk cache is a dilemma, actually. That is because there are different
advantages to having the cache in either dom0 or domUs. For example,
booting a whole system where you have between 2 and 4 VMs starting in
rapid succession goes much quicker if dom0 is caching the VM images.

OTOH, having VMs cache more of their own data means less buffer-copying
between dom0 and domUs while running applications.

> More importantly, swap activity is a major performance killer on bare
> hardware; even more so on a virtual machine. There's a case to let some
> stuff swap out (unused x-servers in sys-net/sys-firewall, etc.) but the
> default is way too aggressive for good performance, IMHO.
>
> The kernel has a "vm.swappiness" (0 to 100) parameter that controls the
> balance between grabbing a new free page from the cache or swapping a page
> out to re-use. The default is 60, which leans towards swapping instead of
> cache use. Not ideal.
>
> In dom0 and all my VM's, I tried changing the swappiness factor to see
> what the effect would be:
>
> # echo 0 >/proc/sys/vm/swappiness
> or
> $ sudo sysctl -w vm.swappiness=0
>
> To empty existing swap space, I did a "swapoff -av && swapon -av"
>
> Performance is noticeably better, with no swapping happening in any of the
> VM's, nor in dom0.
>
> And memory usage reported by the Vm's seems to be smaller; a heavy
> browsing session used to crank a VM up to 1.5G; now it seems to be more
> like 800M. So I can run more VM's, get more done.

I have experimented with this myself. How well it works depends on how
much physical RAM you have.

I suspect the effects of dom0 swapping are being felt more in R3.2
because dom0 now has a size limit. In 3.1 and prior, swapping seemed
less common.

There is also vm.vfs_cache_pressure which affects cache size.

Chris