Excessive swapping & non-optimal qmemman heuristics

[Jean-Philippe Ouellet]

Hey all,

Am I the only one who seems to have noticed chrome reaching max mem &
swapping way more often than used to happen in the past? Some of my
workflows result in having a bunch of tabs (not even _that_ many,
maybe 20-30+) open in DispVMs. Unfortunately, this reaches the 4gb max
pretty quickly, and causes swapping, which results in the whole VM
becoming pretty unusable (can't even interact with it enough to close
tabs - often end up doing qrexec calls to pill the highest-mem chrome
renderer process - essentially me being a manual OOM killer).

I've also noticed some VMs getting more memory than it makes sense,
for example minimal VMs that just have a text editor, some notes, and
a few terminals open often end up with 1200mb+ pre-allocated when they
have no reason to need that much, which results in not having enough
unallocated mem to be able to start new domains unless you shutdown
VMs (that have accumulated large allocations while still having a
small user-meaningful working set).

Qubes has always been memory-hungry, and I've always considered it a
worthwhile tradeoff, but now I feel like I need to upgrade to a 32gb
machine just to run a couple browsers - this is getting ridiculous.

I'm going to look into this more systematically (first try tweaking
the qmemman algo, then maybe experiment with tmem), but I figured I'd
just rant here first in case anyone happens to have looked into this
already (beyond the usual "limit dom0 & sys-* maxmem" which seems to
always get parroted whenever this subject comes up).

Cheers,
Jean-Philippe

P.S.: Here's a script to tail qmemman's logs while translating from
xen domain IDs to VM names (because qmemman is purposefully simple for
reliability, avoiding taking a dependency on qubesd nor making extra
xc calls to get the name, so doesn't do this translation itself). Has
been helpful in seeing what's going on when things aren't behaving as
expected:
#!/bin/sh
journalctl -f -n 100 -u qubes-qmemman \
| sed -u -r 's/\.[0-9]+//g' \
| $(
qvm-ls --raw-data --fields xid,name \
| grep -v '^-' \
| sed -u \
-e 's/^/-e s\/([^0-9:])/' \
-e 's/|/([^0-9])\/\\\\1/' \
-e 's/$/\\\\2\/g/' \
| xargs echo sed -u -r
) \
| sed -u -r \
-e 's/([^[p])([0-9])[0-9]{3}([^0-9]|$)/\1\2k\3/g' \
-e 's/([0-9])[0-9]{3}k/\1m/g' \
-e 's/([0-9])([0-9])[0-9]{2}m/\1.\2g/g'