Unikernel and Memory balancing

[alain pierre]

Hi Qubes-devel,

Some qubes-mirage-fw users have recently reported that some AppVM are stuck at their lowmem limit ([1,2]). After some investigations, it might be related to qubes-mirage-fw being running and the system being under high memory pressure (a reproducible test was to boot a huge AppVM using 25+GB RAM, dom0 4GB, sys-{net,usb} using ~1GB and qubes-mirage-fw at 32MB on a 32GB laptop, a new AppVM is then stuck at 400MB).

A plausible explanation is that qubes-mirage-fw leads to a mess in the memory management process, and does not permit to properly reduce the huge AppVM memory in favor to the newly created one even if the unikernel is excluded from memory balancing with qubes-manager GUI (the situation seems to be different with the default sys-fw, so there's probably something to be done on the mirage side).

Does anyone have any hints on where to look at to move forward on that issue?

Best,

Pierre

[1]: https://forum.qubes-os.org/t/new-usability-issues-dom0-processes-making-system-unusable/18301/2

[2]: https://forum.qubes-os.org/t/memory-allocation-problem-remains-in-low-allocation-for-minutes/18787

[alain pierre]

I think I better understand the issue now.

Regarding the comment [1] I've tried to never report memory free to Xenstore which solves the issue (I understand the lines as the unikernel is removed from memory donnors and the memory balance continues with success).

Now I can see multiple ways to effectively fix the issue and I not sure what is the best way to proceed:

- never report free memory to Xenstore (the current patch, but probably not the clever)

- report maximum usage at startup (the unikernel will always use all the memory allowed)

- add some code in the unikernel to interract with Xenstore about memory (but again, the unikernel will always use all the memory allowed)

- evict in qmemman every domain that do not want to be included in the memory balancing (even if they claim to have free memory), I'm not sure how the checkbox in qubes-manager is used.

Does anyone have any advice on the best path to follow ?

Best,

Pierre

[1]: https://github.com/QubesOS/qubes-core-admin/blob/8e0de909c1defa5b979aac8f28449cd07cad1720/qubes/qmemman/__init__.py#LL228C2-L229C2