Memory balancing very inefficient

[r.wie...@web.de]

Today I noticed that many VMs do get a lot more RAM than they actually
use. While using only about 200-300MB small vms like -net and -firewall
get gigabytes of memory and this seem to be the case even if memory is
running out (sum of all VMs approaches physical RAM size). Also dom0 is
using only about 700MB but gets 4GB.

1) does memory balancing take back memory from a VM at all?
2) how does it happen that VMS get assigned this ridiculously larger
amount of memory compare to their usage?
3) is there something that can be done besides manually setting limits
for all VMs?

I current think about limiting all small VMs to 256MB and dom0 to 2 GB
of RAM (by GRUB parameter) lacking any idea for a better approch.

[haaber]

> Today I noticed that many VMs do get a lot more RAM than they actually
> use. While using only about 200-300MB small vms like -net and -firewall
> get gigabytes of memory and this seem to be the case even if memory is
> running out (sum of all VMs approaches physical RAM size). Also dom0 is
> using only about 700MB but gets 4GB.
>
> 1) does memory balancing take back memory from a VM at all?

apparently, as much as there is enough, each appVM gets MaxMem-size
(kernel param, usually 4G). When memory is gettng tight the qmemmman
manager does "balooning" whatever that is exactly.
This behaviour might be linked to errors (e.g. my qubes install does not
support 5.x xen kernels: crashes can be caused by "memory stress" and
even if not, they always finish by loads of qmemman log entries, before
deep freeze (not even a kernel panic, just sudden death)

> 2) how does it happen that VMS get assigned this ridiculously larger
> amount of memory compare to their usage?

by design, as explained.

> 3) is there something that can be done besides manually setting limits
> for all VMs?

Good question.

> I current think about limiting all small VMs to 256MB and dom0 to 2 GB
> of RAM (by GRUB parameter) lacking any idea for a better approch.

Tell us if that works! My qubes has no grub. But you can set kernel
params in /boot/efi/EFI/qubes/xen.cfg

[r.wie...@web.de]

Am 22.02.21 um 20:40 schrieb haaber:

> This behaviour might be linked to errors (e.g. my qubes install does not
> support 5.x xen kernels: crashes can be caused by "memory stress" and
> even if not, they always finish by loads of qmemman log entries, before
> deep freeze (not even a kernel panic, just sudden death)

What does "your qubes install" mean? Mine has been auto-updated to
kernel 5.4.88-1

>> I current think about limiting all small VMs to 256MB and dom0 to 2 GB
>> of RAM (by GRUB parameter) lacking any idea for a better approch.
>
> Tell us if that works! My qubes has no grub. But you can set kernel
> params in /boot/efi/EFI/qubes/xen.cfg

Again: What is special about "my qubes" ?

I modified /boot/grub2/grub.cfg (changing all dom0_mem=max:4096M values)
and this works as expected. After a few hours I could not figure out
limitations having the limits in place.

[Bernhard]

>> This behaviour might be linked to errors (e.g. my qubes install does not
>> support 5.x xen kernels: crashes can be caused by "memory stress" and
>> even if not, they always finish by loads of qmemman log entries, before
>> deep freeze (not even a kernel panic, just sudden death)
> What does "your qubes install" mean? Mine has been auto-updated to
> kernel 5.4.88-1

mine too. But since this kernel crashes after 2-30 minutes I downgraded
(xen!) kernel back to 4.19.163. That works, at least. My App-VM's run
smoothly on 5.x kernels, even kernel-latest does fine.

>>> I current think about limiting all small VMs to 256MB and dom0 to 2 GB
>>> of RAM (by GRUB parameter) lacking any idea for a better approch.
>>
>> Tell us if that works! My qubes has no grub. But you can set kernel
>> params in /boot/efi/EFI/qubes/xen.cfg
>
> Again: What is special about "my qubes" ?

that is a poor try on non-violent communication. I have no grub
installed: qubes (4.0) came out of the box like that (using UEFI, as did
qubes 3.x before). I am surprised that qubes uses grub in other settings
:))

> I modified /boot/grub2/grub.cfg (changing all dom0_mem=max:4096M values)
> and this works as expected. After a few hours I could not figure out
> limitations having the limits in place.

very good news, thank you. Bernhard

[Ulrich Windl]

grep -i qubes /boot/grub2/grub.cfg

On 2/23/21 11:36 AM, Bernhard wrote:
>>> This behaviour might be linked to errors (e.g. my qubes install does not
>>> support 5.x xen kernels: crashes can be caused by "memory stress" and
>>> even if not, they always finish by loads of qmemman log entries, before
>>> deep freeze (not even a kernel panic, just sudden death)
>> What does "your qubes install" mean? Mine has been auto-updated to
>> kernel 5.4.88-1
>
> mine too. But since this kernel crashes after 2-30 minutes I downgraded
> (xen!) kernel back to 4.19.163. That works, at least. My App-VM's run
> smoothly on 5.x kernels, even kernel-latest does fine.
>
>>>> I current think about limiting all small VMs to 256MB and dom0 to 2 GB
>>>> of RAM (by GRUB parameter) lacking any idea for a better approch.
>>>
>>> Tell us if that works! My qubes has no grub. But you can set kernel
>>> params in /boot/efi/EFI/qubes/xen.cfg
>>
>> Again: What is special about "my qubes" ?
> that is a poor try on non-violent communication. I have no grub
> installed: qubes (4.0) came out of the box like that (using UEFI, as did
> qubes 3.x before). I am surprised that qubes uses grub in other settings
> :))

??? Qubes??? Qubes is using GRUB, and UEFI is part of the BIOS."
Try "grep -i qubes /boot/grub2/grub.cfg".

[r.wie...@web.de]

Coming back to this topic again. I am using Qubes 4.1 rc1 now and have
made the same observations again. Start of new VM/Qube is aborted due to
low RAM even though some of the running qubes have >>500MB free. So
again I start to manually limit all VMs to RAM sizes that seem
reasonable to me. Memory balancing really seems to fail hard, even in
the current version.