Qubes Dom0 init memory against Xen best practices?

[Victor L.]

Looking for a solution to the 4306MB mem limit for an AppVM in my 16GB PC, regardless of the max memory I set for it...

I stumbled upon this Xen project wiki regarding Dom0 best practices:
   http://wiki.xenproject.org/wiki/Xen_Project_Best_Practices#Why_should_I_dedicate_fixed_amount_of_memory_for_Xen_Project_dom0.3F

It stresses the importance of limiting Dom0's boot memory, and its previous section defines how to apply that boot time limit.
However I find no such limit in Qubes' /etc/default/grub.

Is this being bypassed for some other architectural reason? Thanks,

    Victor

[Marek Marczykowski-Górecki]

On 10.09.2014 13:19, Victor L. wrote:
> Looking for a solution to the 4306MB mem limit for an AppVM in my 16GB PC,
> regardless of the max memory I set for it...

Do you mean "xen:balloon: reserve_additional_memory: add_memory() failed: -17"
problem?

> I stumbled upon this Xen project wiki regarding Dom0 best practices:
>
> http://wiki.xenproject.org/wiki/Xen_Project_Best_Practices#Why_should_I_dedicate_fixed_amount_of_memory_for_Xen_Project_dom0.3F
>
> It stresses the importance of limiting Dom0's boot memory, and its previous
> section defines how to apply that boot time limit.
> However I find no such limit in Qubes' /etc/default/grub.
>
> Is this being bypassed for some other architectural reason? Thanks,

Qubes is different from typical Xen installation in (at least) one way: there
is graphical environment in dom0, which require noticeable amount of memory.
So it is hard to find a good _static_ memory size. But indeed, applying some
limit (2GB?) can save some memory wasted on kernel structures.
Also note that Qubes has own memory manager, which dynamically balance memory
across VMs based on their needs. Not a simple "autoballoon" Xen feature.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Victor L.]

On Tuesday, September 16, 2014 1:03:50 AM UTC+2, Marek Marczykowski-Górecki wrote:

On 10.09.2014 13:19, Victor L. wrote:
> Looking for a solution to the 4306MB mem limit for an AppVM in my 16GB PC,
> regardless of the max memory I set for it...

Do you mean "xen:balloon: reserve_additional_memory: add_memory() failed: -17"
problem?

Yes, where should I look for info on how to solve this? or whose bugzilla should I ping?
 

> I stumbled upon this Xen project wiki regarding Dom0 best practices:
>    
> http://wiki.xenproject.org/wiki/Xen_Project_Best_Practices#Why_should_I_dedicate_fixed_amount_of_memory_for_Xen_Project_dom0.3F
>
> It stresses the importance of limiting Dom0's boot memory, and its previous
> section defines how to apply that boot time limit.
> However I find no such limit in Qubes' /etc/default/grub.
>
> Is this being bypassed for some other architectural reason? Thanks,

Qubes is different from typical Xen installation in (at least) one way: there
is graphical environment in dom0, which require noticeable amount of memory.
So it is hard to find a good _static_ memory size. But indeed, applying some
limit (2GB?) can save some memory wasted on kernel structures.
Also note that Qubes has own memory manager, which dynamically balance memory
across VMs based on their needs. Not a simple "autoballoon" Xen feature.

Understood, thank you! :-)
 

[Marek Marczykowski-Górecki]

On 16.09.2014 11:20, Victor L. wrote:
>
>
> On Tuesday, September 16, 2014 1:03:50 AM UTC+2, Marek Marczykowski-Górecki
> wrote:
>>
>> On 10.09.2014 13:19, Victor L. wrote:
>>> Looking for a solution to the 4306MB mem limit for an AppVM in my 16GB
>> PC,
>>> regardless of the max memory I set for it...
>>
>> Do you mean "xen:balloon: reserve_additional_memory: add_memory() failed:
>> -17"
>> problem?
>>
>
> Yes, where should I look for info on how to solve this? or whose bugzilla
> should I ping?

I didn't investigated the issue before, it isn't so common to have more than
4GB RAM in a single VM... Anyway did a quick search and found this bugreport,
with a lot of information:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1304001
It is mostly about HVM, but people report there the same issue also for PV.

There is also linked patch, but it only mute the message.

Then I found that even with maxmem=8192, VM kernel reports only ~4.5GB available:
[user@testvm ~]$ dmesg|grep Memory
[ 0.000000] Memory: 269840K/4505212K available (5049K kernel code, 576K
rwdata, 2348K rodata, 952K init, 992K bss, 4235372K reserved)
It is also consistent with:
[ 0.000000] e820: last_pfn = 0x113000 max_arch_pfn = 0x400000000

So this looks like the problem - kernel do not notice memory available above
that limit. Even with "mem=8G" parameter. Initially I though this is a VM
kernel problem, but now I cannot rule out toolstack(xl) or hypervisor itself.

Xen-balloon driver parameters confirming stuck at 4400MB:
[user@testvm xen_memory0]$ pwd
/sys/devices/system/xen_memory/xen_memory0
[user@testvm xen_memory0]$ cat target_kb
5057596
[user@testvm xen_memory0]$ cat info/current_kb
4505600

Currently this is really low prio problem, surely I will not investigate it
further before final R2 release. But if you want to help, you can search
xen-devel ML for similar reports (that kernel message is good search term) and
posting above details there, or starting new thread there.

BTW Attached debug messages from xl create, you can get those by replacing
line 1721 of /usr/lib64/python2.7/site-package/qubes/modules/000QubesVm.py with:
xl_cmdline = ['sudo', '/usr/sbin/xl', '-v', 'create', self.conf_file,
'-d', '-p']
(added -v and replaced -q with -d)

[Marek Marczykowski-Górecki]

I have one another idea: perhaps initial 400MB it simply too small for
allocating kernel structures for full 8GB? Indeed it looks to be the case (or
at least somehow connected): when I change initial memory to 800, the VM
reports full 8130MB (VM maxmem) available:
[ 0.000000] Memory: 619084K/8332924K available (5049K kernel code, 576K
rwdata, 2348K rodata, 952K init, 992K bss, 7713840K reserved)

And indeed, now VM can use more memory, no more xen:balloon errors.

[Victor L.]

IT WORKS!

- I set initial memory to 2GB, and max memory to 14GB.
- Then I start the VM in Qubes VM Manager by right-clicing and start/resume AppVM (so no programs are taking up memory)
- The VM boots up with 2GB memory and then scales back to 697 MB (maybe the required memory for kernel structs. to allow 14 GB?)

- Then I start all the VM's processes I can and the memory scales up without "add_memory() failed" messages, no application breaks/stops and no swap is used (actually I opened enough apps to get the AppVM up to 11 GB, but I'm sure more is possible, it's just that I couldn't find more tough processes to run at the same time)

Maybe default initial memory should be a high value such as 1 GB since the memory manager scales back nicely after VM start-up (from 2GB to 697MB in my VM's case)? Or maybe the minimum allowed value in VM manager should be larger if the max memory is above 4GB?

In any case this was absolutely necessary for engineering apps that just eat up lots of memory. Thank you so much!!


PS: going back to the initial Dom0 init memory: this leads to believe that at least several hundreds of MB of memory might be lost in Dom0 since it won't ever need my 16GB of RAM, but the kernel structures are anyways made there at boot time. I'll try limiting it to 2 or 4 GB to be on the safe side and then report back.

Best regards,

      Victor

[Marek Marczykowski-Górecki]

The problem is you need that initial memory to start the VM. It would be hard
to have free 1GB RAM for example on 4GB system (when some VMs are already
running).

> Or maybe the minimum allowed value in VM manager should be
> larger if the max memory is above 4GB?

That would be a better idea. Can you propose some formula? I guess something
like maxmem<=memory*11, but it is based on only one point of data ;)

> In any case this was absolutely necessary for engineering apps that just
> eat up lots of memory. Thank you so much!!
>
>
> PS: going back to the initial Dom0 init memory: this leads to believe that
> at least several hundreds of MB of memory might be lost in Dom0 since it
> won't ever need my 16GB of RAM, but the kernel structures are anyways made
> there at boot time. I'll try limiting it to 2 or 4 GB to be on the safe
> side and then report back.

You can experiment using 'xl mem-set dom0 xxx' and check how much memory is
usable for dom0 then, depending on memory limit.

I guess 2GB limit should be enough, but it would be great if you want to check
some other values also.

[Victor L.]

> Or maybe the minimum allowed value in VM manager should be
> larger if the max memory is above 4GB?

That would be a better idea. Can you propose some formula? I guess something
like maxmem<=memory*11, but it is based on only one point of data ;)

Please find attached my proposed patch (to git: marmarek/qubesmanager) reducing the ratio to 10.

I tested using several processes (on a single testvm AppVM) started like this (each reserves 1GB for itself):
      stress -m 1 --vm-bytes 1g --vm-hang 100

("stress" tool installed from fedora repos)

Results:
init_mem ==> reachable max mem ==> ratio max/init
400                          4300                       10.75
700                          7554                       10.79
800                          8635                       10.79
1024                       11051                       10.79
1200                       12954                       10.79
1300                       14038                       10.79
1500                       14045                       my PC is 16GB, so this looks like my limit

Respecting this ratio means that the VM may reach the actual max mem above and additionally the VM's log won't show any xen:balloon: reserve_additional_memory: add_memory() failed: -17


However, it is interesting to note that, while conducting the tests and reaching >12GB with just 4 VMs running: testvm, netvm, firewallvm and dom0, the system gets so stressed that from that moment UP UNTIL REBOOT (even if you shutdown the testvm, reclaiming all its memory back) the Dom0 log shows, every some 30 seconds the message:

      xen:balloon: reserve_additional_memory: add_memory() failed: -17

NOTE: This happens on Dom0's log only, the message does not appear any more on AppVMs that respect above ratio.

> PS: going back to the initial Dom0 init memory: this leads to believe that
> at least several hundreds of MB of memory might be lost in Dom0 since it
> won't ever need my 16GB of RAM, but the kernel structures are anyways made
> there at boot time. I'll try limiting it to 2 or 4 GB to be on the safe
> side and then report back.

You can experiment using 'xl mem-set dom0 xxx' and check how much memory is
usable for dom0 then, depending on memory limit.

I guess 2GB limit should be enough, but it would be great if you want to check
some other values also.

From the above findings, a PC with 8GB RAM would waste in its Dom0 between 400 and 800 MB due to Dom0 not ever really using more than 4GB of RAM, so it may be really interesting to limit this. (on my 16GB PC I could be wasting 1 GB due to these structures)

The way I'm testing it is slower however:
1) In dom0 terminal: sudo su -
2) nano /etc/default/grub
3) change and save this: GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=max:4096M"
4) grub2-mkconfig -o /boot/grub2/grub.cfg
5) grub2-install /dev/sda
6) reboot

2GB seems fine from light testing, however I've set it to 4GB just to be on the safe side for the moment (gotta work a bit too :-)

Thanks for your help!

     Victor

[Victor L.]

Please find attached my proposed patch (to git: marmarek/qubesmanager) reducing the ratio to 10.

Sorry, attached the correct patch (disregard previous one)

[Victor L.]

On Tuesday, September 16, 2014 9:53:33 PM UTC+2, Victor L. wrote:

> Or maybe the minimum allowed value in VM manager should be
> larger if the max memory is above 4GB?

That would be a better idea. Can you propose some formula? I guess something
like maxmem<=memory*11, but it is based on only one point of data ;)

Please find attached my proposed patch (to git: marmarek/qubesmanager) reducing the ratio to 10.

BTW, I've noticed that my TemplateVM has, by default, an init memory of 400MB and a max mem of 8175MB.
As explained above this max is not reachable, so please find attached another patch, this one is for joanna/core-admin.git and adds the "max_mem <= 10*init_mem" check to QubesVm.py


NOTE: THIS PATCH IS FOR A DIFFERENT REPO THAN THE ONE BEFORE (Linux-VM-specific-force-minimum-init-mem-to-allow-sc_fix.patch)



Also please note that this discussion may be directly related to the following tickets:
   https://qubes-os.org/ticket/732
   https://qubes-os.org/ticket/733

There is some code at qvm-create-defatult-dvm that is forcing a 4GB RAM limit for DispVMs, as described in those tickets but unreferenced in their descriptions, which can be found here, just in case somebody wants to lift that limit:
    http://git.qubes-os.org/gitweb/?p=joanna/core-admin.git;a=blob;f=qvm-tools/qvm-create-default-dvm;h=a6e860a5f208803bd92a0fcafe713386074ca3fa;hb=HEAD#l40

Regards,

     Victor

[Marek Marczykowski-Górecki]

Thanks!

I've added those two patches to my queue, to be released as an update for R2.
We don't want to introduce any more change (not being a critical bugfix)
before final Qubes R2 to not risk regressions (this applies especially to the
second patch for core-admin). Also I tend to use 10.79, or 11 as a ratio in
core-admin to not artificially limit the VM (even if that mean sometimes error
messages in VM kernel log).

Indeed this can be also related to DispVM problem (which for now have working
workaround). And this is just another reason to not to mess with this code
just a few days before release.

[Victor L.]

However, it is interesting to note that, while conducting the tests and reaching >12GB with just 4 VMs running: testvm, netvm, firewallvm and dom0, the system gets so stressed that from that moment UP UNTIL REBOOT (even if you shutdown the testvm, reclaiming all its memory back) the Dom0 log shows, every some 30 seconds the message:
      xen:balloon: reserve_additional_memory: add_memory() failed: -17

This is related to my changing the max Dom0 memory to 4GB (through grub2's: GRUB_CMDLINE_XEN_DEFAULT="

console=none dom0_mem=max:4096M").

Without that change there are no more error messages either in Dom0 or the AppVMs

[Victor L.]

second patch for core-admin). Also I tend to use 10.79, or 11 as a ratio in
core-admin to not artificially limit the VM (even if that mean sometimes error
messages in VM kernel log).

I agree, though I'd use at most 10.75 since those error messages do also mean applications crash

 

Indeed this can be also related to DispVM problem (which for now have working
workaround). And this is just another reason to not to mess with this code
just a few days before release.

Days... cool!

[Marek Marczykowski-Górecki]

Indeed it can be - qmemman does support maxmem for VMs[1] (it do not not try
to extend VM beyond its maxmem), but does not support it for dom0...

[1] https://wiki.qubes-os.org/ticket/736