[Bug] dispvm sporadically hangs at 100% CPU usage

[Andrew B]

This may well be an upstream bug, but I'll report it here just in case.

I've seen this behavior a number of times before in both regular AppVMs and DispVMs, but I just grabbed the log now.  The bug typically, though not exclusively, manifests while browsing the web with Chrome.  It seems more 'activity' makes the presentation more likely.

[Marek Marczykowski]

Thanks for the report. It looks like some kernel bug in code responsible for
dynamic memory management (aka xen balloon driver). I remember some similar
bug reports on xen-devel mailing list - will find out if it is fixed somewhere.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Andrew B]

The last few times this happened, I noticed the VM was assigned around 4090-4098MB of RAM.  I seem to remember some report about processes being assigned more than that amount of memory on a system with 16GB of RAM. I think I followed the steps given as a fix, but perhaps the changes were wiped out when I upgraded qubes some time?

[Marek Marczykowski]

On 08.10.2012 00:15, Andrew B wrote:
> The last few times this happened, I noticed the VM was assigned around
> 4090-4098MB of RAM. I seem to remember some report about processes being
> assigned more than that amount of memory on a system with 16GB of RAM. I
> think I followed the steps given as a fix, but perhaps the changes were
> wiped out when I upgraded qubes some time?

I think it is separate issue, but I'm not sure. Did it happened also on 3.2.x
kernel?
The fix was only for DispVM. Currently I haven't any system with more than 8GB
RAM and haven't managed to reproduce this bug quickly.
You can limit memory for this AppVM and check if this bug still will be
triggered - just to check if this is related to previous bug.

[Andrew B]

Well, I've never tried any 3.2.x kernel on this machine, since I needed the Ivy Bridge-compatible kernel and Xorg.

I'm actually talking specifically about DispVMs here.

Right now the DispVM starts with 4098MB of memory and immediately runs up 100% CPU usage.  I checked 'qvm-prefs -l fedora-17-x64-dvm' and I saw, rather interestingly:

memory           : 400 

maxmem         : 4088

 

So, is the VM starting with an amount of memory it's not allowed to have? No, of course not... so I start a DispVM, 'qvm-prefs -l disp4' and:

memory           : 400 

maxmem         : 8053

 

Hmmm... so the DispVM template config file isn't used to create each actual disposable VM instance's config file?

There seems like a deeper bug here, but just having disposable VMs' associated template qvm-prefs respected would provide an easy workaround.  Any clues to how I should go about fixing this?

Andrew

[Marek Marczykowski]

On 08.10.2012 07:03, Andrew B wrote:
> Well, I've never tried any 3.2.x kernel on this machine, since I needed the
> Ivy Bridge-compatible kernel and Xorg.
>
> I'm actually talking specifically about DispVMs here.
> Right now the DispVM starts with 4098MB of memory and immediately runs up
> 100% CPU usage. I checked 'qvm-prefs -l fedora-17-x64-dvm' and I saw,
> rather interestingly:
>
> memory : 400
>
> maxmem : 4088

How many VMs do you have running now (not counting netvm and firewallvm, which
have static memory assignment)? Just to know if that 4098MB is caused by
DispVM needs, or just because you have plenty of free RAM.

Perhaps maxmem 4088 is still to big?

> So, is the VM starting with an amount of memory it's not allowed to have?
> No, of course not... so I start a DispVM, 'qvm-prefs -l disp4' and:
>
>>
>> memory : 400
>
> maxmem : 8053
>
>
> Hmmm... so the DispVM template config file isn't used to create each actual
> disposable VM instance's config file?

fedora-17-x64-dvm settings are applied to dispvm template only at
qvm-create-default-dvm call. Also DispVM haven't all settings saved in
qubes.xml (which is loaded by qvm-prefs).

Config acually used by DispVM you can found in /tmp/qubes-dvm-*.xl. If there
maxmem is different from 4088, this is real bug.

> There seems like a deeper bug here, but just having disposable VMs'
> associated template qvm-prefs respected would provide an easy workaround.
> Any clues to how I should go about fixing this?

[Andrew B]

Well, I 'tricked' it back to 3GB by very quickly 'xm set'ing the memory to 3GB, then shutting the DispVM down, launching a new one, shutting it down, etc.  That seems to have reduced initial memory to 3GB.

I just experienced a different one, though: 99% CPU usage with only 3082MB, while dom0 has 9+GB.  I've attached the kernel crash messages -- it crashed at the same location in p2m.c, so I'm guessing it's the same bug, and not necessarily related to assigned memory size.

Andrew