Xen high CPU usage, but nothing is running in the VM

[lok...@gmail.com]

I noticed that my laptop fan was blowing more than usual, so I started ‘xentop’ in dom0 to see which VM was using CPU. I noticed two VM's were using 150% and abour 70% respectively (according to ‘xentop’ output).

However, when running ‘top’ in the VM's themselves, I saw nothing using any CPU. That is what I would expect, as they were not running anything CPU-intensive at the time.

Has anyone seen this before, and what can I do to diagnose this problem when/if it happens again?

[Vít Šesták]

What CPU usage does Qubes Manager show? I guess is shows low CPU usage.

Do you see any other symptoms of high CPU usage like heat or fan activity?

I guess the Xen just allocates some CPU time for some VMs, but the time is not used. As a result, xentop seems to overestimate actual CPU usage.

Regards,
Vít Šesták 'v6ak'

[lok...@gmail.com]

I've restarted the offending VM's now, so I can't test anymore.

The fan was blowing at maximum speed (which is why I looked into this in the first place) so there was definitely something happening.

The VM that was running with the most CPU usage according to xentop (150% or so) had been used to run an Atari ST emulator (which uses lots of CPU). However, the emulator had been killed and the machine left idle while I was off doing other things for at least 30 minutes. It was when I came back to the computer that I noticed that the fan was blowing and that's when I noticed the problem.

[Chris Laprise]

On 06/17/2017 06:40 AM, lok...@gmail.com wrote:
> I've restarted the offending VM's now, so I can't test anymore.
>
> The fan was blowing at maximum speed (which is why I looked into this in the first place) so there was definitely something happening.
>
> The VM that was running with the most CPU usage according to xentop (150% or so) had been used to run an Atari ST emulator (which uses lots of CPU). However, the emulator had been killed and the machine left idle while I was off doing other things for at least 30 minutes. It was when I came back to the computer that I noticed that the fan was blowing and that's when I noticed the problem.
>

This happens to me sometimes on the current Xen/Linux versions. When I
look at top in the offending VM its 'kswapd' that has gone berserk.

--

Chris Laprise, tas...@openmailbox.org
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Vít Šesták]

Interesting, I'd expect kswapd to be capable of performing I/O berserk, nou CPU berserk. The only CPU-intensive part should be dm-crypt, but it runs in dom0, not in standard AppVMs (unless you adjust it accordingly).

Regards,
Vít Šesták 'v6ak'

[cooloutac]

ya man I wouldn't just restart them I'd delete them and recreate them.

I don't think I notice this happening on my machine. I have noticed that xentop shows cpu more accurately though then qubes manager in the past. it will show cpu usage qubes manager doesn't show. But I would still see something happening qubes manager at least. And I don't notice anything weird on idle vms.

But I always shut down more trusted ones that have net access just in case they get attacked form other vms.

I have noticed, when first started using qubes, that sometimes when an appvm is open it will check for updates. Or that updates won't be checked until you open up an appvm that use that template, but I don't know if thats changed since earlier Qubes versions. I never dug into what process, just correlated the cpu activity with the network activity that would go at the same time on sys-net.

Its too hard for me to monitor so many vms on a polylithic system like Qubes. So at first sign of anomaly I just delete the vm so much easier. I mean unless I was experimenting I really don't give a crap whats causing something at this point in my life. Possible malicious? ok wipe it. Qubes makes it easy. If it keeps happening then you need to find out whats going on.

[Vít Šesták]

BTW, I remember having such issue (including real CPU load), but that time, avahi-daemon was to blame, related to VPN. But it was shown in htop. Disabling avahi-daemon has helped.

On recreating VMs (or VM templates): It might help if the load is shown in Qubes Manager. If the load is not shown in Qubes Manager, I'd guess it is a Xen-related or dom0-related issue.

Regards,
Vít Šesták 'v6ak'

[lok...@gmail.com]

I have tried to recreate the circumstances under which this happened, but I haven't been able to.

The fact that none of the diagnostics tools I ran, both on dom0 and the VM other than xentop was showing anything suggests to me that it's definitely Xen-related as you said.

If it happens again, is there something I can do to collect more information in order to figure out what is happening? Some tools I forgot to run, or some logs I never looked at?

[Vincent Adultman]

This happens to me sometimes on the current Xen/Linux versions. When I

look at top in the offending VM its "kswapd" that has gone berserk.

--

Chris Laprise, tas...@openmailbox.org

https://twitter.com/ttaskett

PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

--

I can confirm I've also experienced this with kswapd in either sys-net or sys-firewall, can't remember which. Only way around it I found was to reboot the VM.

[qubenix]

'Vincent Adultman' via qubes-users:

I have also experience with kswapd going crazy in a Debian vm with
intense I/O and CPU. Kswapd persisted to rock the cpu even after all the
software was done, for about 10 minutes, until I shutdown the vm. Seems
to be a rare condition though because I experience it only 1/50 times
with this vm.

--
qubenix
GPG: B536812904D455B491DCDCDD04BE1E61A3C2E500

[Chris Laprise]

I recently switched to kernel 4.9 and I can't remember it happening
since then. So maybe the problem was with the 4.8 kernel I was using...

[qubenix]

Chris Laprise:

> On 06/18/2017 10:01 AM, qubenix wrote:
>> 'Vincent Adultman' via qubes-users:
>>> This happens to me sometimes on the current Xen/Linux versions. When I
>>> look at top in the offending VM its "kswapd" that has gone berserk.
>>>
>>> --
>>>
>>> Chris Laprise, tas...@openmailbox.org
>>> https://twitter.com/ttaskett
>>> PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
>>>
>>> --
>>> I can confirm I've also experienced this with kswapd in either
>>> sys-net or sys-firewall, can't remember which. Only way around it I
>>> found was to reboot the VM.
>>>
>>
>> I have also experience with kswapd going crazy in a Debian vm with
>> intense I/O and CPU. Kswapd persisted to rock the cpu even after all the
>> software was done, for about 10 minutes, until I shutdown the vm. Seems
>> to be a rare condition though because I experience it only 1/50 times
>> with this vm.
>>
>
> I recently switched to kernel 4.9 and I can't remember it happening
> since then. So maybe the problem was with the 4.8 kernel I was using...
>

I haven't noticed it since 4.9 either. I just recently switched as well.

--
qubenix
GPG: B536812904D455B491DCDCDD04BE1E61A3C2E500