Troubleshooting Qubes graphical slowness

[tetra...@danwin1210.me]

Periodically all graphics-heavy apps (Firefox, ...) in all VMs seem to
slow down simultaneously. Rebooting fixes the situation. Running `sudo
journalctl -f` in dom0 doesn't show anything unusual. What would you
suggest as a next step towards locating the problem?

[tetra...@danwin1210.me]

vim also appears to be affected by the slowdown.

[tetra...@danwin1210.me]

Further inspection shows there's a LOT of disk I/O going on.

after installing iotop in dom0, this appears to be coming from command
[NN.xvda-0], presumably one of the VMs. How do I map the NN (number) to
a given running VM?

[awokd]

tetrahedra via qubes-users:

Check xl top. I think you can find the offending VM with that. You might
be running out of system RAM too, which would be shown at the top.

--
- don't top post
Mailing list etiquette:
- trim quoted reply to only relevant portions
- when possible, copy and paste text instead of screenshots

[tetra...@danwin1210.me]

On Fri, Dec 27, 2019 at 08:49:02AM +0000, 'awokd' via qubes-users wrote:
>> Further inspection shows there's a LOT of disk I/O going on.
>>
>> after installing iotop in dom0, this appears to be coming from command
>> [NN.xvda-0], presumably one of the VMs. How do I map the NN (number) to
>> a given running VM?
>>
>Check xl top. I think you can find the offending VM with that. You might
>be running out of system RAM too, which would be shown at the top.

xl top / xentop doesn't show any two-digit number identifying a VM.

However by trial and error it looks like the extreme levels of disk I/O
are a symptom rather than a cause. After shutting down all slowed-down
VMs the disk I/O ended. Then when I re-started a DispVM with Firefox,
the high levels of disk I/O (constant read > 50MB/sec) came back and
Firefox was slow (as before).

Unfortunately I need to get work done so have to reboot to "just make it
go away" but I am still interested in troubleshooting ideas (for when it
happens next).

[tetra...@danwin1210.me]

On Fri, Dec 27, 2019 at 09:57:16AM +0100, tetrahedra via qubes-users wrote:
>Unfortunately I need to get work done so have to reboot to "just make it
>go away" but I am still interested in troubleshooting ideas (for when it
>happens next).

One thing I noticed on reboot -- the initial round of stop jobs (when
shutting down the system, things like unmounting LUKS volumes) all timed
out. Not sure if related.

[awokd]

tetrahedra via qubes-users:

> On Fri, Dec 27, 2019 at 09:57:16AM +0100, tetrahedra via qubes-users wrote:
>> Unfortunately I need to get work done so have to reboot to "just make it
>> go away" but I am still interested in troubleshooting ideas (for when it
>> happens next).

Investigate xl top more thoroughly. You can identify offending VMs with
it, and see if all your RAM is in use which triggers swapping to (slow)
disk.

> One thing I noticed on reboot -- the initial round of stop jobs (when
> shutting down the system, things like unmounting LUKS volumes) all timed
> out. Not sure if related.
>

Don't think it's related.

[tetra...@danwin1210.me]

On Sun, Dec 29, 2019 at 01:44:28PM +0000, 'awokd' via qubes-users wrote:
>tetrahedra via qubes-users:
>> On Fri, Dec 27, 2019 at 09:57:16AM +0100, tetrahedra via qubes-users wrote:
>>> Unfortunately I need to get work done so have to reboot to "just make it
>>> go away" but I am still interested in troubleshooting ideas (for when it
>>> happens next).
>
>Investigate xl top more thoroughly. You can identify offending VMs with
>it, and see if all your RAM is in use which triggers swapping to (slow)
>disk.

My disk is a pretty fast SSD, and I did use xentop (`xl top` is just an
alias for xentop) and it didn't show anything unusual as far as I can
recall. Perusing the xentop man page doesn't show any potentially
relevant options except for `--full-name` and that option doesn't seem
to do anything. Pressing "B" (for "vBds") seems to list a number of
devices for each VM but none of them have any 2-digit unique identifying
number (as `iotop` seems to display).

[Steve Coleman]

I have had graphics slowdown issues in the past on two occasions that
acted like this, so here are some things to try:

1) Add the 'nopat' argument to the 'kernel opts:' boot command line.

> qvm-prefs <AppVM> -s kernelopts nopat

2) The second, I can not seem to locate that email exchange at the
moment, but it was a option on the graphics subsystem that needed to be
turned off. Something like backing store, but I'm sure that is not the
correct name for it. I'll keep looking for that one until I hear back if
#1 above fixed your problem or not.

Steve

[rec wins]

so how many VMs are open at one time, how much RAM have you? you know
you can go into qubes settings and change the max RAM / per VM to say
1500 or so ?

I still believe "speed step" in the UEFI was what was my slowness
problem before, I believe I still have it enabled, slowness for
everything but esp boot times YMMV

[Steve Coleman]

Ok, I still could not find that email exchange, but the second thing to
try is in the XFCE "Window Manager Tweaks" Compositor tab, and try to
disable the "Enable display compositing" entry.

Let us know if either if these works for you. Both worked for me for
their respective graphics performance issue.

Steve

[tetra...@danwin1210.me]

On Mon, Dec 30, 2019 at 05:31:58PM -0500, Steve Coleman wrote:
>>I have had graphics slowdown issues in the past on two occasions
>>that acted like this, so here are some things to try:
>>
>>1) Add the 'nopat' argument to the 'kernel opts:' boot command line.
>>
>> > qvm-prefs <AppVM> -s kernelopts nopat

I just checked, and the VMs in question (all VMs on my system?) already
have `nopat` in the kernelopts

>>
>>2) The second, I can not seem to locate that email exchange at the
>>moment, but it was a option on the graphics subsystem that needed to
>>be turned off. Something like backing store, but I'm sure that is
>>not the correct name for it. I'll keep looking for that one until I
>>hear back if #1 above fixed your problem or not.
>
>Ok, I still could not find that email exchange, but the second thing
>to try is in the XFCE "Window Manager Tweaks" Compositor tab, and try
>to disable the "Enable display compositing" entry.

Disabling display compositing does seem to have improved performance,
but no so much that it fixed the problem. It seems to be something
separate from whatever's going on.

[tetra...@danwin1210.me]

On Sun, Dec 29, 2019 at 01:44:28PM +0000, 'awokd' via qubes-users wrote:
>tetrahedra via qubes-users:
>> On Fri, Dec 27, 2019 at 09:57:16AM +0100, tetrahedra via qubes-users wrote:
>>> Unfortunately I need to get work done so have to reboot to "just make it
>>> go away" but I am still interested in troubleshooting ideas (for when it
>>> happens next).
>
>Investigate xl top more thoroughly. You can identify offending VMs with
>it, and see if all your RAM is in use which triggers swapping to (slow)
>disk.

Looks like my RAM is about 43% free, according to xentop (xl top).