AppVMs start but zero of the apps are starting?

[Stumpy]

Last night I noticed that I was not able to start an appvm via dom0
(qvm-run appvm programname) so then tried to start another app in
another appvm, nothing.
I had a bunch of things open still from earlier in the afternoon so
figured I needed to reboot, did, and still nothing, except I dont have
any previous apps open up so my system is barely usable.

When i try to run an app, say from the xfce menu (or directly from the
dom0 terminal) I get the little pop up that the VM is starting, and it
shows up in zentop and the qubes manager, but no app. I tried to start
apps from the right click "run" option in the qubes manager but nothing,
doesnt matter if the appvm is already running or not.

The only thing i have done in dom0 in terms of "messing around" was to
modify the sudo vi /etc/qubes/quid.conf so that one of my appvms would
open w/o a border (which I since undid in attempts to get things
working) i cant think of anything else i have done in dom0?

This is killing me as I am not working on my work/win comp :( so any
help would really really really be appreciated. Please let me know if
there are any logs or something i should be posting?

[Steve Coleman]

This sounds very similar to a problem I have been having (at home and at
work), only your issue sounds like a much worse case of it.

Ref: Qubes 4.0.1, Fedora-30 template

When I come into work in the morning, or upon booting my workstation at
home, if I launch an app in a non-running VM (sometimes subsequent
re-launches of a VM) the app I used to initiate that VM does not come
up. The pop-up message of the starting VM appears, then nothing. The VM
gets started and the disk is whirring away, but the app never appears.
If I launch the same app again sometimes both instances of that app
appear one right after they other. Between those two invocations I might
even wait until all the disk activity settles down and everything works
fine.

If for instance I use Qubes Manager to "update qube" the window almost
never comes up the first time. The second time it will. If I start the
template first and then select "update qube" it almost always comes up
correctly, unless something is chewing up all the CPU or hitting the
disk pretty hard.

My issue seems to be related to too much activity of the AppVM's
services creating enough lag to the system that the qrexec either times
out or gets slowed down to the point of not completing the launching of
the app. This is frustrating because after the first launch it seems to
work better, so testing of why it isn't starting clearly needs to be
planned in advance. Perhaps some resources get cached in memory the firt
time so it starts that VM quicker, and thus the qrexec doesn't time out?

I would suggest turning on the "Run in debug mode" option in the Qubes
Manager's AppVM configuration so you can collect better logging
information and see if that tells you anything. That is what I am
planning to do tomorrow morning before launching anything. I had just
turned it on for one VM this morning that sometimes acts up, and
wouldn't you know it, it has not repeated that problem launching that VM
the second time. Maybe tomorrow, or if I leave the machine alone for a
while I might get it to repeat again. I think I will make a habit of
turning on the debugging before launching any new VM's just in case I
can catch it in the act of not acting properly.

[Stumpy]

On 2019-12-18 19:08, Steve Coleman wrote:
> On 2019-12-18 07:57, Stumpy wrote:
>> Last night I noticed that I was not able to start an appvm via dom0
>> (qvm-run appvm programname) so then tried to start another app in
>> another appvm, nothing.
>> I had a bunch of things open still from earlier in the afternoon so
>> figured I needed to reboot, did, and still nothing, except I dont have
>> any previous apps open up so my system is barely usable.
>>
>> When i try to run an app, say from the xfce menu (or directly from the
>> dom0 terminal) I get the little pop up that the VM is starting, and it
>> shows up in zentop and the qubes manager, but no app. I tried to start
>> apps from the right click "run" option in the qubes manager but
>> nothing, doesnt matter if the appvm is already running or not.
>>
>> The only thing i have done in dom0 in terms of "messing around" was to
>> modify the sudo vi /etc/qubes/quid.conf so that one of my appvms would
>> open w/o a border (which I since undid in attempts to get things
>> working) i cant think of anything else i have done in dom0?
>>
>> This is killing me as I am not working on my work/win comp :( so any
>> help would really really really be appreciated. Please let me know if
>> there are any logs or something i should be posting?
>>
>
> This sounds very similar to a problem I have been having (at home and
> at work), only your issue sounds like a much worse case of it.

Yeah, its killing me. I dont suppose you have figured out a fix?

A detail that i forgot to mention is that, AKAIK all the dom0 apps are
opening ok, so the terminal program, task manager, system settings
etc... just *no* appVMs.

Please, Qubes Gurus, any help would really be appreciated!

[Stumpy]

On 2019-12-18 19:08, Steve Coleman wrote:

Thanks for pointing out where to turn the logging on.
Questions though, I was attempting to look through the three logs of two
different appvms but was not really sure where to start (was just shy of
6k lines).
1) Would posting the output to like a pastebin be useful/appropriate?
2) How sensitive is the info in the logs? (In looking through them it
didnt seem like I was giving much info away but at the same time I am
barely able to make sense of most of it).

Thanks for the help, I of course want to get this figured out... I love
my qubes box, all my friends live inside it :P

[Stumpy]

Well as I have not been able to find (understand) anything wrote I
thought I'd cross my fingers and post logs from two different appvms
(debian and fedora). It occured to me though that if this is happening
with all appvms, which it is, then perhaps its an issue with dom0 but
anyway, am posting just in case. If there are dom0 logs that would be of
use then please let me know.

https://pastebin.com/LAXPSyBc

[Steve Coleman]

What I see in the log that looks suspicious is:

domain dead
Failed to connect to gui-agent

if it connected you would have seen something like:

[ 18.849522] audit: type=1130 audit(1576676114.676:22): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=qubes-gui-agent comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

And that "audit:" message is not anywhere in your logs, so I think your
AppVM's template might be broken. If you have no gui-agent then nothing
would display when it is run, which pretty much seems to be your issue.
Mine is running, so our problems are likely unrelated.

What happens when you use "qvm-run -p -v AppVM appname" to run the same
app in that same vm? If the application runs but has any runtime errors
like opening the display you might see what is happening from the dom0
command line. if the app allows some type of -v switch try that too. You
can also run ps and other system commands to see what services are
running etc. You can also run dnf that way to reinstall a package on the
fly if you can figure out which one is broken

qvm-run --user root -p AppVM dnf -y reinstall <package-name>

That would only be a temporary fix until you do the same in the template
that AppVM is based on, but it might help you figure out what qubes
packages are broken.

Another thing you should try is to just use a new template for that same
AppVM and see if that allows your application to run properly. Just use
Qubes Manager to switch your AppVM to a new template and see what
happens. I you don't have another template you can use

sudo qubes-dom0-update --action=search qubes-template

to look for a new template and then run

sudo qubes-dom0-update <template-name>

to install that new one. If all your AppVM's are using the same template
and they all do not run, then that might be the answer.

Let us know what you find out.

Steve