dispvm savefile creation fails when .qubes-dispvm-customized

[Steve Coleman]

I have been having issues with my newly upgraded fedora-25 template for
starting the newly re-created fedora-25-dvm. It appears to _not_ start
anytime I have the ".qubes-dispvm-customized" link in the user
directory. When removing the file it works fine in creating a savefile
and running the dvm, but with the link in place it consistently fails.

All I have customized in the dispvm template is the firefox home page,
but it doesn't matter if the .mozilla directory is there or not. if
.qubes-dispvm-customized is present then the savefile fails and the
fedora-25-dvm vm continues running forever.

There are lots of "A start job is running for Initiali...t" at the end
of the log file (see attached) when it fails. Recreating the
fedora-25-dvm from scratch makes no difference.


========== test with customization ================

[Dom0 ~]$ qvm-run --auto -p fedora-25-dvm 'mv
.qubes-dispvm-customized.bak .qubes-dispvm-customized ; shutdown -h now'

[Dom0 ~]$ echo firefox | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell
dom0 DEFAULT red

# dispvm savefile creation failed

[Dom0 ~]$ qvm-kill fedora-25-dvm



===========test without customization===============

# now try without customization

[Dom0 ~]$ qvm-run --auto -p fedora-25-dvm 'mv .qubes-dispvm-customized
.qubes-dispvm-customized.bak ; shutdown -h now'

[Dom0 ~]$ echo firefox | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell
dom0 DEFAULT red

# success!! ?? Well sort of, there is *no* customization in this DispVM



This is 100% repeatable!

How can I find out what is happening inside during this qfile-daemon-dvm
auto generation process?


Attached is the boot log of the fedora-25-dvm failed attempt to build
the savefile.

[Connor Page]

qvm-create-default-dvm fedora-25

Try that to check if it fails. If not then starting dvms won't be a problem anymore. If it does then there's something wrong with your template.
I use a heavily customized F25 template for dvms and it never fails, but I always run qvm-create-default-dvm after any changes to its private image.

[Steve Coleman]

I have run this command many times, to recreate the dvm, so running it
again simply does not help.

The dvm works fine as long as Qubes does not try to process the
"customization" of the DVM prior to launching it. I have no way
presently to see what portion of the customization process that gets hung.

What I need is to find out how to run that customization code (script?)
under trace or debugger control so I can identify the actual problem.
The question is where is this customization script, dom0 or in the
template? The only script I have found so far (still looking) is dom0
"startup-dvm.sh", but all it does is create symbolic links and set
permissions for the savefile. It never even tests for the existence of
".qubes-dispvm-customized".

[Unman]

When this has come up in the past on the list the answer has been to
delete the saved dvm file and to recreate it, add customization, and
then run qvm-create-default-template again.

If you want to check the script operations, look at
/usr/lib/qubes/init/mount-dirs.sh and
/usr/lib/qubes/init/setup-dvm-home.sh

They are simple scripts, so it shouldnt take much debugging.

unman

[Steve Coleman]

The scripts appear to be working just fine. The real issue is that the
DVM template will not shut down due to:

Failed to get properties: Failed to activate service
'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

..and then it fails to shutdown properly. The failure to rebuild the
customization savefile must somehow trigger the shutdown issue, but I
can't imagine how that would be any different. The fedora-25 template
has no problem shutting down at all. Removing the savefile and
recreating the DVM does not change anything, as there is something
different about how the DVM is running/configuring its dbus/systemd
services. I'll have to look at this some more tomorrow when I get time.
In the mean time the dvm system logs are attached in case anyone has a
clue.

Thanks unman!