libvirt.libvirtError: internal error: libxenlight failed to create new domain

[Pascal Dupuis]

Hello,

context: Qubes 3RC0, x86_64 4 cores 8Gb Ram

from time to time launching an AppVm in a new domain fails. Today it fails continuously ...

Trying to launch it manually from dom0 result in the message given as subject. There are four domain (including dom0, sys-net and sys-firewall) active with low CPU activity and not that many memory. The new domain is based upon the same template.

I guess shutdown down every VM but dom0 will solve it. Any hint on what's going wrong ?

Regards

Pascal

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Check /var/log/libvirt/libxl/VMNAME.log (replace VMNAME with name of
failed VM). There should be some details. Unfortunately the source of
the problem is probably not in the last line (after VM startup fails,
libvirt cleanup the VM remainings and it also produces some log
entries), and there is no timestamps... If you'd like to copy
substantial part of the log, take a look here[1] for instruction how to
extract data from dom0.

[1] https://www.qubes-os.org/doc/CopyToDomZero/

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVkkvKAAoJENuP0xzK19csZiQH/07DuN55r4hC/ZnJwV59yeH7
PALUA7p2cZ0uMNMokvNno9y4WB5AIFSdPjX3cUL4b2d2IKOMQjZ8nf/dgCMvqrB4
wQSkMqP1HWZ2ZxRvZySNnxfj2j7TK/y1Xpj8F0P62OfZnnCyJgmpIbAXssrnBppC
yXKvcebu07hG3r2UHlFxIzOt1F+7WXbgD6fmqFMhspq1KfFV7hKTomCrSkHezTs1
2ntuLlnYrf79/NjM6fnFpxqDCfNoCoDbp9FMk+qxCDHctTzGOdv9HAv1wet6EZpk
gccInEYq9vak9K2aT+XCC1fYdOCGcIOi2IJV+CdAiOfowZ+H1DCQp1OUAWdqrSo=
=umg8
-----END PGP SIGNATURE-----

[Pascal Dupuis]

Le mardi 30 juin 2015 09:57:08 UTC+2, Marek Marczykowski-Górecki a écrit :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Jun 30, 2015 at 12:15:19AM -0700, Pascal Dupuis wrote:
> Hello,
>

Check /var/log/libvirt/libxl/VMNAME.log (replace VMNAME with name of
failed VM). There should be some details. Unfortunately the source of
the problem is probably not in the last line (after VM startup fails,
libvirt cleanup the VM remainings and it also produces some log
entries), and there is no timestamps... If you'd like to copy
substantial part of the log, take a look here[1] for instruction how to
extract data from dom0.

 

Hello Mareck,

thank you for your answer, I'll check that this evening. IN the meantime, shutting down the 'personal' domain failed. I got many popups with the warnings: VM did not close. Wait another 20 seconds or kill it ? After many attempts, as the CPU utilisation was at 0 in the offending machine as well as in the dom0, I killed it. The scenario was as follows: I started the AppVM and launched dropbox. There was a big update pending as I cleaned the content and removed around 2 GB. I _suppose_ that it interfered badly with the COW mechanism ?

 So question:
- is there some limit to the amount of data which can be modified per session ? The disk storage for this VM is 64 Gb.
- is it possible at regular intervals (let say one hour) to flush the pending modifications to the disk storage ?
- if not, is is possible, from dom0, to start some AppVM from a script. Said AppVM should start dropbox, let it run for one hour, then close it and gracefully close. The dom0 should then restart the AppVM.
The two last options should limit the amount of data to write back when closing the AppVM.

Regards

Pascal

[JPL]

Slightly off topic, but once I have closed down any apps running within them I now kill VMs from VM Manager rather than shutting them down. I haven't noticed any bad effects or lost data yet and kill is almost immediate whereas shutdown can sometimes take minutes. What are the risks in using Kill instead of Shutdown? 

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Jun 30, 2015 at 01:12:38AM -0700, Pascal Dupuis wrote:
>
>
> Le mardi 30 juin 2015 09:57:08 UTC+2, Marek Marczykowski-Górecki a écrit :
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On Tue, Jun 30, 2015 at 12:15:19AM -0700, Pascal Dupuis wrote:
> > > Hello,
> > >
> >
> > Check /var/log/libvirt/libxl/VMNAME.log (replace VMNAME with name of
> > failed VM). There should be some details. Unfortunately the source of
> > the problem is probably not in the last line (after VM startup fails,
> > libvirt cleanup the VM remainings and it also produces some log
> > entries), and there is no timestamps... If you'd like to copy
> > substantial part of the log, take a look here[1] for instruction how to
> > extract data from dom0.
> >
>
>
>
> > Hello Mareck,
> >
>
> thank you for your answer, I'll check that this evening. IN the meantime,
> shutting down the 'personal' domain failed. I got many popups with the
> warnings: VM did not close. Wait another 20 seconds or kill it ? After many
> attempts, as the CPU utilisation was at 0 in the offending machine as well
> as in the dom0, I killed it.

Check the VM console log (/var/log/xen/console/guest-VMNAME.log) - it is
accessible from Qubes Manager (right click on the VM). Maybe there was
some crash during VM shutdown? Or simply VM ignored shutdown request...

> The scenario was as follows: I started the
> AppVM and launched dropbox. There was a big update pending as I cleaned the
> content and removed around 2 GB. I _suppose_ that it interfered badly with
> the COW mechanism ?

As long as all the changes are in /home (or anywhere else in /rw), it
shouldn't be a problem.
The only limitation is physical space in dom0. As long as you have disk
space, it shouldn't be a problem.

> So question:
> - is there some limit to the amount of data which can be modified per
> session ? The disk storage for this VM is 64 Gb.
> - is it possible at regular intervals (let say one hour) to flush the
> pending modifications to the disk storage ?
> - if not, is is possible, from dom0, to start some AppVM from a script.
> Said AppVM should start dropbox, let it run for one hour, then close it and
> gracefully close. The dom0 should then restart the AppVM.
> The two last options should limit the amount of data to write back when
> closing the AppVM.

There is no need for such actions to "write back" the data of AppVM. The
only place where data is committed at VM shutdown is TemplateVM. For
AppVM all the changes (especially in /home) are written to disk
directly.

The only possible issue are some temporary data, like swap, /tmp etc.
Those can fill up during long VM run time. Once again - it shouldn't be
a problem if you have disk space in dom0. But if you want to free some
disk, you may want to restart long running VMs.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVkmnUAAoJENuP0xzK19cs8yIH/R2atnhRH05zRp80sr6QXKYY
OzpCtDS7u6GGe8x+Lvu93yjltujbWcg3UzY+L6MRf2kKZtkFpNhI5YYjAv9KIxD/
GPBDSNQ7k/Yg0o2ngfOkVjx9GpfwvXcHwjgePT10SKqiqx6kigE08Wx0ikURMSuL
XoXJLZEWpijpYam/XPadnZen9NMViVEdTGh9FaYlowYDcdnSsjRJo5mVXmBxIGOC
h3YeO0JBr3UpqJ0/CjGvWz690ueWiQaO8mp6KLDfCA9NgFQxCgiD4F9E5S0WeKs8
CA2E2yvEu0KmCbDuRSYnOVbmz3L6sMO9BDlwX3Pxw9oYZq3Np7lg78arOhbm1Io=
=Fqc+
-----END PGP SIGNATURE-----