VM`s not starting

[23radi23]

I encountered a problem when starting a vm once in a while where the vm won`t start and "cannot execute qrexec-daemon!" pops up. I ignored it, because it happend rarely and some tries later the vm would start or at least after a reboot everything would work again.

I just updated dom0 and now any vm wont start any more, always with the error message stated above. This means not only my custom vm's wont start any more, also netvm and the template vm`s wont start any more.

/var/log/qubes/huid.fedora-20-x64.log states a lot of messages like "invalid PMaxSize for 0x3a00039" or "invalid PResizeInc for 0x3a00039" before finally "event channel disconneted" "Icon Size: 128x128" "domain dead"

Since xen-hotplug.log was really big, I deleted the file, recreated it and tried again to start the template. After the first unsuccsesful attempt to start I shutted down and xen-hotplug.log stated "losetup: /dev/loop4: detach failed: no such device or address". I checked and /dev/loop4 exists. anyway... when I repeat trying to start and shutdown vm`s there are no new lines in xen-hotplug.log, so i guess its not the main problem here.

I`m new to linux but since I like Qubes very much I´m using qubes as my main os by this time. Any suggestions are welcome since all my data are in the vms.

[23radi23]

I`m still stuck, anybody any idea?

I will gladly post more information if anybody could tell me which information could help to find a solution.

[23radi23]

Still stuck, but i might be able to add useful information.

I learned about the "sudo xl console vmname" command and was able to log in to my netvm.

"ls - A | grep qr" returned nothing so I tried to start "/lib/qubes/qubes-agent" wich stops saying "libvchan_server_init: No such file or directory"

Again... any suggestion is welcome.

[23radi23]

something else that might help, systemctl says:
...
...
proc-fs-nfsd.mount   loaded   failed   failed   RPC Pipe File System
proc-sys...infmt_misc.mount   loaded   failed failed Arbitrary Executable File Fo

[Joonas Lehtonen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

> "libvchan_server_init: No such file or directory"

I'm also having regular problems with qubes-agent not coming up..
(not anymore since I upgraded the VM kernel but I have no long time
data yet)

I observed that almost every time this happens kernel modules won't
load (you can see that when connection via xl console + scrolling up
if it happens also on your system.. you will see it if it happened
because it is the only red line within many green lines)

I upgraded my VM kernel, but that requires some manual steps.

-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJU8ae0AAoJENGIB/ssoMC2WSEQAKEtw7vXVGy75q84bUl0HXOZ
yK60s0w0SrqG/gifDmCZJyf88jffsk5GFR+Wlg2/aOEKzFpVYgsRGkUPdUy5OBR5
WvOx1Em0s+dgqjAh80JWVBfV6Tq4fRZAfMYHB41h8uaHdY58M3e1IImuuniPC6rd
d4YsJQJD/j2hRxq863SGGZNfYwkD5t7sQLUG5zK/QGrmt9/MSBRrjLAibnUj8aFh
59G+nncHxeqU4Hzrp2b3KxJsK8GqSuoxqyseAMlerm5NzZEjMr7ashj6QusccPY0
Jy9I0ChfhnDMhTV0kiddNN8VU+iZpIQA8yKVzPsm9BI0RJ1U60LWXAJSArIyp5YR
FBBvVoanyhdlEDE21IVQWAOJn9Kmdb4YOloBXeQdNa4JMHnnmQ1w8iRYGleyVXgq
jWCTcUAVN1eTA5zX+UZvzcpv6cf6THC9uNTuA7FqCQ+CLdnrlaDvwt8wvqbN5eav
+2K0CqVlOV0JanY1cJCtTFxEGiL/mVqcSEo9OYoN+kSB5fDjPgHgdGUUz3uFytZ9
c07PLVSXbxrtq+LAAAMiU1ezePtNsqTDKbRV9irE5jE/ni9LHDK7hV+1g7WbqeLI
c+uv+LKOH1PTDVFCzg6U77UZADA+rAxKwJUQXFqGPVBI2JXHgsQKN5uPDTxZ9adD
y2q1U/hEhweL0HJE1CXf
=iYKq
-----END PGP SIGNATURE-----

[holy6...@gmail.com]

Having the same issue caused by a fedora-20-x64 template VM update.
uname -r says 3.12.23-1.pvops.qubes.x86_64

All in all I'm unable to start any VMs. Probable reasons are mentioned in the log:
"Failed to start load kernel modules
Failed to start mount arbitrary executablefile formats file system
Failed to start set up additional binary formats"

Unfortunately I cannot copy the logs as I have to work on a non-Qubes machine atm for obvious reasons.

Anyone got a clue how to fix this?
Otherwise I guess the template VM has to be replaces with an older one...

[David Hobach]

Going back to 3.12.18-1 fixed the issue for me. You can select the
kernel version in the Qubes VM Manager -> Settings -> Advanced window
per VM.

Honestly I don't understand why the issue came up though as I don't
remember any recent update to 3.12.23-1 and the 3.12.23-1 folder exists
since July 2014.

Problems came up after the following fedora 20 template VM update:

sudo yum history info 198
Loaded plugins: langpacks, post-transaction-actions, refresh-packagekit,
yum-qubes-hooks
Transaction ID : 198
Begin time : Sat Feb 28 16:02:34 2015
Begin rpmdb : 2391:2bb912393f4600b40e45b567a2d0a666bc684810
End time : 16:03:05 2015 (31 seconds)
End rpmdb : 2391:b117bf4f3b55e0e0a6610307c21af965a1520c35
User : System <unset>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-2.fc20.x86_64 @updates
Installed yum-3.4.3-152.fc20.noarch @updates
Installed yum-metadata-parser-1.1.4-9.fc20.x86_64 @fedora
Packages Altered:
Updated curl-7.32.0-18.fc20.x86_64 @updates
Update 7.32.0-19.fc20.x86_64 @updates
Updated dump-1:0.4-0.21.b44.fc20.x86_64 @fedora
Update 1:0.4-0.24.b44.fc20.x86_64 @updates
Updated firefox-35.0.1-5.fc20.x86_64 @updates
Update 36.0-1.fc20.x86_64 @updates
Updated gnutls-3.1.28-1.fc20.x86_64 @updates
Update 3.1.28-2.fc20.x86_64 @updates
Updated ibus-1.5.9-8.fc20.x86_64 @updates
Update 1.5.9-10.fc20.x86_64 @updates
Updated ibus-gtk2-1.5.9-8.fc20.x86_64 @updates
Update 1.5.9-10.fc20.x86_64 @updates
Updated ibus-gtk3-1.5.9-8.fc20.x86_64 @updates
Update 1.5.9-10.fc20.x86_64 @updates
Updated ibus-libs-1.5.9-8.fc20.x86_64 @updates
Update 1.5.9-10.fc20.x86_64 @updates
Updated ibus-setup-1.5.9-8.fc20.noarch @updates
Update 1.5.9-10.fc20.noarch @updates
Updated ibus-wayland-1.5.9-8.fc20.x86_64 @updates
Update 1.5.9-10.fc20.x86_64 @updates
Updated libcurl-7.32.0-18.fc20.x86_64 @updates
Update 7.32.0-19.fc20.x86_64 @updates
Updated librsvg2-2.40.1-1.fc20.x86_64 @fedora
Update 2.40.7-1.fc20.x86_64 @updates
Updated libsmbclient-2:4.1.15-1.fc20.x86_64 @updates
Update 2:4.1.17-1.fc20.x86_64 @updates
Updated libwbclient-2:4.1.15-1.fc20.x86_64 @updates
Update 2:4.1.17-1.fc20.x86_64 @updates
Updated samba-client-2:4.1.15-1.fc20.x86_64 @updates
Update 2:4.1.17-1.fc20.x86_64 @updates
Updated samba-common-2:4.1.15-1.fc20.x86_64 @updates
Update 2:4.1.17-1.fc20.x86_64 @updates
Updated samba-libs-2:4.1.15-1.fc20.x86_64 @updates
Update 2:4.1.17-1.fc20.x86_64 @updates
Updated thunderbird-31.4.0-1.fc20.x86_64 @updates
Update 31.5.0-1.fc20.x86_64 @updates
Updated vorbis-tools-1:1.4.0-13.fc20.x86_64 @updates
Update 1:1.4.0-14.fc20.x86_64 @updates
history info

I also tested to downgrade gnutls, but it didn't fix the issue. Only the
other kernel version helped.

I guess that's why Joonas started the other thread on Qubes OS kernel
package update policies...

[Marek Marczykowski-Górecki]

I cannot reproduce the problem with all the Fedora updates installed and
with 3.12.23-1 kernel (or any other). Can you provide VM startup log?
The most useful would be to access failed VM console using "sudo xl console"
and copy /var/log/messages or output of journalctl -l -b --no-pager.

> I guess that's why Joonas started the other thread on Qubes OS kernel
> package update policies...

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[David Hobach]

Attached the journalctl output. Thanks for looking into this one!

[Marek Marczykowski-Górecki]

This is the problematic part:
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: Waiting for /dev/xvdd
> device...
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: mount: /dev/xvdd is
> write-protected, mounting
> read-only
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: mount: wrong fs type,
> bad option, bad
> superblock on /dev/xvdd,
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: missing codepage or
> helper program, or other
> error
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: In some cases useful
> info is found in syslog -
> try
> M??r 01 09:16:06 localhost dracut-pre-pivot[365]: dmesg | tail or so.
> M??r 01 09:16:06 localhost kernel: EXT3-fs (xvdd): error: can't find
> ext3 filesystem on dev
> xvdd.

This messages is about mounting /lib/modules in the VM. It looks like
you have /var/lib/qubes/vm-kernels/3.12.23-1/modules.img file broken.
Perhaps you've run out of disk space during creation of that file? Or
some unclean system shutdown?

The file is created in post installation script of kernel-qubes-vm
package, you can recreate it by manually calling appropriate part of
that postinstall script as root (rpm -q --scripts kernel-qubes-vm-3.12.23).

Theoretically you can just reinstall kernel-qubes-vm package, but
currently it isn't easy to do that using only qubes-dom0-update tool -
it will refuse to download already installed package...

[David Hobach]

Awesome Marek, that fixed it!
I'd never have guessed that from the logs...

Kind of strange though that multiple people are affected by the same
modules.img file corruption problem during such a short time frame.

[Marek Marczykowski-Górecki]

Maybe this is about some Fedora dom0 update?

[23radi23]

It took me quite some time to figure out how to get the logs from the vm.... but finally here they are.

Unfortunately rpm -q --scripts kernel-qubes-vm-3.12.23 didn`t work for me and I have no other option then 3.12.23 whe i choose the kernel at qubes vm manager

[Marek Marczykowski-Górecki]

On Mon, Mar 02, 2015 at 05:11:56PM -0800, 23radi23 wrote:
> It took me quite some time to figure out how to get the logs from the
> vm.... but finally here they are.

Looks like the same issue. I have no idea why this happened...

> Unfortunately rpm -q --scripts kernel-qubes-vm-3.12.23 didn`t work for me
> and I have no other option then 3.12.23 whe i choose the kernel at qubes vm
> manager

But you've figured that you need to execute commands printed by that rpm
-q (postinstall part), right?

[inf...@gmail.com]

FYI - I also had a massive issue yesterday with none of my updated templates starting, error "cannot execute qrexec-daemon!" (and VM stays yellow an never goes green)

I have no logs (sorry) because I needed to get back up and running, so I did a re-install and restore from backup

Now, after updating dom0 and templates (cautiously) I cannot reproduce problem

There should have been plenty of disk, and because this happened with several templates updated over a few hours, seems doubtful if was a corrupted/incomplete template update. (I am now keeping a "fallback" last-good template)

Only other relevant change was https://groups.google.com/d/msg/qubes-users/oMorJUIkdSM/sFRai9VdKpAJ - I was a bit freaked that I had been hacked! (is that change dangerous?)
--
CB

[23radi23]

But you've figured that you need to execute commands printed by that rpm
-q (postinstall part), right?

Thank you for looking into my logs Marc. I`m sorry, but I dont think that I figured what you mean !?
I typed "rpm -q --scripts kernel-qubes-vm-3.12.23" at dom0 console ? Should it go somewhere else ?

Sorry for my limited linux knowledge.... I`m still learning.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Mar 03, 2015 at 03:23:35AM -0800, 23radi23 wrote:
>
>
> >
> > But you've figured that you need to execute commands printed by that rpm
> > -q (postinstall part), right?
> >
>
> Thank you for looking into my logs Marc. I`m sorry, but I dont think that I
> figured what you mean !?
> I typed "rpm -q --scripts kernel-qubes-vm-3.12.23" at dom0 console ? Should
> it go somewhere else ?

You'll receive something like:
- -----------------
postinstall scriptlet (using /bin/sh):

mkdir /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64
truncate -s 400M /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64.img
mkfs -t ext3 -F /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64.img >
/dev/null
mount /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64.img
/tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64 -o loop
cp -a -t /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64
/var/lib/qubes/vm-kernels/3.12.23-1/modules/3.12.23-1.pvops.qubes.x86_64
mkdir /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64/firmware
cp -a -t /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64/firmware
/var/lib/qubes/vm-kernels/3.12.23-1/modules/firmware/3.12.23-1.pvops.qubes.x86_64
umount /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64
rmdir /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64
mv /tmp/qubes-modules-3.12.23-1.pvops.qubes.x86_64.img
/var/lib/qubes/vm-kernels/3.12.23-1/modules.img

# If qubes-prefs isn't installed yet, the default kernel will be set by
# %post
# of qubes-core-dom0
type qubes-prefs &>/dev/null && qubes-prefs --set default-kernel
3.12.23-1

exit 0
preuninstall scriptlet (using /bin/sh):

if [ "`qubes-prefs -g default-kernel`" == "3.12.23-1" ]; then
echo "This kernel version is set as default VM kernel, cannot
remove"
exit 1
fi
if qvm-ls --kernel | grep -qw "3.12.23-1"; then
echo "This kernel version is used by at least one VM, cannot remove"
exit 1
fi

exit 0
- ---------------

Now copy a part between "postinstall" and "preuninstall", switch to root
("sudo -i") and paste it there.

> Sorry for my limited linux knowledge.... I`m still learning.
>

- --

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU9Z94AAoJENuP0xzK19csAh0H/jdc6wspmjSlJ9Od098Wgm1t
m5NJHmrOM6HJFqDSGYHPTK0+lYutDJrNm7u2QjSlhES9YSfL9a8VTCrol8K3QUsC
BX9lN2Y8vWftMur1vULvD71JaiePUENKVsnAEw966roECwMjk7x9GXFqxBk8f2/U
VD0bEIzezw+GZCUkcxUNypnPCUjwuWg2TbK76X/dk0uO2NHsICam4Sr/XRQ/WOPv
i1BOmLRjpLv76VwhOweHCH8zN8TGUzL2OCIgNHNgkbAi1qISHL8r2XYFxZPwMJQL
1riIoyMheFfzy2ViBYwiiEKyDTZi9Iu4pivu1T3rtgRNmRINGjAYq01OAuJ5IcA=
=+tWQ
-----END PGP SIGNATURE-----

[23radi23]

Marek, you are awesome. I am writing this from my restored qubes system!

If this helps anybody else I would like to mention that copying the commands results in a confirmation to overwrite 1 file. accepting this would break copying the remaining commands and pasting only the remaining part would not do the trick. I had to manually remove the file, then copy the commands and everything is back to normal.

I must admit that I scared of rebooting or applying pending updates right now. ;)
Thanks again Marek

[Gustav van der Merwe]

This issue sounds similar to what I had encountered and discussed/documented here -> https://groups.google.com/forum/#!topic/qubes-users/V_rU-PVMBi4/discussion

I keep encountering this issue. I've now come to the point that when I want to update templates I shutdown everything, boot up fresh, update everything and then reboot again to start working.

But sometimes I encounter this without updates, just a timing issue or something around starting/stopping vms.

Regards,
Gustav van der Merwe