AEM boot doesn't load serviceVM's since Xen 4.6.3

97 views
Skip to first unread message

Lorenzo Lamas

unread,
Dec 4, 2016, 10:49:13 AM12/4/16
to qubes-users
Since upgrading to Xen 4.6.3-21 from Xen 4.6.1-20, booting with AEM fails to start serviceVM's(netVM, usbVM, firewallVM). When the boot process finally completes, trying to manually launch the VMs through VM Manager doesn't work either. When I choose to boot without AEM, everything works as expected. Problem remains with the latest Xen 4.6.3-24.
It is on a HP Elitebook 8460p with IOMMU running Qubes 3.2

I'm not very familiar with Linux, so if you want logs, please tell me where to find them.

In case it may not be related to the new Xen version but some other update installed at the same time, here is the update history:

Transaction ID : 13
Begin time : Tue Nov 22 19:56:34 2016
Begin rpmdb : 934:24862fb1256d9f648273fa8d88ac172b1a06d3d9
End time : 19:58:15 2016 (101 seconds)
End rpmdb : 939:d785b8e09d66f2d440768eaa6b01d1e77e21ac7c
User : <user>
Return-Code : Success
Command Line : --exclude=qubes-template-fedora-23,qubes-template-fedora-24,qubes-template-whonix-ws,qubes-template-whonix-gw,qubes-template-debian-8, upgrade
Transaction performed with:
Installed dnf-1.1.10-1.fc23.noarch @anaconda/rawhide
Installed rpm-4.13.0-0.rc1.13.fc23.x86_64 @anaconda/rawhide
Packages Altered:
Upgraded glusterfs-3.7.16-1.fc23.x86_64 @qubes-dom0-cached
Upgrade 3.7.17-1.fc23.x86_64 @qubes-dom0-cached
Upgraded glusterfs-api-3.7.16-1.fc23.x86_64 @qubes-dom0-cached
Upgrade 3.7.17-1.fc23.x86_64 @qubes-dom0-cached
Upgraded glusterfs-client-xlators-3.7.16-1.fc23.x86_64 @qubes-dom0-cached
Upgrade 3.7.17-1.fc23.x86_64 @qubes-dom0-cached
Upgraded glusterfs-libs-3.7.16-1.fc23.x86_64 @qubes-dom0-cached
Upgrade 3.7.17-1.fc23.x86_64 @qubes-dom0-cached
Upgraded libraw1394-2.1.0-6.fc23.x86_64 @anaconda/rawhide
Upgrade 2.1.2-1.fc23.x86_64 @qubes-dom0-cached
Upgraded qubes-core-dom0-3.2.11-1.fc23.x86_64 @qubes-dom0-cached
Upgrade 3.2.12-1.fc23.x86_64 @qubes-dom0-cached
Upgraded qubes-core-dom0-doc-3.2.11-1.noarch @qubes-dom0-cached
Upgrade 3.2.12-1.noarch @qubes-dom0-cached
Upgraded qubes-db-3.2.1-1.fc23.x86_64 @anaconda/rawhide
Upgrade 3.2.3-1.fc23.x86_64 @qubes-dom0-cached
Upgraded qubes-db-dom0-3.2.1-1.fc23.x86_64 @anaconda/rawhide
Upgrade 3.2.3-1.fc23.x86_64 @qubes-dom0-cached
Upgraded qubes-db-libs-3.2.1-1.fc23.x86_64 @anaconda/rawhide
Upgrade 3.2.3-1.fc23.x86_64 @qubes-dom0-cached
Upgraded xfce4-datetime-plugin-0.6.2-6.fc23.x86_64 @anaconda/rawhide
Upgrade 0.7.0-1.fc23.x86_64 @qubes-dom0-cached
Upgraded xfce4-diskperf-plugin-2.5.5-2.fc23.x86_64 @anaconda/rawhide
Upgrade 2.6.0-1.fc23.x86_64 @qubes-dom0-cached
Upgraded xfce4-fsguard-plugin-1.0.2-2.fc23.x86_64 @anaconda/rawhide
Upgrade 1.1.0-1.fc23.x86_64 @qubes-dom0-cached
Upgraded xfce4-systemload-plugin-1.1.2-3.fc23.x86_64 @anaconda/rawhide
Upgrade 1.2.0-1.fc23.x86_64 @qubes-dom0-cached
Upgraded quota-1:4.02-5.fc23.x86_64 @anaconda/rawhide
Upgrade 1:4.02-6.fc23.x86_64 @qubes-dom0-cached
Upgraded quota-nls-1:4.02-5.fc23.noarch @anaconda/rawhide
Upgrade 1:4.02-6.fc23.noarch @qubes-dom0-cached
Upgraded libpng-2:1.6.23-1.fc23.x86_64 @anaconda/rawhide
Upgrade 2:1.6.26-1.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-hvm-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-hypervisor-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-libs-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-licenses-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Upgraded xen-runtime-2001:4.6.1-20.fc23.x86_64 @anaconda/rawhide
Upgrade 2001:4.6.3-21.fc23.x86_64 @qubes-dom0-cached
Scriptlet output:
1 sed: can't read /etc/sysconfig/prelink: No such file or directory
2 Redirecting to /bin/systemctl start xenstored.service
3 Generating grub configuration file ...
4 Found theme: /boot/grub2/themes/system/theme.txt
5 Found linux image: /boot/vmlinuz-4.4.14-11.pvops.qubes.x86_64
6 Found initrd image: /boot/initramfs-4.4.14-11.pvops.qubes.x86_64.img
7 Found linux image: /boot/vmlinuz-4.4.14-11.pvops.qubes.x86_64
8 Found initrd image: /boot/initramfs-4.4.14-11.pvops.qubes.x86_64.img
9 done

Chris Laprise

unread,
Dec 4, 2016, 12:49:24 PM12/4/16
to Lorenzo Lamas, qubes-users
On 12/04/2016 10:49 AM, Lorenzo Lamas wrote:
> Since upgrading to Xen 4.6.3-21 from Xen 4.6.1-20, booting with AEM fails to start serviceVM's(netVM, usbVM, firewallVM). When the boot process finally completes, trying to manually launch the VMs through VM Manager doesn't work either. When I choose to boot without AEM, everything works as expected. Problem remains with the latest Xen 4.6.3-24.
> It is on a HP Elitebook 8460p with IOMMU running Qubes 3.2
>
> I'm not very familiar with Linux, so if you want logs, please tell me where to find them.
>
> In case it may not be related to the new Xen version but some other update installed at the same time, here is the update history:
>

This sounds familiar. Try removing the network devices from sys-net to
see if it will start then. Next, re-add the network devices and try
starting sys-net again.

Chris

Marek Marczykowski-Górecki

unread,
Dec 4, 2016, 4:27:19 PM12/4/16
to Lorenzo Lamas, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Dec 04, 2016 at 07:49:13AM -0800, Lorenzo Lamas wrote:
> Since upgrading to Xen 4.6.3-21 from Xen 4.6.1-20, booting with AEM fails to start serviceVM's(netVM, usbVM, firewallVM). When the boot process finally completes, trying to manually launch the VMs through VM Manager doesn't work either. When I choose to boot without AEM, everything works as expected. Problem remains with the latest Xen 4.6.3-24.
> It is on a HP Elitebook 8460p with IOMMU running Qubes 3.2
>
> I'm not very familiar with Linux, so if you want logs, please tell me where to find them.

What exactly you get when starting sys-net manually? Also, take a look
at /var/log/libvirt/libxl/libxl-driver.log and
/var/log/xen/console/hypervisor.log for any related messages. For
example you can use `tail -f` while starting sys-net manually, to see
what messages will show up there:

sudo tail -f /var/log/libvirt/libxl/libxl-driver.log
/var/log/xen/console/hypervisor.log

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYRIoyAAoJENuP0xzK19csBTEH/AtfVWEDUUToA/MLt2sLxbmc
wW2fhRsegzJNKNZhBDmXlZw1Oa+B7oFqVL5o1lFrCrvpwXhNatL3GV78XBK2hI/F
sWi6Yi/WkAz68I71dKWX9ixKY1/UZxXhf8Cl6pnFWmFBeyePL/U/vLtFG7mR2qNd
MTP/7jnKsKBJMsuwvX2Maa0EJe+apjetgmSyCybvFw/Sgvozc2NEDKyxJhMn82Ss
Na9MTNbN6CB5RA/5jPpU0ppS0EO3GH4TfN9AGBPWLimdvJkObgJmtCOHweSHn6h2
Dpm+eL78UTqHN7HNrAifi9rufEzz0yeAMXYfXRi1Nj/Oa9QD/IxUYBcY6ewsWAc=
=02Xd
-----END PGP SIGNATURE-----

Lorenzo Lamas

unread,
Dec 5, 2016, 2:35:21 PM12/5/16
to qubes-users, lama...@gmail.com
On Sunday, December 4, 2016 at 10:27:19 PM UTC+1, Marek Marczykowski-Górecki wrote:
> What exactly you get when starting sys-net manually? Also, take a look
> at /var/log/libvirt/libxl/libxl-driver.log and
> /var/log/xen/console/hypervisor.log for any related messages. For
> example you can use `tail -f` while starting sys-net manually, to see
> what messages will show up there:
>
> sudo tail -f /var/log/libvirt/libxl/libxl-driver.log

It says cannot create qrexec-daemon.
During boot it says "Failed to start Qubes NetVM startup" and "Failed to start Start Qubes VM sys-net"(and some more for the other serviceVMs)

This is the log from libxl-driver.log:

[user@dom0 ~]$ sudo tail -f /var/log/libvirt/libxl/libxl-driver.log
2016-12-05 17:24:33 CET libxl: error: libxl_pci.c:1047:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:23:00.0
2016-12-05 17:24:33 CET libxl: error: libxl_pci.c:1047:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:23:00.2
2016-12-05 17:24:43 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/4/0 not ready
2016-12-05 17:24:43 CET libxl: error: libxl_pci.c:1321:do_pci_remove: xc_physdev_unmap_pirq irq=16: Invalid argument
2016-12-05 17:24:54 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/4/0 not ready
2016-12-05 17:24:54 CET libxl: error: libxl_pci.c:1047:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:23:00.0
2016-12-05 17:25:04 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/4/0 not ready
2016-12-05 17:25:04 CET libxl: error: libxl_pci.c:1321:do_pci_remove: xc_physdev_unmap_pirq irq=18: Invalid argument
2016-12-05 17:25:04 CET libxl: error: libxl_pci.c:1047:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:23:00.2
2016-12-05 17:25:14 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/4/0 not ready
2016-12-05 17:29:34 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/5/0 not ready
2016-12-05 17:29:45 CET libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/5/0 not ready

and hypervisor.log:
[user@dom0 ~]$ sudo tail -f /var/log/xen/console/hypervisor.log
(XEN) 0000000000000000 ffffffff81c03f30 ffffffff81d53c6a 3e55772b4f5cc499
(XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000
(XEN) 0000000000000000 ffffffff81c03f40 ffffffff81d53339 ffffffff81c03ff8
(XEN) ffffffff81d56b96 000206a700000000 0000000103100800 000000001f898b75
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 00010102464c457f
(XEN) 0000000000000000 00000001003e0003 00000000000009b0 0000000000000040
(XEN) d6v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08022c643 create_bounce_frame+0x12b/0x13a
(XEN) Domain 6 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.6.3 x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff81d57b3a>]
(XEN) RFLAGS: 0000000000000292 EM: 1 CONTEXT: pv guest (d6v0)
(XEN) rax: 0000000000000000 rbx: 0000000000012c00 rcx: 00000000ffffffff
(XEN) rdx: ffffffff81a3f810 rsi: 000000000000003f rdi: 0000000000000000
(XEN) rbp: ffffffff81c03e70 rsp: ffffffff81c03da8 r8: 0000000000000011
(XEN) r9: 0000000000000168 r10: 0000000100000000 r11: 0000000000000000
(XEN) r12: 00000000000a9b00 r13: 0000000080000000 r14: 0000000111d00000
(XEN) r15: 0000000000000011 cr0: 0000000080050033 cr4: 00000000000426e0
(XEN) cr3: 0000000333409000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c03da8:
(XEN) 00000000ffffffff 0000000000000000 ffffffff81d57b3a 000000010000e030
(XEN) 0000000000010092 ffffffff81c03de8 000000000000e02b ffffffff81d57b3a
(XEN) ffffffff81df7920 0000000000000000 0000000000000000 ffffffff81c03e10
(XEN) ffffffff00000001 0000000000111d00 0000000000012c00 0000000012c00000
(XEN) 7ff0ffff81c03e88 ffffffff00000013 ffffffff81dfc020 db3d2ed3ec447c09
(XEN) 0000000001000000 ffffffff81c03f00 ffffffff81df7920 0000000000000000
(XEN) 0000000000000000 ffffffff81c03e88 ffffffff81d5e406 0000000001000000
(XEN) ffffffff81c03ef0 ffffffff81d5bd08 ffffffff00000010 ffffffff81c03f00
(XEN) ffffffff81c03eb8 db3d2ed3ec447c09 697a696c61697469 db3d2ed3ec447c09
(XEN) ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000
(XEN) 0000000000000000 ffffffff81c03f30 ffffffff81d53c6a db3d2ed3ec447c09
(XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000
(XEN) 0000000000000000 ffffffff81c03f40 ffffffff81d53339 ffffffff81c03ff8
(XEN) ffffffff81d56b96 000206a700000000 0000000101100800 000000001f898b75
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 00010102464c457f
(XEN) 0000000000000000 00000001003e0003 00000000000009b0 0000000000000040



> This sounds familiar. Try removing the network devices from sys-net to
> see if it will start then. Next, re-add the network devices and try
> starting sys-net again.
>
> Chris
Yes, removing network devices from sys-net makes it possible to start it, forgot to re-add them and try again, I'll do that later.


Lorenzo Lamas

unread,
Dec 7, 2016, 10:38:17 AM12/7/16
to qubes-users, lama...@gmail.com
> > This sounds familiar. Try removing the network devices from sys-net to
> > see if it will start then. Next, re-add the network devices and try
> > starting sys-net again.
> >
> > Chris
> Yes, removing network devices from sys-net makes it possible to start it, forgot to re-add them and try again, I'll do that later.

If I re-add them, it is again unable to start.

eldo...@riseup.net

unread,
Dec 7, 2016, 3:01:30 PM12/7/16
to Lorenzo Lamas, qubes-users
I have exactly the same problem with booting using AEM
Look at this :
https://groups.google.com/forum/#!topic/qubes-users/Vs2QDsU1zJQ

Lorenzo Lamas

unread,
Dec 7, 2016, 5:18:15 PM12/7/16
to qubes-users, lama...@gmail.com, eldo...@riseup.net
On Wednesday, December 7, 2016 at 9:01:30 PM UTC+1, eldo...@riseup.net wrote:
> I have exactly the same problem with booting using AEM
> Look at this :
> https://groups.google.com/forum/#!topic/qubes-users/Vs2QDsU1zJQ
You don't mention AEM there. Everything works fine on my machine if I boot without AEM. Does booting without AEM change things for you?

Lorenzo Lamas

unread,
Jan 9, 2017, 2:31:51 PM1/9/17
to qubes-users, lama...@gmail.com
On Sunday, December 4, 2016 at 10:27:19 PM UTC+1, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Sun, Dec 04, 2016 at 07:49:13AM -0800, Lorenzo Lamas wrote:
> > Since upgrading to Xen 4.6.3-21 from Xen 4.6.1-20, booting with AEM fails to start serviceVM's(netVM, usbVM, firewallVM). When the boot process finally completes, trying to manually launch the VMs through VM Manager doesn't work either. When I choose to boot without AEM, everything works as expected. Problem remains with the latest Xen 4.6.3-24.
> > It is on a HP Elitebook 8460p with IOMMU running Qubes 3.2
> >
> > I'm not very familiar with Linux, so if you want logs, please tell me where to find them.
>
> What exactly you get when starting sys-net manually? Also, take a look
> at /var/log/libvirt/libxl/libxl-driver.log and
> /var/log/xen/console/hypervisor.log for any related messages. For
> example you can use `tail -f` while starting sys-net manually, to see
> what messages will show up there:
>
> sudo tail -f /var/log/libvirt/libxl/libxl-driver.log
> /var/log/xen/console/hypervisor.log
>
> - --
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab

Not trying to be impatient or rude, but do the logs give any useful information?

Lorenzo Lamas

unread,
Jan 31, 2017, 10:20:36 AM1/31/17
to qubes-users
I just updated Dom0 to the updates in current-testing, this upgraded both Xen(to 4.6.4-25) and AEM(to 3.0.5-1) but nothing has changed.

Reply all
Reply to author
Forward
0 new messages