Re: [qubes-users] Xen 4.6.1 (testing repo) breaks PCI attachment

84 views
Skip to first unread message

Marek Marczykowski-Górecki

unread,
May 30, 2016, 6:59:54 PM5/30/16
to HW42, Andrew David Wong, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, May 29, 2016 at 04:48:18AM +0200, HW42 wrote:
> Andrew David Wong:
> > On 2016-05-28 11:16, HW42 wrote:
> >> Hi,
> >
> >> with Xen 4.6.1-15.fc20 (which got recently uploaded to the testing
> >> repo) VMs with attached PCI devices don't boot anymore (at least on
> >> my machine).
> >
> >> When I try to boot a VM with attached PCI devices it crashes
> >> immediately. Attached the xen dmesg log of a crash.
> >
> >> Both dom0 and the VM are running the 4.1.24-9.pvops.qubes kernel.

Exactly the same versions works fine for me...

Do you have anything in console log of that VM? If not, try starting it
in debug mode (this enables earlyprintk).

> >> HW42
> >
> >> PS: Do you prefer reports regarding problems with testing packages
> >> on qubes-users or qubes-devel?

Better qubes-devel (moving there).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXTMXiAAoJENuP0xzK19csL4EH/RBL0bP3hWGTCxybqg67k2UZ
RYghDdMmQjGYXE5VC1GJNK5j+g9Iy/fFWqIRXj+GKezSYhXBL/hp/pdF99Ra7V+x
EH7FMqn6UYo0bdpTluDFLlaZFKSqkmy2TpnTW4PiZJ+YbVxTxpp2KzyTsp4vkfYp
n/MxOmreSz/qmYGw/dbj+Cw6Opx5SdSZyGwoksyzj/rZAtGcp+xRsM6whLBp3uaT
/K1D8oFAXsbYnWsbXNnKpO/ZDwk/VsFTW/6iM8CXqkqI3ypP0W/v6Cd3oFJHjNIK
smC3MPBXTcjBm2acXdAywm4AhQY9alSSz3P7iv3I74v0f5XBFunr7be3umLaST4=
=GUP4
-----END PGP SIGNATURE-----

HW42

unread,
May 30, 2016, 11:40:05 PM5/30/16
to Marek Marczykowski-Górecki, Andrew David Wong, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki:
> On Sun, May 29, 2016 at 04:48:18AM +0200, HW42 wrote:
>> Andrew David Wong:
>>> On 2016-05-28 11:16, HW42 wrote:
>>>> Hi,
>>>
>>>> with Xen 4.6.1-15.fc20 (which got recently uploaded to the testing
>>>> repo) VMs with attached PCI devices don't boot anymore (at least on
>>>> my machine).
>>>
>>>> When I try to boot a VM with attached PCI devices it crashes
>>>> immediately. Attached the xen dmesg log of a crash.
>>>
>>>> Both dom0 and the VM are running the 4.1.24-9.pvops.qubes kernel.
>
> Exactly the same versions works fine for me...
>
> Do you have anything in console log of that VM? If not, try starting it
> in debug mode (this enables earlyprintk).

Even with debug mode nothing on console.

Interestingly attaching during runtime seems to work.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJXTQdnAAoJEOSsySeKZGgWPkMP/3pV3V3axLiD1kuxCvtaV7+9
DDR7p0XkJRq8tCEiBvyyLW2ZvAZl6U0qEwSG0gN8jgcHNHzq/V1/DGDC/mBfB3ZU
JsbIfCpz6CTZ1Dy+f2vgy9WZ/tHWJtOUSGB8HlUW0UIkgO/DNTKlC2TCw5x+mWRH
nQSLOcwrVMz0KZRL9illL4cLC6aCIEUO+a+3sebmj8hYBsc4v6N9IsWKZzbT8kCb
Aij+UiUNFZefUfDIihKc/3jlwnA4DRvAd5KEwhuwu7Ot0YNU9b61jlkyqi96fdy3
pREhXd5Mo7o9dg2R4bmSLcZqI8y/GChY4dtCbHv/oxMeQ7eVAabKYIMuY1mqKQqv
Tr6xmFbPN6RsyatXBPnYq+8Qhi2vIsKjAq1zrgSDFXEu7btEhDv5LieABYy+ROBy
e8g2O8NdHRny6z7jGOGbpgGmXuQ5uyZUyzcqMVl9PoqAjbMGurhdCpjqLZE8+jCV
Qi3vtT025RJj5Sa69MQa7+ZRDPzhwiMR9b3iyzoms0yMOxgbdrCc2DGQ94hGMW5E
FAGZPXj7yjMN1giPYY/9l7/hk+WHXU1oQEUOxpoXM/L8aKi/08tj4uJT6wRYOnaf
oz3sVREmsOQPMH4zFHHqVjLfWfFYv1U6qswQ6EGSB28UuEazKRJhWXVsFDS9g3c7
pF00tvzkKkrKlGOwZ8+D
=+/rp
-----END PGP SIGNATURE-----

Marek Marczykowski-Górecki

unread,
May 31, 2016, 3:19:13 AM5/31/16
to HW42, Andrew David Wong, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, May 31, 2016 at 05:39:20AM +0200, HW42 wrote:
> Marek Marczykowski-Górecki:
> > On Sun, May 29, 2016 at 04:48:18AM +0200, HW42 wrote:
> >> Andrew David Wong:
> >>> On 2016-05-28 11:16, HW42 wrote:
> >>>> Hi,
> >>>
> >>>> with Xen 4.6.1-15.fc20 (which got recently uploaded to the testing
> >>>> repo) VMs with attached PCI devices don't boot anymore (at least on
> >>>> my machine).
> >>>
> >>>> When I try to boot a VM with attached PCI devices it crashes
> >>>> immediately. Attached the xen dmesg log of a crash.
> >>>
> >>>> Both dom0 and the VM are running the 4.1.24-9.pvops.qubes kernel.
> >
> > Exactly the same versions works fine for me...
> >
> > Do you have anything in console log of that VM? If not, try starting it
> > in debug mode (this enables earlyprintk).
>
> Even with debug mode nothing on console.
>
> Interestingly attaching during runtime seems to work.

Even when no device was attached at startup at all?
Try booting with attached device, but with empty kernel options (remove
default "iommu=soft swiotlb=8192").
If this doesn't help, try Xen 4.6.1-15.56.fc20 from unstable repo, it
has e820_host option disabled (almost the only change between those
packages, besides Xen version itself). Sources are here:
https://github.com/marmarek/qubes-vmm-xen/tree/bug2038

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXTTrpAAoJENuP0xzK19csYuAH/2vCOjlclNuMlB388IpS3rC3
pGNWSI8ER6dBuTlrYWJDEBgfeKtnYd2916NqAAIAc6JDxVp8iFlo8hgtkf5NitG4
N5gkoVN1lYKSkraWQ3I+tTP3R1b8KFfBq4lAU/wZdeFLPmofJmcNnh50XqdsX9jG
M/3ZOc7+wOvheXl5zY6zDwPzp371D/wvz8omoiAVzaaMOr7XrCnDFQNth2j/mbGs
XV+KsHN3yNHZH3tkPHqrAdLhKSr5sQcr9xkLOCY528oPjoniNZnjzeWpQBY4DZMy
AGvAeABtTFYh169XFTQl5xww6c/DvpNa+TEHoFj0KXzhbrTGreK5HXP7bxF+oTw=
=1w9Z
-----END PGP SIGNATURE-----

HW42

unread,
May 31, 2016, 11:44:25 PM5/31/16
to Marek Marczykowski-Górecki, Andrew David Wong, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki:
> On Tue, May 31, 2016 at 05:39:20AM +0200, HW42 wrote:
>> Marek Marczykowski-Górecki:
>>> On Sun, May 29, 2016 at 04:48:18AM +0200, HW42 wrote:
>>>> Andrew David Wong:
>>>>> On 2016-05-28 11:16, HW42 wrote:
>>>>>> Hi,
>>>>>
>>>>>> with Xen 4.6.1-15.fc20 (which got recently uploaded to the testing
>>>>>> repo) VMs with attached PCI devices don't boot anymore (at least on
>>>>>> my machine).
>>>>>
>>>>>> When I try to boot a VM with attached PCI devices it crashes
>>>>>> immediately. Attached the xen dmesg log of a crash.
>>>>>
>>>>>> Both dom0 and the VM are running the 4.1.24-9.pvops.qubes kernel.
>>>
>>> Exactly the same versions works fine for me...
>>>
>>> Do you have anything in console log of that VM? If not, try starting it
>>> in debug mode (this enables earlyprintk).
>
>> Even with debug mode nothing on console.
>
>> Interestingly attaching during runtime seems to work.
>
> Even when no device was attached at startup at all?

Yes. But I forgot to mention that to actually work I needed to change
the kernel options. When booting without any attached device and the
default kernel options I got this error when attaching a device at
runtime:

kernel: xen:swiotlb_xen: Warning: only able to allocate 1 MB for software IO TLB
kernel: xen:swiotlb_xen: Cannot allocate Xen-SWIOTLB buffer
(rc:-12)

So I added the 'iommu=soft swiotlb=8192' manually. Then I could attach
the device at runtime and it worked.

> Try booting with attached device, but with empty kernel options (remove
> default "iommu=soft swiotlb=8192").

Crashes like with the default options.

> If this doesn't help, try Xen 4.6.1-15.56.fc20 from unstable repo, it
> has e820_host option disabled (almost the only change between those
> packages, besides Xen version itself). Sources are here:
> https://github.com/marmarek/qubes-vmm-xen/tree/bug2038

This version works!
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJXTlnpAAoJEOSsySeKZGgWbWgQAIY4i+DNtf9/0FMVdPNeoHH2
t2t+MKEjVyvGsVJ1xndOXW1InyvFxjZOeXDj+P2JMjekhVBdfI58MdEyC3eUEZHH
8aFyqsZUKvZYx0JvqkflrjYeKH6YS9LxWIRjVOdTBmSO/uHZ/8C9msIpTGeZWPR6
A2inFd7DP0h/3/nh5qLG3ThHEN8M0yIXugGnMIezeE3ScbUPakDu8NHM//tO1VJl
Z3TNG9llUrGgw8twTsIXALBJhG7y8j8tJ/cQnFTOZuPDJaOJNR6OAtfYqJ8+v9rc
yyBvd7oE+ZS9YstkMUfo2S6JfGNdcXvwbohXoGzJH9YZ5kuftNxdWFVJortURvbU
vVV7OBZdng9D3IId2YOh6kpK2kP/MBnqUo+GgCTgrJs623CEwBSk2mGp8ENwOdxo
YRWnJQgRXKUFPjDdC3b/5aHbxMJmAbtuJsVzBo+Y7KmV8RFyHqKZv9v5YaDCAuFU
SQa11PglYszcAWpHnLesyqjtYaBYeD1FkdrOv/DEfbjH9oy0S7i6GKUpxB//AG2P
FwCzbAfFkpClFihLY9s7IWmokPUMm+XjdQSU3vNa7zrXsPlgMgh6Z164jLcUaauw
jQJ3Xg6kIdAtZ4VEGyMrWtzB7RbJzzWjdiB277exrELRF2zci7c8Pu1iFTJxmlU+
5ZFfYU3w8FFmHyipB3KO
=C5QA
-----END PGP SIGNATURE-----

HW42

unread,
Jun 29, 2016, 5:42:06 PM6/29/16
to Marek Marczykowski-Górecki, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

HW42:
When upgrading to R3.2 I discovered that this problem only seems to
exists when booting with AEM. I.e.:

AEM, pci_e820_host=True => doesn't work
AEM, pci_e820_host=False => works
without AEM => works
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJXdEB4AAoJEOSsySeKZGgWSWAP/R9JE9kNeGwSCHJI8OwF1GEz
PEdmZjWm0c9jgGDL+C9hYurajaknrFKgkcdvtG4crowPPE8+0tVbSssL9FelvwJ1
yxvc2Ow/EC+UkP/EV1xMI+/wocMEOgiG1YCkxYJx1pndsL9yX5folK08PbKVsVyF
V+6MCvQj/jEPqLULwk/iNLkNfAsk9udwvRslTUUjhTgZoWfEUsoaCD5jWOTmDgNk
urmNVSUN/BCQDjWup4/t8SX6vc7k9KsP+v3aMITs5IFRxL8JpPx/6r8JDMQz56sh
4htAUSr6Nhqie1v4rSVaUk3VI3uWRckASvtE8+6bmKfQjhjMk5XNXAxMsZ95uGuq
OpVCkutgcJwbl9h1kBNuz1kbCoQbXl9J5i2kY79ZME0n3hRXkvUfGC5aStpSpNSu
u+khuGMaNWS+Soypir/4XmvhvzJ15gznrsh+bO1T2r/aOiFmQkiOb+wqalIjBdUo
zRLaqVe0uMiFEWpomRnByivh0rDulzzXvQcGdnQa6I9V9g0rRuekOUEiAcLN0Zrm
9uxnWqHUkaKyC/7ImnOmv12Jh5K+s23q7F5h/mh62AC/FHJt+AcArQrMkDY6sWbG
rZD6qdRo323IS0/vamrINaUEruwWThaCGQ3lrNI74nHUBY1wKOWqam4Guiclzrl9
ytex3m/94fKjyjOt54ZH
=HsFV
-----END PGP SIGNATURE-----

Marek Marczykowski-Górecki

unread,
Jun 29, 2016, 5:54:29 PM6/29/16
to HW42, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Interesting, are you also affected by the other memory related problem
with tboot (requirement of min_ram option to see all the memory)?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXdEOMAAoJENuP0xzK19cs7MQH/0X/EUeG5NdPM8wuoVWRhPGX
YAS/an2C49QVp2ujJHRTAztyqRUxWYzt2eyP9lfopMVo+Z0fnaZvo8J8quDMR/Id
I/HrGyuiElnRLsAbRVGCCfB452bz6GttM/+6rr5fX9zxJM5VGJGQohzORWCVyBii
iNppQFGtqA4JRlUp5Xn3pNoV73nR3SS5QhQSZgWWlFqTl1Fvp83r32FZj46wdyAL
qrobsJwYrPLbF5pSP6C7qiuZH+3GM3ErnGXd8VGDoLXmhJCq0KaRMy8Xx3A3nj1F
SQqTIIkzgNc7ALjYNJJyRcDiizfRkOzAW0s8myUGkh4UEJYOkykazS+nnJBp+mQ=
=9UPK
-----END PGP SIGNATURE-----

HW42

unread,
Jun 29, 2016, 7:15:19 PM6/29/16
to Marek Marczykowski-Górecki, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki:
No, min_ram=0x2000000 doesn't seem to change anything. With tboot Xen
uses about 6 MB less memory, which seems fine.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJXdFZQAAoJEOSsySeKZGgWfxEP/iNWzXxzYdOFAiVH8Qa+S2dN
5GnHi3UgImzEuXF0lGDIG5tcQdRddbXliCl49OWna2FZDgx/Cdhvpj8qM2N7bhAZ
nSUZoPisSoHMGmM4TNXXHNKSfeL03dvzFcfFANJTk1TYjL10/97W28QCy1P7IPea
xCuloTLthiq91RFbBjSy4ruNS/ApRIm8QikIMNbs+AvQCQex4wQaZ/SDr064qrqP
yHD2FFTamabn+Oej2pueNPCH+9b9ox19V4m3VN/odUsKj1sh2jgm/s6KTUET2UFp
8XSjyXHv3cqQZIa8CpZnn9J74n3yo4+7MRKeXX62qwustZJR9FD3Hb7lHSLSCu0t
wQHIH4NQ5Eec63fxuhhjY/B4e+W8iV+x7GlgJ+3cR2vVdYqmC08ZKd0rbZ1C86Vl
kE5X+zWQ6+SxACL9UWh4xIHQNAyL3KXV2VkQNq3lUJ7p1t6km6PYsURbTVcPOoRW
MgLrtZ+3t7ZMwd5zdxHR9P8rpCfwazNUv9MLCpW/ko15yxRQRLK9fDllDLKtGlMJ
rsyXIflzYkM5ANuCtCY+n0BVgL4xL/mX4vazgQL99rIkMB6HhEJFRl7g2yjojAAc
Ect6iopW+lh4pOfkHGn1LlUM4OXME0f9wvfmqLWM9IsT5JW59FTB2Rn8isPKrG5+
aRCTXPh+llOWrxQ/jbXv
=PKTJ
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages