boot fails with newer kernels

[John Doe]

Hi,
I'm running qubes 3.0 and I can only boot with kernel 3.12.40-2. It
fails with both 3.18* and 4.1*. I tried also with the latest 4.3.0-rc3,
from git with no success. I get a black screen and reboot right after
the string "(XEN) Dom0 has maximum 8 VCPUs", unfortunately i have have
no serial adapter to use for console debugging (is there a way to do it
without one?).

My hardware is a brand new intel skylake i7-6700k with a z170 chipset
and I really need a newer kernel to make the integrated GPU work; right
now I'm using an old geforce 6800.

Can you help me?

Thank you,
John.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

You can add "noreboot=true" option to the Xen (not the kernel, so just
after xen.gz) - this will hopefully give you a chance to read the error
message. Adding "modeset=0" to kernel cmdline also may help.

I have some early test images of Qubes R3.1-pre-pre-alpha (links in
github issue #794) with kernel 4.1.9 and somehow updated X drivers, but
if you've already tried 4.1 kernel, this probably wouldn't help.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWED1bAAoJENuP0xzK19csBV4H/RRCMXDUJmxRBTdn0jtl2CKC
quihaPQKsIf7efHdXzzasHbIOIuwWPuxUCB9a6JT7kMKWyYbhDmnVvSS6OmZJ3Pl
HAzYPwEJJAEbKExpAaLixkmOlcZceLM06icFYNTsFLy7LHtNJzyXPP9PJO6ZTu/i
XCnTAZrQwO2TI/Pn/+fnOOfowd0VuIbHZvYQJLkLGDIGZyERPQf2Gk16z+Z2mIzv
jEQqtICWyG+ODyXiaRUuengIbycUgMqGUIMNllL6N/89wG/k9MHqlhpjnF9OX2Ab
vcbMG9CGfwluYv8+1ojrdkJ+ZXipxztMMMjnZv5rDe393pOtZYpf6LhR0ko9uUA=
=5rZJ
-----END PGP SIGNATURE-----

[John Doe]

On 03/10/2015 22:40, Marek Marczykowski-Górecki wrote:
> On Sat, Oct 03, 2015 at 05:52:17PM +0200, John Doe wrote:
>> Hi,
>> I'm running qubes 3.0 and I can only boot with kernel 3.12.40-2. It
>> fails with both 3.18* and 4.1*. I tried also with the latest 4.3.0-rc3,
>> from git with no success. I get a black screen and reboot right after
>> the string "(XEN) Dom0 has maximum 8 VCPUs", unfortunately i have have
>> no serial adapter to use for console debugging (is there a way to do it
>> without one?).
>
>> My hardware is a brand new intel skylake i7-6700k with a z170 chipset
>> and I really need a newer kernel to make the integrated GPU work; right
>> now I'm using an old geforce 6800.
>
>> Can you help me?
>
> You can add "noreboot=true" option to the Xen (not the kernel, so just
> after xen.gz) - this will hopefully give you a chance to read the error
> message. Adding "modeset=0" to kernel cmdline also may help.
>
> I have some early test images of Qubes R3.1-pre-pre-alpha (links in
> github issue #794) with kernel 4.1.9 and somehow updated X drivers, but
> if you've already tried 4.1 kernel, this probably wouldn't help.
>
>

I had no luck with those flags, but with a serial adapter i got some
verbosity. As you can see in the attachments, Xen hangs right after the
"about to get started" string without any apparent error message.
I also tried with kernel 4.1.9-6 from the unstable repo, the only
noticeable difference seems to be:

(XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000081 from
0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000082 from
0xffff82d0802b3000 to 0xffffffff81772ed0.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000083 from
0xffff82d0802b3080 to 0xffffffff81775870.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000174 from
0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000175 from
0xffff82d0802b7fc0 to 0x0000000000000000.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000176 from
0xffff82d08021aff0 to 0xffffffff81775640.
(XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000084 from
0x0000000000034700 to 0x0000000000047700.

Any clue?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Oct 10, 2015 at 01:51:02AM +0200, John Doe wrote:
> On 03/10/2015 22:40, Marek Marczykowski-Górecki wrote:
> > On Sat, Oct 03, 2015 at 05:52:17PM +0200, John Doe wrote:
> >> Hi,
> >> I'm running qubes 3.0 and I can only boot with kernel 3.12.40-2. It
> >> fails with both 3.18* and 4.1*. I tried also with the latest 4.3.0-rc3,
> >> from git with no success. I get a black screen and reboot right after
> >> the string "(XEN) Dom0 has maximum 8 VCPUs", unfortunately i have have
> >> no serial adapter to use for console debugging (is there a way to do it
> >> without one?).
> >
> >> My hardware is a brand new intel skylake i7-6700k with a z170 chipset
> >> and I really need a newer kernel to make the integrated GPU work; right
> >> now I'm using an old geforce 6800.
> >
> >> Can you help me?
> >
> > You can add "noreboot=true" option to the Xen (not the kernel, so just
> > after xen.gz) - this will hopefully give you a chance to read the error
> > message. Adding "modeset=0" to kernel cmdline also may help.
> >
> > I have some early test images of Qubes R3.1-pre-pre-alpha (links in
> > github issue #794) with kernel 4.1.9 and somehow updated X drivers, but
> > if you've already tried 4.1 kernel, this probably wouldn't help.
> >
> >
>
> I had no luck with those flags, but with a serial adapter i got some
> verbosity.

Try adding "console=hvc0 earlyprintk=xen" to linux cmdline to get also
its messages.

> As you can see in the attachments, Xen hangs right after the
> "about to get started" string without any apparent error message.
> I also tried with kernel 4.1.9-6 from the unstable repo, the only
> noticeable difference seems to be:
>
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000081 from
> 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000082 from
> 0xffff82d0802b3000 to 0xffffffff81772ed0.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000083 from
> 0xffff82d0802b3080 to 0xffffffff81775870.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000174 from
> 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000175 from
> 0xffff82d0802b7fc0 to 0x0000000000000000.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 0000000000000176 from
> 0xffff82d08021aff0 to 0xffffffff81775640.
> (XEN) traps.c:2524:d0 Domain attempted WRMSR 00000000c0000084 from
> 0x0000000000034700 to 0x0000000000047700.

The same is also on successfully running system, so probably not
related.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWGFdSAAoJENuP0xzK19csqa0H/0PM1D/N7hX/i0N+RmOFzzYz
7Q/L7bX2SZYfHoELN9oH30BdUMESdxXZgwx7b+pSZiLMcXU3IrksACCOvdcKXBJD
9V6fLJyNNDLhJY4He2z+zZ8C3wqu2oWA/Wz/wU0avE6ulRNkrj2f1oFDryJruv3q
Vcet1BjzIGXArWJGFrjFuyUewPZb22ieeOR639zShWjHuXjkhuOb0yp35vuu64AT
zdrwyS95KvhGFSkFj96SApvd7Y94cRLiFqbZIdprPZsa+aeRS16qpveIbO63Pz2c
LmHFHB59hCZCTcb9UBFu32FRDRFvBd+EoMEq/dbPvTlbwYrElbW9DXx7YHWSYuw=
=zWsD
-----END PGP SIGNATURE-----

[John Doe]

Here it is!
general protection fault: 0000 [#1] SMP
(...)
[ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!

I attached the stack trace.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> Here it is!
> general protection fault: 0000 [#1] SMP
> (...)
> [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>
> I attached the stack trace.

(...)

> [ 0.000000] Call Trace:
> [ 0.000000] [<ffffffff810237b0>] xsave_init+0x30/0x40

Try adding xsave=0 to xen cmdline.

> [ 0.000000] [<ffffffff8102201e>] fpu_init+0x9e/0xb0
> [ 0.000000] [<ffffffff8102a1f4>] cpu_init+0x2f4/0x3e0
> [ 0.000000] [<ffffffff81d54f05>] trap_init+0x2a2/0x312
> [ 0.000000] [<ffffffff81d4def8>] start_kernel+0x25a/0x4bf
> [ 0.000000] [<ffffffff81d4da8e>] ? set_init_arg+0x55/0x55
> [ 0.000000] [<ffffffff81d4d5ee>] x86_64_start_reservations+0x2a/0x2c
> [ 0.000000] [<ffffffff81d50a38>] xen_start_kernel+0x518/0x524
> [ 0.000000] Code: 48 89 35 df 8a 17 00 48 39 f8 74 0f 65 48 89 3d 3a e0 2b 7e ff 14 25 f8 c4 c2 81 48 8b 05 c4 8a 17 00 31 c9 48 89 c2 48 c1 ea 20 <0f> 01 d1 48 8b 05 e5 aa fc ff a8 08 7
> [ 0.000000] RIP [<ffffffff81d58fad>] xstate_enable_boot_cpu+0xde/0x288
> [ 0.000000] RSP <ffffffff81c03de8>
> [ 0.000000] ---[ end trace 1f0c5608e200b15b ]---

> [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!

> (XEN) Domain 0 crashed: rebooting machine in 5 seconds.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWGGEcAAoJENuP0xzK19cs3coIAIBvTVofuuKsUdFqJ3hc+sLh
EbYZrnE2c2Jb/+9dgt8E9UkMUA0Stjq1e+gFa3cPDpodYVpOIHempVABuZuZ5I3Q
N/tFKJY9/yPabvaDNwKnuyI1ew/2PGgsm8+sqXVE+wiAysyH6WCeEW1Yh+cxRrx+
3PpihUfRWamoJOB7bof/WbRJWdxH2qcrMpingNtHISGWl+Ud+PvWk2bv6qY0mj+d
BLRlgu5j1VABBKKe5TQulE/b0imTt6ll2w8gmlJyQ0l/Uup5TJ3Oy72lOdT9TWiN
2FJrqO1N/siCiBSo0sGnSlm+KU1IDThcUGHceYGoWHRiunYTxXtDCDrB0BA+h+I=
=Pu5v
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

And it's probably a good idea to forward this stack trace to xen-devel
folks, with cc: x...@kernel.org.

> > [ 0.000000] [<ffffffff8102201e>] fpu_init+0x9e/0xb0
> > [ 0.000000] [<ffffffff8102a1f4>] cpu_init+0x2f4/0x3e0
> > [ 0.000000] [<ffffffff81d54f05>] trap_init+0x2a2/0x312
> > [ 0.000000] [<ffffffff81d4def8>] start_kernel+0x25a/0x4bf
> > [ 0.000000] [<ffffffff81d4da8e>] ? set_init_arg+0x55/0x55
> > [ 0.000000] [<ffffffff81d4d5ee>] x86_64_start_reservations+0x2a/0x2c
> > [ 0.000000] [<ffffffff81d50a38>] xen_start_kernel+0x518/0x524
> > [ 0.000000] Code: 48 89 35 df 8a 17 00 48 39 f8 74 0f 65 48 89 3d 3a e0 2b 7e ff 14 25 f8 c4 c2 81 48 8b 05 c4 8a 17 00 31 c9 48 89 c2 48 c1 ea 20 <0f> 01 d1 48 8b 05 e5 aa fc ff a8 08 7
> > [ 0.000000] RIP [<ffffffff81d58fad>] xstate_enable_boot_cpu+0xde/0x288
> > [ 0.000000] RSP <ffffffff81c03de8>
> > [ 0.000000] ---[ end trace 1f0c5608e200b15b ]---
> > [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
> > (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
>
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWGGJ4AAoJENuP0xzK19cslLEIAJjW7D/OuAI0EpY7kFSecyJ7
AToss9d6B8M4VVi5/JTo8j+6t4ydaC/VhRPm4Fz8dadDaTnvj98vKOemFKzhNHdf
SjkUn7xkdYIBycn6aaqZvKNDJoRh3YovdJyUVdMD1QZ1ntzMi+dUzjKGh8w12q/J
aKwLY/CxzLLeX8JLJm//LW1mIfj0y3L/KqrC6MYbLIpDbP3uEDM2PA1eL3RGcAvJ
Jafek/mLaDLrqJ5LNumm+AtbNc7Niemf6aaEz6JGzuSCvUZIZEIwQsMOkBZ+Fmx9
Zchpa0RoDPV0YhBOamlFYSD8M+IAFzCqbVZhbf/Dmnn7WqyXV9x2YiV3E6bQbvA=
=WtO6
-----END PGP SIGNATURE-----

[John Doe]

It works! thank you so much :)

J.

[Eric Shelton]

There is no need to go through xen-devel to sort out the crashes discussed above, as there is already a commit out there (e8121c54) that fixes the problem, and eliminates the need for the xsave=0 option.  I have attached the patch for vmm-xen, and I can confirm it worked for me on a Skylake 6600K (successfully booted 3.18.17-7 without the xsave=0 option).

I will try spinning up an ISO including the patch, to see if it resolves getting the problem with getting the install ISO to work on Skylake machines.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Does that patch work on Xen 4.4, or 4.6?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

Version: GnuPG v2

iQEcBAEBCAAGBQJWV0vlAAoJENuP0xzK19csCCwH/i1m/mWiEMaeAoBfPcfqLSRr
x22VcHl6q3pvtCPuqHOlp+3IFo3ca5M/qCPeOSnGHNIXvlN/S6SghlnNya5+1lnD
YeWpeG2EdPiB3cU8+2jUm76r0NhLdy7EX71S3GzPJNyEKvexZ1k+lDgjqR63SI2W
/8fdUkTnqlpvOW5maWvOQIsFRZe2Vi1qmS3AddRGJ7aMooUyzAD1HfWd8uBTQbMO
9E1UUN0VXC2973Lw/1Cm+rsvPDyHxZa+1yDvo8npFwUMlpPkGYC48GB17GSZTMz6
VqwV9r6f6wJH8zI8bF6ZMi8B5NxTruwPRnvANNMmqVBV5Ekad/PJUgDznuRwvo8=
=GdAv
-----END PGP SIGNATURE-----

[Eric Shelton]

4.4 (built against 4.4.3-9, to be precise).  The commit should already be in 4.6.

By the way, can you help with a problem building the ISO?  I cannot even get through make vmm-xen without the build failing at the end of vmm-xen, where it is packaging the RPMs (specifically, the vmm-xen-xm RPMs - the vmm-xen RPMs build fine).  The error message is:

Makefile.generic:19: *** Building packages for jessie not supported by any of configured plugins.  Stop.
Makefile:193: recipe for target 'vmm-xen-vm' failed
make: *** [vmm-xen-vm] Error 1

This is brand new Qubes build appvm I set up.  I am using the qubes-os-r3.0.conf to do the build.  The only deviation I needed from the instructions at https://www.qubes-os.org/doc/qubes-r3-building/ was to add python-sh to the 'sudo yum install' command at the beginning.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

If you want to build Debian packages, you need to uncomment
builder-debian plugin at the end. But probably you don't want to, so
simply remove jessie and wheezy from DISTS_VM (at the beginning).

> This is brand new Qubes build appvm I set up. I am using the
> qubes-os-r3.0.conf to do the build. The only deviation I needed from the
> instructions at https://www.qubes-os.org/doc/qubes-r3-building/ was to add
> python-sh to the 'sudo yum install' command at the beginning.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWV1PSAAoJENuP0xzK19cs7icH/A7LLBZCje0d20bXDxWDVSJR
I4VcweHhHEwn+YEkqQf/PHeYDSRPAYqnMMCmrLhwnnlwNlEUvGRFRB1/h466K+2y
+fC6djfYhn13AvnleRzW0+1qH0zZm5qFFrBLgbwrwtKpa9jhZ19PxkF93ZrQLUXx
uQMsbRqxez0P3xthQnocss/2P1ecwg6WZ9Rmz4P4y6LsnEyMePJ+okDVkrOa5jXT
rsGeFUl9NKsotA3ULT9BUV4dkjueEPM9K/qgt/VRPcIflDOclajv9QSqO2owld+e
/lgpnrVsKGhrY01jek0opzy8DuoZz2ODNkLZAMiGnxwaE9hi9I8WYgYuXXFqiQU=
=Va1x
-----END PGP SIGNATURE-----

[John Doe]

Wow thanks dude, i will try to build a patched xen asap!

If you manage to make the integrated GPU or the vt-d to work, please
report back.

J.

[Eric Shelton]

I am slowly working my way there - might need the 4.3 kernel for graphics.  Also, Xen is reporting a few unusual things about Vt-d and interrupt remapping that I need to investigate.  First, I need to see if it looks any different on Xen 4.6.

Marek - since you mentioned 4.6 above, does that mean there is a development version of 4.6 for Qubes that I might be able to try out?

 Eric

[marmarek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Depending on what you mean by "development version". Ready ISO image -
not yet. Commits in repo - yes. FYI just today we've decided to go with
4.6 in Qubes R3.1:
https://github.com/QubesOS/qubes-issues/issues/1361

So RPM packages will be available really soon.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

Version: GnuPG v2

iQEcBAEBCAAGBQJWV4YEAAoJENuP0xzK19csYfwH/3KsVGhEJ4MJtYz0dnjwKS7U
L3F3uOCN5oarN0BUd5P+74SXH3zAAK1EQvC1s2ZlHrkyFtMZIt4m+ubUfRkAB6iC
QEOgx8WF5awJrqibpSNopPyGCYWl1W1h/evoi5vdmqiom9oc66qnnkevNeazK6iP
hmVDYKNYfUufUGo+cR8RSJyw9z12mCBO1Q9sT1Mtsy4H4XRG8shpLj+WCXVFhcWZ
IWQ1GJj0RxARNAxYnLUWjSVe/y/yapgTOpxQRlIyBKGVlbTyNl1OkQiEuwwiZMeW
Y1VRJrVv67dgHC5njHecAOm1MgyO7SAQ2r0IKUeirMkc4gUVdvErvREaHsbGdSI=
=lUN/
-----END PGP SIGNATURE-----

[Eric Shelton]

It looks like I overlooked the devel/xen-4.6 branch.  I will give it a try.

I rebuilt the install ISO with the above patch, and it successfully ran through the installer on my Skylake machine (which I could not do with the official 3.0 installer ISO - the Skylake system would just reboot).  Given that we have already started seeing people on these lists writing about problems installing on Skylake systems, I propose releasing a 3.0.1 ISO sooner rather than later.

However, I did run into a problem on firstboot after the installer ran - the VM creation stage failed with the following error messages:

"firstboot failure!

Creating default NetworkVM failed:

['/usr/bin/qvm-pci', '-a', 'sys-net', '06:00.0'] failed:

A VM with the name 'sys-net' does not exist in the system.

Creating default DisposableVM failed:

['su', '-c', '/usr/bin/qvm-create-default-dvm --default-template --default-script', 'username'] failed:

No default template ?

Creating handy AppVMs failed:

['su', '-c', '/usr/bin/qvm-create untrusted --label red', '-', 'username'] failed:

No default TemplateVM defined!"

06:00.0 is a second network adapter.  This error does not occur when I run the R3.0 official release installer on a non-Skylake machine, and run firstboot on the Skylake machine.  I assume this happened because of some changes made in the last couple of months are getting pulled in by the qubes-os-r3.0.conf example config.  I ran into problems like this with trying to build R2.0 ISOs well into R3.0 development, where I was unable to build an R2.0 installer.  What should builder.conf look like if I want to respin the official R3.0 installer?  For example, do all of the BRACH_xxx lines in qubes-os-r3.0.conf need to say "release3.0" instead of "master"?

Eric

[marmarek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I'm working on R3.1-rc1, just some final testing. But the we can
consider some R3.0 update.

> However, I did run into a problem on firstboot after the installer ran -
> the VM creation stage failed with the following error messages:
>
> "firstboot failure!
>
> Creating default NetworkVM failed:
> ['/usr/bin/qvm-pci', '-a', 'sys-net', '06:00.0'] failed:
> A VM with the name 'sys-net' does not exist in the system.
>
> Creating default DisposableVM failed:
> ['su', '-c', '/usr/bin/qvm-create-default-dvm --default-template
> --default-script', 'username'] failed:
> No default template ?
>
> Creating handy AppVMs failed:
> ['su', '-c', '/usr/bin/qvm-create untrusted --label red', '-', 'username']
> failed:
> No default TemplateVM defined!"

That's probably true - you haven't included default template on the ISO.
Unfortunately tool used for building Fedora derived installation image
(pungi) doesn't treat missing package as fatal error...

> 06:00.0 is a second network adapter. This error does not occur when I run
> the R3.0 official release installer on a non-Skylake machine, and run
> firstboot on the Skylake machine. I assume this happened because of some
> changes made in the last couple of months are getting pulled in by the
> qubes-os-r3.0.conf example config. I ran into problems like this with
> trying to build R2.0 ISOs well into R3.0 development, where I was unable to
> build an R2.0 installer. What should builder.conf look like if I want to
> respin the official R3.0 installer? For example, do all of the BRACH_xxx
> lines in qubes-os-r3.0.conf need to say "release3.0" instead of "master"?

You can use "R3.0" tag name instead of branch there. So simply put
BRANCH=R3.0 at the top and remove all others BRANCH_xxx settings.

The content of qubes-os-r3.0.conf should give you R3.0, with all
subsequent updates included. Yes, that bunch of BRACH_xxx=master should
be there.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWWIQKAAoJENuP0xzK19csnMUH+wQuJv5yVFHr91NJlzoo5oTL
k2n6FprQckk1hDvSUaUix016Wbx123DPDQR5/9Yz2P4NtgTXZL/9AvdpmTpoV8vu
2Q5HEXeVGdFVEPz4ma9w+cX8BYMp3/MHQxDlc1Jri7L6HAveeij5oikmYs6zLhEM
ICkHINLbEluh+TVSpKEMExh/vXn3bHfZqXpUuwnoazxE8Qf4RTAk2DH7jUrmuFib
xMJ9RauGAdpeIRfEv+glH/2Ju7xMrIXSbSH3g2+77StSGNxRsYANzuMMtNRXQKMB
A7eayxJdsh9KLJIS804tgQ6qa1pWgsl8Pyr2aUfDEWzoMaziqLUpT14u6QgXyIU=
=GHHe
-----END PGP SIGNATURE-----

[Eric Shelton]

Just a heads up that I ran into trouble building xen 4.6 from a clean build environment.  'make vmm-xen' errors out with:

+ autoreconf
/var/tmp/rpm-tmp.WExsIW: line 35: autoreconf: command not found

I added autoconf to qubes-src/builder-fedora/build-pkgs-base.list, which then resulted in another error:

+ autoreconf
Can't exec "aclocal": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 326.
autoreconf: failed to run aclocal: No such file or directory

However, once both autoconf and automake were added to qubes-src/builder-fedora/build-pkgs-base.list, the build proceeded as normal.  Perhaps just automake would be enough, if it depends on autoconf.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Fix is already in repo (missing BuildRequires: header).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWWuwiAAoJENuP0xzK19cswxEH/0OEKSM/Uhderd08O24NsQai
vsROE2wCybInLwNpJzA38h++17govd027wnckcVMDh+rzHzfP35ZeEKVugAAlnuK
6Xm3hf4cd5KdGeLYtF4gKcfEjlxN7LUHNCN05Xm/8HXFDhh5NmqrK3uYIDEQZhcM
rIx1+l++Xh18L7vo53/BlHaRkWGDC9a5A6E2+jqo4eZQKhcNf1ti0jJRWveT1A77
SInRh1MkiPkdk/o7ag4j9YOgZjruHtipZ6N9V6Nq+LTIYWCxwH/qJufSu5nC4Jei
CUXUm80ecIKwQ/wk+haeGnrpJDdgEDqOH82own6g3vFqtTxWPpLfYKYqecfsdVI=
=2iJt
-----END PGP SIGNATURE-----

[marmarek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Nov 26, 2015 at 02:14:26PM -0800, Eric Shelton wrote:

FYI Xen 4.6 packages are already uploaded to R3.1 repository:
http://yum.qubes-os.org/r3.1/current-testing/dom0/fc20/rpm/

You'll need also to update libvirt and gui-daemon. Or simply wait for
R3.1-rc1 (which is currently being built).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

Version: GnuPG v2

iQEcBAEBCAAGBQJWW32WAAoJENuP0xzK19csIgoH/Remc8HrjIsVoReIHuLcDoaA
G/M+Y0IboYQAayO2sHPNUxE0AFml4sW73kCoqDPuMHNX6xXaACgbSkpy15JIE3UA
94lHFIEy/Y2lQ2O5L2G+5N6xUaOH6q+r6IiCeXTJ7KnCGjPheJ9hp1UB0a45Ny75
rF2ZvFoxjmItQxqJhsDGjyg+adj592tBmxdUWNWLzSCeYuS7ZOBCO0OLMeFSTbC3
QmtjiniXuguEm9R4o213x8BmfPj+u7ygyStWqGKJA1oQkxnn8X6bJqKsvc/cMZ13
5HgqzPqcn8cfa0SR43aJP/GtBkgF3OtfrMwYolM8LFK9XHfvs4+gDh6FW+CbK2o=
=OrfZ
-----END PGP SIGNATURE-----

[Eric Shelton]

OK.  I have now gotten both Skylake video and audio working on Qubes.  Four things are needed to make it work:

1) Xen 4.4 with xsave patch, or Xen 4.6 (otherwise, Xen won't boot on Skylake with more recent kernels)

2) Linux 4.3 kernel (first release with full Skylake video support)

3) Skylake firmware blobs.  Skylake is the first to require firmware blobs for i915.  The most recent version is available at https://01.org/linuxgraphics/intel-linux-graphics-firmwares  However, Fedora 23's linux-firmware package has a close enough version - look under /lib/firmware/i915)

4) Updated X server, to provide recent enough i915 support for Skylake.  If someone can tell me how to do this update (for example, 'yum install xxx') on dom0 on R3.0, it would be greatly appreciated!

It should be possible to work out a set of updates to R3.0 that include the above features.

Below is what I did.  However, I seem to be running into stability issues in this pre-R3.1-rc1 environment that has been cobbled together (appvm windows keep crashing, requiring me to kill the appvm and restart it).

1) Build the Qubes installer, using example-configs/qubes-os-master.conf

This effectively gives you R3.1-rc1, I suppose.  For me, it was the easiest path to having Xen 4.6, updated X (needed to support the Skylake i915 driver), and something closer to the linux 4.3 kernel.  This step isn't strictly necessary, it was just the most convenient way to get various things in place.  In fact, I am having some stability issues with this build - so you probably don't want to follow this path yet.

2) Install Qubes using the above ISO.

3) Build the Linux 4.3 kernel RPMs.

4.3 is pretty much needed to get Skylake graphics working.  I used 'make oldconfig' in the Linux 4.3 source directory to generate a new config file for building the kernel (see attached 'config' file). 

To do this, in qubes-src/linux-kernel, I downloaded linux-4.3.tar.xz and linux-4.3.tar.sign from kernel.org, then renamed each to linux-4.3.0.tar.xz and linux-4.3.0.tar.sign  I also edited version to contain '4.3.0' and release to 1.  Because of a mismatch between 4.3 and 4.3.0 cropping up in the build process, I modified kernel.spec as follows:

--- kernel.spec.orig    2015-11-30 21:46:59.140360578 -0500
+++ kernel.spec    2015-11-30 21:48:40.456360578 -0500
@@ -101,6 +101,7 @@
 
 mkdir -p %kernel_build_dir
 
+ln -s linux-4.3 linux-4.3.0
 cd linux-%version
 
 if [ -r %_sourcedir/series-%{version}.conf ]; then

The above patch won't be required for 4.3.1 and onward.

I additionally changed patches.rpmify/makefile-after_link.patch and patches.xen/pvops-blkfront-removable-flag.patch.  The updated copies are attached.

4) Install the resulting RPMs in dom0.

5) Add the Skylake firmware blobs to initramfs

sudo bash
mkdir tmp
cd tmp
gunzip -c /boot/inintramfs-4.3.0-1.pvops.qubes.x86_64.img | cpio -
cd usr/lib/firmware
mkdir i915
cd i915
[copy in skl_dmc_ver1_23.bin and skl_guc_ver4_3.bin]
ln -s skl_dmc_ver1_23.bin skl_dmc_ver1.bin
ln -s skl_guc_ver4_3.bin skl_guc_ver4.bin
cd ../../../..
mv /boot/inintramfs-4.3.0-1.pvops.qubes.x86_64.img /boot/inintramfs-4.3.0-1.pvops.qubes.x86_64.img.OLD
find . | cpio -H newc -o gzip -9 > /boot/inintramfs-4.3.0-1.pvops.qubes.x86_64.img
cd ..
rm -rf tmp
exit

As I noted above, I am running into serious stability issues with the above setup.  It is possible this is caused by the 4.3 kernel.  If so, it might be possible to use the 4.1.13 kernel with 'i915.preliminary_hw_support=1' added to the kernel command line in grub.cfg.

Best of luck.  I hope to turn this around into a more formal contribution soon - [erhaps something else that makes sense to work into a possible 3.0.1 release.

Eric

[Eric Shelton]

I can confirm that the 4.1.13 kernel will work with the 'i915.preliminary_hw_support=1' command line option - video and audio work.  It also seems to be much more stable with Qubes than 4.3.  So my recommendation is to go with 4.1.13.

Marek:

If you add 'i915.preliminary_hw_support=1' to the linux command line options for 4.1.13 in grub.cfg, and get the Skylake firmware blobs added to initramfs, I think R3.1-rc1 will install and run just fine on Skylake systems.

Eric

[John Doe]

Thank you for your feedback Eric, you deserve a medal.
I will try building 3.1-rc1 myself and follow your well-detailed guide!
Do you have any limitation when using 4.1.13 with
i915.preliminary_hw_support?

J.

[Eric Shelton]

None that I have noticed, but I haven't put through any serious tests.

Keep in mind that you may not need to spin R3.1 (which takes a while), if you can just get the i915 driver updated on dom0. It sounds like the packages are out there, I just don't know what they are. If you go the R3.0 route, the instructions for updating initramfs apply for 4.1.13 pretty much the same.

Eric

[Eric Shelton]

On Tuesday, December 1, 2015 at 2:23:52 PM UTC-5, Eric Shelton wrote:

Encountered my first graphics issue - things went weird with the pointer and the screen after unlocking the screen (after display blanking had kicked in).  There are some settings in KDE that affect how the display is rendered - maybe one of them will mitigate this issue.

Eric