Getting Qubes R3.0 working with Skylake's integrated GPU

[Eric Shelton]

I have worked out a series of steps for getting Skylake's integrated GPU working under Qubes R3.0.  Please note that two things are needed to get through the above steps: (1) a network adapter other than the Skylake integrated nic (since this is not supported by the kernel included in the ISO installer) so updates can be downloaded, and (2) another GPU besides the integrated GPU (seeing as that is what we're trying to get to work here).  This is easier to deal with on a desktop system than a notebook (although the notebook may have a supported wifi card, and some include a secondary GPU that will work under Qubes).  One way to get around this is to run the below steps on a different non-Skylake computer, and then put the hard drive in the Skylake system once all is complete.

1) Boot the R3.0 installer ISO.  Press tab, and add 'xsave=0' shortly after xen.gz.  Without this option for Xen, the system will simply reboot, rather than proceed with the installer.

2) Install Qubes R3.0

3) Once things are fully installed, right-click on dom0 in Qubes Manager, and choose Update.  Update all of the packages.

4) Edit /etc/yum.repos.d/qubes-dom0.repo  Duplicate the section [qubes-dom0-current-testing], and call the new one [qubes-dom0-r31-current-testing].  Replace '$releasever' with '3.1', and set gpgcheck to 0.  The resulting new section should look like this:

[qubes-dom0-r31-current-testing]

name = Qubes Dom0 Repository (r3.1 updates-testing)

baseurl = http://yum.qubes-os.org/r3.1/current-testing/dom0/fc20

enabled = 0

metadata_expire = 7d

gpgcheck = 0

gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-qubes-$releasever-primary

(it is possible this works without setting gpgcheck to 0 - I did not test that)

5) In a dom0 console, run 'sudo qubes-dom0-update --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'.  This will install the 4.1.13 Linux kernel, which provide drivers for the integrated GPU and Skylake ethernet.

6) Update the initramfs for 4.1.13 to have firmware blobs for Skylake:

mkdir dl

cd dl

wget https://01.org/sites/default/files/downloads/intelr-graphics-linux/sklgucver43.tar.bz2

wget https://01.org/sites/default/files/downloads/intelr-graphics-linux/skldmcver123.tar.bz2

tar xf sklgucver43.tar.bz2

tar xf skldmcver123.tar.bz2

cd ..

mkdir rfs

cd rfs

gunzip -c /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img | cpio -h

cd lib

mkdir firmware  (may exist)

mkdir firmware/i915

cd firmware/i915

cp ../../../../dl/skl_dmc_ver1_23/skl_dmc_ver1_23.bin ../../../../dl/skl_guc_ver4_3/skl_guc_ver4_3.bin .

ln -s skl_dmc_ver1_23.bin skl_dmc_ver1.bin

ln -s skl_guc_ver4_3.bin skl_guc_ver4.bin

cd ../../..

mv /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img.OLD

find . | cpio -H newc -o | gzip -9 > /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img

Note: the above may/will need to be done any time initramfs has been rebuilt.

7) Update grub.cfg (for example, 'sudo vi /boot/grub2/grub.cfg').  For the entry/entries that boot up the 4.1.13 kernel, revise the xen command line to include 'xsave=0', and the linux kernel command line to include 'i915.preliminary_hw_support=1'.  For example:

multiboot /xen-4.4.3.gz placeholder  console=none dom0_mem=min:1024M xsave=0 ${xen_rm_opts}

module /vmlinuz-4.1.13-6.pvops.qubes.x86_64 placeholder root=/dev/mapper/qubes_dom0-root ro rd.lvm.lv=qubes_dom0/root vconsole.font=latarcyrheb-sun16 rd.luks.uuid=luks-7dd366b7-49c4-4dd7-d5f2-6cddd796232b rd.lvm.lv=qubes_dom0/swap i915.preliminary_fw_support=1 rhgb quiet 

8) Install updated x11 driver for i915 (along with others)

sudo qubes-dom0-update --enablerepo=qubes-dom0-r31-current-testing xorg-x11-drivers xorg-x11-drv-ati xorg-x11-drv-dummy xorg-x11-drv-evdev xorg-x11-drv-fbdev xorg-x11-drv-intel xorg-x11-drv-modesetting xorg-x11-drv-nouveau xorg-x11-drv-openchrome xorg-x11-drv-qxl xorg-x11-drv-sisusb xorg-x11-drv-synaptics xorg-x11-drv-v4l xorg-x11-drv-vesa xorg-x11-drv-vmmouse xorg-x11-drv-vmware xorg-x11-drv-void xorg-x11-drv-voodoo xorg-x11-drv-wacom xorg-x11-server-Xorg xorg-x11-server-common

9) Go to System Tools->System Settings.  Under Power Management, disable Screen Energy Saving.  If you do not, the GPU will not be in a good state when it wakes from sleep.  This is a known bug with the version of the Skylake driver in the 4.1 kernels (apparently fixed in 4.2+).  Unfortunately, it also is a significant limitation for notebook users.  It is the best I have to offer for the time being.

That should do it.  Best of luck!

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Yes, it should. Anyway, qubes-dom0-updates tool checks the signatures
anyway, so setting gpgcheck=0 doesn't really disable signature checking.

> 5) In a dom0 console, run 'sudo qubes-dom0-update
> --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'. This will
> install the 4.1.13 Linux kernel, which provide drivers for the integrated
> GPU and Skylake ethernet.

You can do the same with unstable repo for r3.0 - 4.1 kernel is uploaded
also there.

Is it really needed to be done manually? I think dracut should handle
that as long as you have the firmware in /lib/firmware. I'll look into
upgrading firmware package for R3.1...

Thanks!

@Axon - maybe we should add it somewhere to the documentation? Where
would be the best place?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWYs35AAoJENuP0xzK19csol0H/05mgqCdiiPovvNPpqA3lJz6
3/mdBCfLdFTyYfKn0+Ly4IswTfROv7JiJN6wds1/cJSxx7EWppwalSUfJWTvrWVZ
sLaVyvReXG3qXfMq+8U5lcnOm2MEX2Ui84qsgy8ptA9lE7HcIHDs/OOPSwqVwqja
Y4H0ZwvL9FN9HFhgBONXfNjQWFDB2jUpre6sScJZ/oiMfDYtort6NKj93DjjEqvL
En50JkbTZ+5bksBRcDP2VZO3HKMwK5i2xtGqwneJulDoXHQ/W984z4dWYG5f42RR
erWlZADSW5AdtpAFxlJdL2IbACWBJoQELXZ1lN/1eVtOcQf/6O2NYGCDrSI0D/8=
=PRtC
-----END PGP SIGNATURE-----

[Eric Shelton]

Great. Mostly I didn't know if 3.1 had its own signing key, and if 3.0 would have the needed key.

>> 5) In a dom0 console, run 'sudo qubes-dom0-update
>> --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'. This will
>> install the 4.1.13 Linux kernel, which provide drivers for the integrated
>> GPU and Skylake ethernet.
>
> You can do the same with unstable repo for r3.0 - 4.1 kernel is uploaded
> also there.

That one is 4.1.12, whereas 3.1 had 4.1.13. Since I was already pulling in rpms from 3.1 for X11, I opted to get the newer kernel at the same time.

It might make sense to have the X11 rpms in 3.0 unstable too. Then there will be no need to modify the qubes-dom0.repo file to access the 3.1 stuff.

Another step would be to update /lib/firmware with the i915 firmware blobs. They are not in linux-firmware-20150520, but they are in the 20151030 version. I opened an issue for this a couple of days ago. Perhaps another rpm to include in unstable repo.

Also, if you run through the update process on a non-Skylake machine, or where the igpu is disabled (as some bioses will to use a secondary gpu as a primary), dracut will not know it needs to bake in the i915 stuff at all. In that case, I think you have to do a manual update (although perhaps there is a way to force dracut to respond and bake in the i915 stuff).

Note: even on a system where /lib/firmware/i915 is populated correctly, dracut only copies skl_dmc*, but not skl_guc*. My understanding is both are needed. I have not investigated, but instead I have just avoided the issue by doing the manual copy for now. Admittedly a bit on the hacky side of things.

Sure. I was pleased to find a way for people to do it with pre existing rpms.

The last thing that remains is getting something post 4.1 running, since that will fix the display sleep issue. However, 4.3 has some kind of issue where vchan communication or such breaks down after a little bit (sound stops working, windows go down, Qubes Manager cannot control a domain any more). I need to see if a 4.4 series kernel works any better, and if not bisect my way to the responsible kernel patch.

A final note: going through this display driver stuff has given me a much greater appreciation for the idea of pulling the gui into a separate domain in Qubes 4.0, if for no other reason than simplifying package management and updating for new display adapters, etc. This is a real pain to deal with inside of dom0, and issues of competing technical problems and solutions within dom0 can be a real headache that can be resolved by splitting its tasks and roles into separate domains.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Actually you can use --releasever=3.1 ;)

Check if module metadata have it correctly - modinfo should list
required firmware names. Maybe as still experimental in 4.1, it isn't
listed there?

Generally we try to ship in Qubes only kernels with long term support -
this is why we have 4.1 and not anything newer. But if there is any
particular fix in newer version, I think it is an option to 1) request
backport in upstream kernel to the stable branch, and if that fail
consider carrying the patch locally.

> A final note: going through this display driver stuff has given me a much greater appreciation for the idea of pulling the gui into a separate domain in Qubes 4.0, if for no other reason than simplifying package management and updating for new display adapters, etc. This is a real pain to deal with inside of dom0, and issues of competing technical problems and solutions within dom0 can be a real headache that can be resolved by splitting its tasks and roles into separate domains.

Yes, this is the main reason to move that stuff to separate domain.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWYuyBAAoJENuP0xzK19csXEgH/jS3pnw/ukJNgNjM5IfwwrQr
RIIch8/vbOpjhNVMoFeLaXjO64P/SNm4ZAym8KLImO53I7+33hUqzJ3DanoOVQoN
eLYWmXwMiMePmUwRSUuWMUXAjf07DO5m2uXxEoiL3Oyqv6E9baLptRDqrr6EB4Hp
EQ7XgW65w0QQuskNKSz6EPMkAopzX26mRtzwTi7mplnGFFSb9Lcc9Rq0nWPmbVkw
rQWbBmuBdGrt9oORkjN+4OioQFU3YJ8hFmY52VzAQH77XXeJ1xonQ9kumkAVvXVB
Geka/R3Rih6mxz0ls2ymbFI40ucWyOYdqFSfrEnPz3u7SZS2/o9N41vNt2q3+S4=
=yKGp
-----END PGP SIGNATURE-----

[Eric Shelton]

It sounds like I should focus on getting 4.1 to work. I have been involved in preliminary investigation of backporting the 4.3 i915 driver to 4.1.13. It is not as trivial as simply replacing drivers/gpu/drm/i915 in the kernel source tree - it is too dependent upon the underlying DRM code, which had various changes between 4.1 and 4.3 that are reflected in the i915 source. Luckily, the entire drivers/gpu/drm branch of the source tree is mostly self contained - it looks like it will be a drop in replacement from 4.3 to 4.1 (and some related include directories), with only a very few patches here and there to reflect some other changes made in the kernel between 4.1 and 4.3.

This is definitely not something we can get upstream to pull into 4.1, but would Qubes find this Franken-kernel approach acceptable? As I said, DRM as a whole seems to be a self contained unit of code, and it keeps Qubes up to date on graphics drivers. My proposed build process would be along the lines of downloading both the 4.1 and 4.3 source trees, replace drivers/gpu/drm and one or two other folders, and apply a few patches. I see this as a cleaner, clearer, and more maintainable approach than doing effectively the same thing via a mega patch.

Eric

[Eric Shelton]

On Saturday, December 5, 2015 at 9:26:08 AM UTC-5, Eric Shelton wrote:

The 4.2.6 kernel resolves the screen blanking bug, although Skylake is still considered preliminary, so the i915.preliminary_hw_support=1 option is still required.  4.2.6 does not demonstrate the instability seen with 4.3.  The only significant limitation I have with 4.2.6 is that my desktop's DisplayPort output does not work.  I don't know if this could present problems with some laptop display panel configurations; we will just have to wait and see (if someone is, the initial workaround would be to plug in a monitor into the affected laptop's HDMI or VGA port).

For the time being, I propose putting up RPMs for the 4.2.6 kernel to give Skylake users - desktop and notebook - something that will work for the integrated GPU, with the understanding that, as the repo name suggests, it is considered unstable.  It would be nice if an official install ISO were available that would just work on Skylake systems - 3.0 "Skylake edition", 3.0.1, or otherwise.  Skylake seems to have a good crop of laptops in terms of Qubes compatibility, and is likely what users are going to be buying going forward.  The current plan for 4.1.13 in R3.1-rc1 will not quite do the job due to the screen blanking bug.

At some point either a post-4.1 kernel will be needed, or an effort made to backport the 4.3 DRM code into 4.1 as discussed in my previous post, to end up with something you might designate as stable.

Eric

[John Anderton]

Thanks for your hard work Eric. I tried these steps for a Dell XPS 13 (9350), but I still cannot get Qubes (3.0 or 3.1rc1) to boot. I used a different (non-Skylake) PC to run the installer and then booted external HDDs, but the XPS will not boot Qubes. I'm almost certain the problem is due to graphics, so here is what I tried:

5) In a dom0 console, run 'sudo qubes-dom0-update --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'.

Instead of editing qubes-dom0.repo I executed     sudo qubes-dom0-update --releasever=3.1 kernel-4.1.13*

gunzip -c /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img | cpio -h

I replaced cpio -h with     cpio -i

cp ../../../../dl/skl_dmc_ver1_23/skl_dmc_ver1_23.bin ../../../../dl/skl_guc_ver4_3/skl_guc_ver4_3.bin .

I was confused here, but I assumed I should copy both binary files to the firmware/i915 folder.

i915.preliminary_fw_support=1

I used     i915.preliminary_hw_support=1

Everything else I executed exactly. Did I do it right? I verified the grub changes were respected during boot, and I also tried manually editing the boot commands too.

I can post error messages or specific logs if I know what to look for. Any other advice is greatly appreciated.

[John Doe]

On 12/14/2015 08:10 AM, John Anderton wrote:
> Thanks for your hard work Eric. I tried these steps for a Dell XPS 13
> (9350), but I still cannot get Qubes (3.0 or 3.1rc1) to boot. I used a
> different (non-Skylake) PC to run the installer and then booted external
> HDDs, but the XPS will not boot Qubes. I'm almost certain the problem is
> due to graphics, so here is what I tried:
>
> 5) In a dom0 console, run 'sudo qubes-dom0-update
>> --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'.
>>

> Instead of editing qubes-dom0.repo I executed *sudo qubes-dom0-update
> --releasever=3.1 kernel-4.1.13**

>
>> gunzip -c /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img | cpio -h
>>

> I replaced cpio -h with *cpio -i*
>
>> cp ../../../../dl/skl_dmc_ver1_23/skl_dmc_ver1_23.bin
>> ../../../../dl/skl_guc_ver4_3/skl_guc_ver4_3.bin .
>>

It is also very important to create 2 links in /lib/firmware

skl_dmc_ver1.bin -> /lib/firmware/i915/skl_dmc_ver1_23
skl_guc_ver4.bin -> /lib/firmware/i915/skl_guc_ver4_3.bin

> I was confused here, but I assumed I should copy both binary files to the firmware/i915
> folder.
>
>> i915.preliminary_fw_support=1
>>

> I used *i915.preliminary_hw_support=1*

>
> Everything else I executed exactly. Did I do it right? I verified the grub
> changes were respected during boot, and I also tried manually editing the
> boot commands too.
>
> I can post error messages or specific logs if I know what to look for. Any
> other advice is greatly appreciated.
>

What is the last thing you see during boot?

[John Anderton]

On Monday, December 14, 2015 at 6:54:53 AM UTC-5, securef33d wrote:

It is also very important to create 2 links in /lib/firmware

Yes, I verified the links were created correctly.

 What is the last thing you see during boot?

Trying 3.1rc1 with the boot arguments just shows a permanently black screen. Booting without any arguments shows the blue and white loading bars, but then I get a warning that the luks volume does not exist/was not found.

It generates /run/initramfs/rdspsreport.txt and enters emergency mode. Skimming through it shows the system timed out waiting for a device. It then says Dependency failed for Cryptography Setup for the luks volume, Encrypted Volumes, /sysroot, /Initrd Root File System, etc. I'm guessing it timed out waiting for the user to enter the disk passphrase, because it never prompts the user to enter it.

I had similar results booting the modified 3.0 also. It seems the graphics are still not working. What do you think?

[John Doe]

On 12/14/2015 08:10 AM, John Anderton wrote:

> Thanks for your hard work Eric. I tried these steps for a Dell XPS 13
> (9350), but I still cannot get Qubes (3.0 or 3.1rc1) to boot. I used a
> different (non-Skylake) PC to run the installer and then booted external
> HDDs, but the XPS will not boot Qubes. I'm almost certain the problem is
> due to graphics, so here is what I tried:
>
> 5) In a dom0 console, run 'sudo qubes-dom0-update
>> --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'.
>>

> Instead of editing qubes-dom0.repo I executed *sudo qubes-dom0-update
> --releasever=3.1 kernel-4.1.13**

>
>> gunzip -c /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img | cpio -h
>>

> I replaced cpio -h with *cpio -i*

>
>> cp ../../../../dl/skl_dmc_ver1_23/skl_dmc_ver1_23.bin
>> ../../../../dl/skl_guc_ver4_3/skl_guc_ver4_3.bin .
>>
> I was confused here, but I assumed I should copy both binary files to the firmware/i915
> folder.
>
>> i915.preliminary_fw_support=1
>>

> I used *i915.preliminary_hw_support=1*

>
> Everything else I executed exactly. Did I do it right? I verified the grub
> changes were respected during boot, and I also tried manually editing the
> boot commands too.
>
> I can post error messages or specific logs if I know what to look for. Any
> other advice is greatly appreciated.
>

try removing "rhgb" and boot in text mode, if you still have black
screen i'd double check the initramfs image.

I also had a problem when using 2 GPU on my workstation, i used to run
an old geforce 6800 for debugging and for some reason the intel gpu
started working only after i removed the nvidia. If your laptop has a
discrete card try disabling it in BIOS, if you can.

A serial console could be useful if your laptop has the port.

[Eric Shelton]

On Monday, December 14, 2015 at 8:45:12 PM UTC-5, securef33d wrote:

On 12/14/2015 08:10 AM, John Anderton wrote:
> Thanks for your hard work Eric. I tried these steps for a Dell XPS 13
> (9350), but I still cannot get Qubes (3.0 or 3.1rc1) to boot. I used a
> different (non-Skylake) PC to run the installer and then booted external
> HDDs, but the XPS will not boot Qubes. I'm almost certain the problem is
> due to graphics, so here is what I tried:
>
> 5) In a dom0 console, run 'sudo qubes-dom0-update
>> --enablerepo=qubes-dom0-r31-current-testing kernel-4.1.13*'.
>>
> Instead of editing qubes-dom0.repo I executed     *sudo qubes-dom0-update
> --releasever=3.1 kernel-4.1.13**
>
>> gunzip -c /boot/initramfs-4.1.13-6.pvops.qubes.x86_64.img | cpio -h
>>
> I replaced cpio -h with     *cpio -i*
>

Thank you for catching all of my lame typos, and figuring out the right commands.  The writeup for R3.0 was definitely more crude (scribbled down notes with pen & paper) than for R3.1.  You are correct on all of them.

At this point, I strongly recommend starting with the R3.1-rc1 installer for Skylake, since it basically has all of the minimum bits of software in place, and just needs a little push with the Linux command line.  I made another post (https://groups.google.com/forum/#!topic/qubes-users/ttStZXy_PPY) detailing how to get R3.1-rc1 going.  No typos on that one...

 

>> cp ../../../../dl/skl_dmc_ver1_23/skl_dmc_ver1_23.bin
>> ../../../../dl/skl_guc_ver4_3/skl_guc_ver4_3.bin .
>>
> I was confused here, but I assumed I should copy both binary files to the firmware/i915
> folder.

Graphics do seem to work without the binary blobs, just probably not 100% of whatever features.  My R3.1-rc1 instructions go through the whole installer without the binary blobs.  My experience, on a desktop system, is that I could use the Skylake iGPU with R3.1-rc1 after adding just the 'i915.preliinary_hw_support=1' to the Linux command line.

 

>
>> i915.preliminary_fw_support=1
>>
> I used     *i915.preliminary_hw_support=1*
>
> Everything else I executed exactly. Did I do it right? I verified the grub
> changes were respected during boot, and I also tried manually editing the
> boot commands too.
>
> I can post error messages or specific logs if I know what to look for. Any
> other advice is greatly appreciated.
>
try removing "rhgb" and boot in text mode, if you still have black
screen i'd double check the initramfs image.

I have found one benefit of leaving the 'rhgb' option in place is that if I see the blue and white bars, I quickly know graphics did not get up and running.

 

I also had a problem when using 2 GPU on my workstation, i used to run
an old geforce 6800 for debugging and for some reason the intel gpu
started working only after i removed the nvidia. If your laptop has a
discrete card try disabling it in BIOS, if you can.

In the event that you have a system with both a discrete GPU (AMD or NVIDIA) and iGPU, you can try a couple of things.  (1) Check your BIOS settings.  Can you disable one or the other GPU?  (2) Pass additional Linux command line options to prevent an AMD or NVIDIA GPU from interfering.  To do this, add 'rd.driver.blacklist=nouveau nouveau.modeset=0' or 'rd.driver.blacklist=radeon radeon.modeset=0', depending on if you have NVIDIA or AMD.  Perhaps even the following works: 'rd.driver.blacklist=nouveau,radeon nouveau.modeset=0 radeon.modeset=0'

Another problem I have run into is getting the iGPU to work with a DisplayPort adapter - I have yet to make it work on my system.  I believe some laptop models internally use a DP connection to the display panel.  A couple of additional Linux command line options I have not gotten around to seeing if they resolve this are 'i915.enable_ips=0' and 'i915.disable_power_well=0'.  You might get lucky with one or these if none of the above stuff pans out.  In case this is happening, I suggest also hooking up an external monitor (HDMI, DVI, or VGA), just to see if maybe that works.

Best of luck.  I did this stuff with a desktop system, which gave me the luxury of falling back to a secondary GPU and still get a console, etc.  Unfortunately, the notebook experience may be more all or nothing.  Come back and let us know how it goes.  If it doesn't work, we will try to keep coming up with ideas.  Your efforts will help the many future Skylake notebook users coming after you.

Eric