Qubes 3.2 only booting with onboard GPU ?? (ASUS KGPE-D16 + AMD Radeon 6450)

[qube...@openmailbox.org]

Hi everyone,

I recently got an ASUS KGPE-D16 mainboard, since I read over here, that it would be fully compatible with Qubes OS.
Installation went fine, the only problem is the low screen resolution (VGA). Unfortunately, I can't get any external graphic cards to work.

As soon as I connect my graphic card to the mainboard (AMD Radeon 6450), the boot process halts shortly after I enter my LUKS password.
After entering my password, the boot screen switches back from graphics into text mode (black/white) with the following error message:

ERST: Failed to get Error Log Address Range.
BERT: Can't request iomem region <00000000ccffa....................................>
_

And that's it.. Nothing happening from that point on. I can only reboot / force shutdown.

Overall, what I find weird as well, is that it takes much longer for the screen to get a video signal when the external GPU is plugged in. When I use the onboard VGA graphics, the screen reacts / shows the boot screen already a few seconds after booting compared to 30-40 seconds with external GPU ?? Any idea how to fix that as well ?

Important notes:

I upgraded the kernel in dom0 to the latest version. Although I didn't try any GPU with the old Kernel (4.4), thus I don't know if the issues are also present with the older kernel. Does anyone know ? I thought upgrading kernel would always increase hardware support.

I also tried to boot with a NVIDIA 980 GTX graphics card, I had exactly the same boot issue. So I guess it's nothing related to any NVIDIA/AMD driver issues.

Motherboard: ASUS KGPE-D16
GPU: AMD Radeon 6450
dom0 Kernel: 4.9.56-21

Thanks you for help!

So far I tried to use 2 different graphics cards, first a NVIDIA 980 GTX and then an AMD Radeon 6450.
The Radeon card is supposedly fully supported according to the Qubes and Xen Wiki's.

The problem is exactly the same for both cards, so I don't really believe it is an AMD/NVIDIA driver issue. But maybe I am wrong ?

[Tai...@gmx.com]

I take it you are using coreboot? if you aren't you should, otherwise
you need to change the "primary video" option in the BIOS - which will
solve your issue.

[Marek Jenkins]

> I take it you are using coreboot? if you aren't you should, otherwise
> you need to change the "primary video" option in the BIOS - which will
> solve your issue.

Hi Tai,

no I tried to use Coreboot with default settings (Seabios), compiling and flashing the .rom on the chip was easy. But with Coreboot BIOS the screen stayed black. What made me change back to the default BIOS was, is the fact that the CPU fans got really loud (full-speed) shortly after booting. I removed some of the RAM which at least solved the issue with the loud fans, so I assumed my RAM (Kingston ValueRam) might be incompatible with Coreboot ?

Do you have an idea why coreboot boot failed for me ?

Thanks a lot for the hint with VGA boot priority!
I have now found the VGA settings in the BIOS and did the following:
- Enable IOMMU
- Set GPU Boot Priority to "PCIe VGA" instead of "Onboard VGA"

This only partly solves my issue. Now, I can at least boot without issues when the PCIe graphics card is connected to the PCI slot. But I can still only finish the boot process, when I use the VGA Onboard graphics because I still get the same error message as before with the PCIe card:

ERST: Failed to get Error Log Address Range.
BERT: Can't request iomem region <00000000ccffa....................................>

Nevertheless, I still have a working signal on my VGA onboard graphics card and can simply continue with the boot process there by switching the connection from HDMI to VGA (hope you get what I mean).

After booting completely, there is still no signal on the PCIe VGA card.

Kind regards!

[Marek Jenkins]

Could using Qubes 4.0rc2 also solve my problem ?

[Tai...@gmx.com]

On 11/27/2017 04:17 PM, 'Marek Jenkins' via qubes-users wrote:

>> I take it you are using coreboot? if you aren't you should, otherwise
>> you need to change the "primary video" option in the BIOS - which will
>> solve your issue.
> Hi Tai,
>
> no I tried to use Coreboot with default settings (Seabios), compiling and flashing the .rom on the chip was easy. But with Coreboot BIOS the screen stayed black. What made me change back to the default BIOS was, is the fact that the CPU fans got really loud (full-speed) shortly after booting.

You need to either run fancontrol/pwmconfig or use the Raptor OpenBMC
beta with the ASMB4-iKVM chip that came with your board, as coreboot
doesn't handle fan control. (and thus sets them at full speed)

I have the same board - and I can use either the onboard vide or my
pci-e GPU without issues. Did you reset the CMOS after you flashed
coreboot like the wiki instructed?

How long did you wait for coreboot to boot? if you keep the log level at
max it will take around 60 seconds, otherwise lower it to 2 (what I use)
for 10 second boot. I thought the same thing when I started using it
(that it was broken) until I got a null model cable and noticed that it
was in fact booting.
Lower the log level in menuconfig. (this really should not be a default)

> I removed some of the RAM which at least solved the issue with the loud fans, so I assumed my RAM (Kingston ValueRam) might be incompatible with Coreboot ?

Naah, you can use any RAM - the memory compatibility issues were fixed a
long time ago.

> Do you have an idea why coreboot boot failed for me ?
>
> Thanks a lot for the hint with VGA boot priority!
> I have now found the VGA settings in the BIOS and did the following:
> - Enable IOMMU

Don't forget to enable HVM etc as well.

> - Set GPU Boot Priority to "PCIe VGA" instead of "Onboard VGA"
>
> This only partly solves my issue. Now, I can at least boot without issues when the PCIe graphics card is connected to the PCI slot. But I can still only finish the boot process, when I use the VGA Onboard graphics because I still get the same error message as before with the PCIe card:
>
> ERST: Failed to get Error Log Address Range.
> BERT: Can't request iomem region <00000000ccffa....................................>

Strange - this would probably be an OS problem as it works fine for me.
What BIOS revision are you using as the default BIOS?

> Nevertheless, I still have a working signal on my VGA onboard graphics card and can simply continue with the boot process there by switching the connection from HDMI to VGA (hope you get what I mean).
>
> After booting completely, there is still no signal on the PCIe VGA card.

Could I have dmesg, xl dmesg and (as root) # lspci -vv please?

[Marek Jenkins]

> How long did you wait for coreboot to boot? if you keep the log level at
> max it will take around 60 seconds, otherwise lower it to 2 (what I use)
> for 10 second boot. I thought the same thing when I started using it
> (that it was broken) until I got a null model cable and noticed that it
> was in fact booting.
> Lower the log level in menuconfig. (this really should not be a default)

Hi! I did reset CMOS and everything else stated in the manual / coreboot wiki. Your advice to change the cable, recompile/reflash .rom with loglevel=2 and wait up to 60 seconds didn't help. Still no success.

> > I removed some of the RAM which at least solved the issue with the loud fans, so I assumed my RAM (Kingston ValueRam) might be incompatible with Coreboot ?
> Naah, you can use any RAM - the memory compatibility issues were fixed a
> long time ago.
> > Do you have an idea why coreboot boot failed for me ?
> >
> > Thanks a lot for the hint with VGA boot priority!
> > I have now found the VGA settings in the BIOS and did the following:
> > - Enable IOMMU
> Don't forget to enable HVM etc as well.

What mainboard revision and BIOS have you used before coreboot (if you remember) ? I am asking, since I haven't seen anything "HVM" in the BIOS (default BIOS), only "enable IOMMU".

My board is Rev. 1.03G
My BIOS Version: KGPE-D16-ASUS-3309.ROM --> Version 3309

> > - Set GPU Boot Priority to "PCIe VGA" instead of "Onboard VGA"
> >
> > This only partly solves my issue. Now, I can at least boot without issues when the PCIe graphics card is connected to the PCI slot. But I can still only finish the boot process, when I use the VGA Onboard graphics because I still get the same error message as before with the PCIe card:
> >
> > ERST: Failed to get Error Log Address Range.
> > BERT: Can't request iomem region <00000000ccffa....................................>
> Strange - this would probably be an OS problem as it works fine for me.
> What BIOS revision are you using as the default BIOS?
> > Nevertheless, I still have a working signal on my VGA onboard graphics card and can simply continue with the boot process there by switching the connection from HDMI to VGA (hope you get what I mean).
> >
> > After booting completely, there is still no signal on the PCIe VGA card.
> Could I have dmesg, xl dmesg and (as root) # lspci -vv please?

I actually formatted the drive already, to see if Qubes 4.0rc3 would solve my problems.
Therefore I can't help with that sorry.

Strangely, the Qubes 4 installer gave an error message saying that IOMMU support was missing even though IOMMU was enabled in the BIOS at the time.

By now, I really think there is some problem with the whole motherboard. The thing is, I bought the board used offline with 2 Opteron 61** CPUs + CPU Fans + 32GB RAM for just over 120 USD total. I thought I got lucky, but I wasted quite a few hours now just to fix weird issues.. I will get the same board new now and see if that solves my problems.

I will let you know in a few days, probably this weekend i'll have the new one.

[Tai...@gmx.com]

On 11/29/2017 06:56 PM, 'Marek Jenkins' via qubes-users wrote:

>> How long did you wait for coreboot to boot? if you keep the log level at
>> max it will take around 60 seconds, otherwise lower it to 2 (what I use)
>> for 10 second boot. I thought the same thing when I started using it
>> (that it was broken) until I got a null model cable and noticed that it
>> was in fact booting.
>> Lower the log level in menuconfig. (this really should not be a default)
> Hi! I did reset CMOS and everything else stated in the manual / coreboot wiki. Your advice to change the cable, recompile/reflash .rom with loglevel=2 and wait up to 60 seconds didn't help. Still no success.
>
>>> I removed some of the RAM which at least solved the issue with the loud fans, so I assumed my RAM (Kingston ValueRam) might be incompatible with Coreboot ?
>> Naah, you can use any RAM - the memory compatibility issues were fixed a
>> long time ago.
>>> Do you have an idea why coreboot boot failed for me ?

You have a very old 61xx CPU, whereas I have 62xx.
I would also suggest the purchase of a null modem cable in addition to
your $30/ea 62xx CPU so you can view the coreboot boot log via a serial
port on another machine

>>>
>>> Thanks a lot for the hint with VGA boot priority!
>>> I have now found the VGA settings in the BIOS and did the following:
>>> - Enable IOMMU
>> Don't forget to enable HVM etc as well.
> What mainboard revision and BIOS have you used before coreboot (if you remember) ? I am asking, since I haven't seen anything "HVM" in the BIOS (default BIOS), only "enable IOMMU".

Its called SVM, see below. It is in the CPU configuration menu.

> My board is Rev. 1.03G
> My BIOS Version: KGPE-D16-ASUS-3309.ROM --> Version 3309
>
>>> - Set GPU Boot Priority to "PCIe VGA" instead of "Onboard VGA"
>>>
>>> This only partly solves my issue. Now, I can at least boot without issues when the PCIe graphics card is connected to the PCI slot. But I can still only finish the boot process, when I use the VGA Onboard graphics because I still get the same error message as before with the PCIe card:
>>>
>>> ERST: Failed to get Error Log Address Range.
>>> BERT: Can't request iomem region <00000000ccffa....................................>
>> Strange - this would probably be an OS problem as it works fine for me.
>> What BIOS revision are you using as the default BIOS?

You should update the OEM BIOS, I got a new board which had the latest
version of both BIOS and revision (not sure which one but it had the
clear CPU socket covers)
The earlier ones had some trouble apparently.

>>> Nevertheless, I still have a working signal on my VGA onboard graphics card and can simply continue with the boot process there by switching the connection from HDMI to VGA (hope you get what I mean).
>>>
>>> After booting completely, there is still no signal on the PCIe VGA card.
>> Could I have dmesg, xl dmesg and (as root) # lspci -vv please?
> I actually formatted the drive already, to see if Qubes 4.0rc3 would solve my problems.
> Therefore I can't help with that sorry.

No problems :3

>
> Strangely, the Qubes 4 installer gave an error message saying that IOMMU support was missing even though IOMMU was enabled in the BIOS at the time.
>
> By now, I really think there is some problem with the whole motherboard. The thing is, I bought the board used offline with 2 Opteron 61** CPUs + CPU Fans + 32GB RAM for just over 120 USD total. I thought I got lucky, but I wasted quite a few hours now just to fix weird issues.. I will get the same board new now and see if that solves my problems.
>
> I will let you know in a few days, probably this weekend i'll have the new one.
>

You got an excellent deal, usually the board goes for over $250 used
$350 new.

Your coreboot issue is probably related to the the 61xx CPU's not the
board, coreboot doesn't really support them as they are very old as
stated on both the coreboot and libreboot wikis. I would pick up a few
62xx CPU's such as the 6287SE (max performance), 6284SE or dual 6220
(affordable) instead of getting a new board as IMO your board is just
fine this is a software problem.

HVM needs to be enabled as "Secure Virtual Machine Mode" in "CPU
Configuration" as stated in the board manual also enable SRAT, C1E etc.

As I have stated before, it works for me - we just have to continue to  
  troubleshoot :D

[Tai...@gmx.com]

Ah 3309 is the latest OEM BIOS version, so that isn't the problem.

Like I said trying enabling SVM in CPU Config in the CMOS menu. For
coreboot you need 62xx or 63xx CPU's however as stated before.

[Marek Jenkins]

Yes I was quite happy as well. I initially only wanted the motherboard, but saved quite a few bucks now for RAM, CPU fans, etc :-)

> Your coreboot issue is probably related to the the 61xx CPU's not the
> board, coreboot doesn't really support them as they are very old as
> stated on both the coreboot and libreboot wikis. I would pick up a few
> 62xx CPU's such as the 6287SE (max performance), 6284SE or dual 6220
> (affordable) instead of getting a new board as IMO your board is just
> fine this is a software problem.

I forgot to mention that I replaced the two 61xx CPUs directly after the purchase with two Opteron 6380 CPUs (used) for performance reasons. I knew about the microcode vulnerability of the 63xx series, but thought this wouldn't be an issue when using coreboot.

That being said, I am not 100% sure if the CPUs are really the problem. Or are there known issues with CPUs of the 63xx series, I thought they were fully supported ?

> HVM needs to be enabled as "Secure Virtual Machine Mode" in "CPU
> Configuration" as stated in the board manual also enable SRAT, C1E etc.

In my opinion the BIOS settings are looking fine. All the things you mentioned were already enabled (mostly even by default it seems).

I don't wanna bore you, but I made a short video with all my BIOS settings and put it on Youtube, maybe you can spot something that's looking wrong with my settings or hardware config ?

My BIOS settings (video): https://youtu.be/GmRB6dzpj1s

> As I have stated before, it works for me - we just have to continue to  
>   troubleshoot :D

Thank you so much for all your help by the way. Greatly appreciated!

[Tai...@gmx.com]

Ah nice, that was a good choice - they are much better.
63xx needs a microcode update for safe operation and a functional IOMMU, verify that is included in CBFS.

That being said, I am not 100% sure if the CPUs are really the problem. Or are there known issues with CPUs of the 63xx series, I thought they were fully supported ?

HVM needs to be enabled as "Secure Virtual Machine Mode" in "CPU 
Configuration" as stated in the board manual also enable SRAT, C1E etc.

In my opinion the BIOS settings are looking fine. All the things you mentioned were already enabled (mostly even by default it seems).

I don't wanna bore you, but I made a short video with all my BIOS settings and put it on Youtube, maybe you can spot something that's looking wrong with my settings or hardware config ? 

My BIOS settings (video): https://youtu.be/GmRB6dzpj1s

Have you tried non-qubes virtualization: maybe try installing a random distro with a version of xen similar to what qubes uses?

I believe that you are suffering from some obscure qubes related bug.

If you can pick up a null modem serial cable it would be helpful to figure out as to why coreboot isn't working for you. Can you give me the coreboot config file?

As I have stated before, it works for me - we just have to continue to   
   troubleshoot :D

Thank you so much for all your help by the way. Greatly appreciated!

I like to be one of the good people in the world that doesn't nickle and dime people who are doing the right thing (buying libre products), there are so few decent folks these days and almost everyone is so greedy so I like to be the exception.

Free tech support from yours truly is one of the benefits that people get from buying libre quality products.

[Tai...@gmx.com]

Status report? is it working?

I still need your coreboot config file, you may also have to wait 5
minutes or so on the first post flash boot so that it can build the
memory reference data.

[Tai...@gmx.com]

I forgot to mention (I presume you have not read my notes about this on
the coreboot wiki) you need microcode updates for IOMMU and secure
operations with the 63xx, please provide a dmesg so I can see if you
have the right version!