HCL - Supermicro X10SAE (with request for assistance)

[Qubes Fan]

HCL Report

Qubes release 2 (R2)
Model Name:    Supermicro X10SAE

Chipset:    00:00.0 Host bridge [0600]: Intel Corporation Haswell DRAM Controller [8086:0c08] (rev 06)
VGA:        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Juniper [Radeon HD 5700 Series] [1002:68b8]
CPU:        Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
BIOS:        1.00
VT-x:        Active
VT-d:        Not Active

After several days of troubleshooting and around twenty reinstallations, here is what I've come up with:

The MOBO and CPU both support VT-d, but I had to disable it in the BIOS in order to get Qubes working.

Explanation: If VT-d is enabled in the BIOS, then installation of Qubes proceeds normally up the last step (VM creation). At this stage, the user is presented with three options. The first two options create some VMs, while the third option creates no VMs (and leaves the user to do it manually later). If either of the first two options is selected, the system completely freezes when installation tries to create the netvm. A hard reboot is necessary. If the third option is chosen (i.e., create no VMs during installation), then the installation completes successfully, and the user can log in. But the system again freezes as soon as a netvm is manually created (with 'qvm-create --net --label red netvm' in dom0), and again a hard reboot is required. (I tried pretty much every possible combination of assigning different devices to the netvm before starting it up, but the system keeps freezing. I also tried updating (i.e., qubes-dom0-update), but no dice.)

Question/Hypothesis: Is this happening because the new Haswell implementation of VT-d (if there is such a thing) is not yet supported by the version of the Linux kernel we're using? Or maybe it's not yet supported by (the version of) Xen we're using? After all, this MOBO/CPU just came out in the last week or so. Maybe I just need to wait for an update?

The CPU has integrated Intel HD 4600 (GT2) graphics, but I had to use a discrete GPU in order to install Qubes.

Explanation: I actually chose the Xeon E3-1245v3 over the E3-1230v3 because the former has Intel graphics, and the HCL page recommends Intel graphics over nVidia/AMD. But, as it turns out (in my case, at least), using an AMD Radeon HD 5770 was necessary. At first, I tried installing Qubes with just the MOBO and CPU (no discrete graphics, no other PCI/e devices of any kind). This caused several errors during the installation, which I have seen others mention (in different situations). The first major thing that happened is that the graphical install failed (I believe it said, 'X startup failed'), and it kicked back to an anaconda text-only installation. I was able to select my options (time zone, storage device target for installation, etc.), but then I received an error which said, 'luks device has no key'. (I gathered from a post by Marek in an old thread that this is probably due to shortcomings in anaconda.) The installation could not proceed. So then I slotted in the Radeon HD 5770, and everything was fine.

Question/Hypothesis: Same as above. Maybe this is a kernel/Xen issue? I'm not personally worried about it, as the HD 5770 is working fine, but this may be important to others. By contrast, I really want VT-d to work.

Request for assistance: Is there anything else I can try to do to get VT-d working?

[Qubes Fan]

Just had another thought: Would attempting a kernel downgrade be a good idea to try to get VT-d working? I have little clue what I'm doing here, so if there's any pointers to anything I could read to try to educate myself, that would be appreciated. I don't really want to attempt this if there's little chance of it helping anyone and a high chance of something going terribly wrong.

[Qubes Fan]

If no one minds, I'm just going to keep replying to myself as new thoughts occur to me. :P

Would I be able to glean any useful information by attempting to install baremetal Xen on this system in order to see whether VT-d works? Would this allow me to determine whether the problem is with Xen, Qubes, my motherboard's BIOS, or something else? I've never used or installed baremetal Xen before (if you don't count Qubes). Should I just follow the instructions in the Xen wiki and guides online? And how will I be able to tell whether VT-d is actually working? The fact that my system freezes shortly after decrypting my boot disk and before the dom0 login prompt shows that VT-d isn't working with Qubes (presumably because the netvm is being started at that point; see above). But I presume that I cannot just enable VT-d (flag  'iommu=1' or something) in a Xen installation and infer from the system not freezing that it's working. I presume I'd have to have it do something roughly equivalent to whatever Qubes does when it starts the netvm, right?

On a slightly different note, reading this page and reading about how several Supermicro motherboard models in the past required a BIOS update in order to get VT-d working correctly makes me fear that it might be a hardware/BIOS problem. If anyone has any thoughts on how to tell, please let me know! Knowing that it's a hardware/BIOS issue would greatly help me to re-evaluate my options. I would really prefer not to reflash my BIOS (for the reasons Joanna has been talking about for years), but I'm not sure if I have much of a choice if that's the only path to VT-d support on this board...

[Qubes Fan]

After reading this post I tried re-enabled VT-d but disabling TXT in the bios. No change. 

[Qubes Fan]

 Tried selecting the Xen 4.1.2 option under the "advanced" options in the bootloader. No change.

[Qubes Fan]

Ok, so I figured out how to upgrade the kernel by doing 'qubes-dom0-update --enablerepo=qubes-dom0-unstable kernel-3.9.2' (from reading this thread). The problem is still occurring as before, BUT I now notice that there is a one-line error message that BRIEFLY flashes before I get the black frozen screen (with a black cursor with a white border). The message is so fast that I can barely make it out, but it says something like (xxx) pciback (xxx) preparation failed (xxx). I wonder what this means.

[Qubes Fan]

I rebooted to try to see more of the error message, and strangely this time there were THREE lines of error messages (two new ones in addition to the aforementioned one). I literally changed nothing; only hit the reset button. Again, WAY too fast to read, but I did see that they both started with *ERROR*, which can't be good!

[Qubes Fan]

Apparently that 'pciback' message is not related to the VT-d issue, because I just noticed that it's there even when VT-d is disabled in the BIOS and the system is usable. 

[Marek Marczykowski-Górecki]

On 19.06.2013 15:35, Qubes Fan wrote:
> On Wednesday, June 19, 2013 5:57:44 AM UTC-7, Qubes Fan wrote:
>
>> On Wednesday, June 19, 2013 12:39:28 AM UTC-7, Qubes Fan wrote:
>>
>>> On Tuesday, June 18, 2013 9:55:53 PM UTC-7, Qubes Fan wrote:
>>>
>>>> On Monday, June 17, 2013 4:27:52 AM UTC-7, Qubes Fan wrote:
>>>>

>>>>> *HCL Report*

>>>>>
>>>>> Qubes release 2 (R2)
>>>>>> Model Name: Supermicro X10SAE
>>>>>>
>>>>>> Chipset: 00:00.0 Host bridge [0600]: Intel Corporation Haswell DRAM
>>>>>> Controller [8086:0c08] (rev 06)
>>>>>> VGA: 01:00.0 VGA compatible controller [0300]: Advanced Micro
>>>>>> Devices [AMD] nee ATI Juniper [Radeon HD 5700 Series] [1002:68b8]
>>>>>> CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
>>>>>> BIOS: 1.00
>>>>>> VT-x: Active
>>>>>> VT-d: Not Active
>>>>>>
>>>>>
>>>>> After several days of troubleshooting and around twenty
>>>>> reinstallations, here is what I've come up with:
>>>>>

>>>>> *The MOBO and CPU both support VT-d, but I had to disable it in the
>>>>> BIOS in order to get Qubes working.*
>>>>>
>>>>> *Explanation:* If VT-d is enabled in the BIOS, then installation of

>>>>> Qubes proceeds normally up the last step (VM creation). At this stage, the
>>>>> user is presented with three options. The first two options create some
>>>>> VMs, while the third option creates no VMs (and leaves the user to do it
>>>>> manually later). If either of the first two options is selected, the system
>>>>> completely freezes when installation tries to create the netvm. A hard
>>>>> reboot is necessary. If the third option is chosen (i.e., create no VMs
>>>>> during installation), then the installation completes successfully, and the
>>>>> user can log in. But the system again freezes as soon as a netvm is
>>>>> manually created (with 'qvm-create --net --label red netvm' in dom0), and
>>>>> again a hard reboot is required. (I tried pretty much every possible
>>>>> combination of assigning different devices to the netvm before starting it
>>>>> up, but the system keeps freezing. I also tried updating (i.e.,
>>>>> qubes-dom0-update), but no dice.)
>>>>>

>>>>> *Question/Hypothesis: *Is this happening because the new Haswell

>>>>> implementation of VT-d (if there is such a thing) is not yet supported by
>>>>> the version of the Linux kernel we're using? Or maybe it's not yet
>>>>> supported by (the version of) Xen we're using? After all, this MOBO/CPU
>>>>> just came out in the last week or so. Maybe I just need to wait for an
>>>>> update?
>>>>>

>>>>> *The CPU has integrated Intel HD 4600 (GT2) graphics, but I had to use
>>>>> a discrete GPU in order to install Qubes.*
>>>>>
>>>>> *Explanation:* I actually chose the Xeon E3-1245v3 over the E3-1230v3

>>>>> because the former has Intel graphics, and the HCL page recommends Intel
>>>>> graphics over nVidia/AMD. But, as it turns out (in my case, at least),
>>>>> using an AMD Radeon HD 5770 was necessary. At first, I tried installing
>>>>> Qubes with just the MOBO and CPU (no discrete graphics, no other PCI/e
>>>>> devices of any kind). This caused several errors during the installation,
>>>>> which I have seen others mention (in different situations). The first major
>>>>> thing that happened is that the graphical install failed (I believe it
>>>>> said, 'X startup failed'), and it kicked back to an anaconda text-only
>>>>> installation. I was able to select my options (time zone, storage device
>>>>> target for installation, etc.), but then I received an error which said,
>>>>> 'luks device has no key'. (I gathered from a post by Marek in an old thread
>>>>> that this is probably due to shortcomings in anaconda.) The installation
>>>>> could not proceed. So then I slotted in the Radeon HD 5770, and everything
>>>>> was fine.
>>>>>

>>>>> *Question/Hypothesis:* Same as above. Maybe this is a kernel/Xen

>>>>> issue? I'm not personally worried about it, as the HD 5770 is working fine,

>>>>> but this may be important to others. By contrast, I *really* want VT-d
>>>>> to work.
>>>>>
>>>>> *Request for assistance:* Is there anything else I can try to do to
>>>>> get VT-d working?
>>>>>
>>>>
>>>> Just had another thought: Would attempting a kernel *downgrade* be a

>>>> good idea to try to get VT-d working? I have little clue what I'm doing
>>>> here, so if there's any pointers to anything I could read to try to educate
>>>> myself, that would be appreciated. I don't really want to attempt this if
>>>> there's little chance of it helping anyone and a high chance of something
>>>> going terribly wrong.
>>>>
>>>
>>> If no one minds, I'm just going to keep replying to myself as new
>>> thoughts occur to me. :P
>>>
>>> Would I be able to glean any useful information by attempting to install
>>> baremetal Xen on this system in order to see whether VT-d works? Would this
>>> allow me to determine whether the problem is with Xen, Qubes, my
>>> motherboard's BIOS, or something else? I've never used or installed
>>> baremetal Xen before (if you don't count Qubes). Should I just follow the
>>> instructions in the Xen wiki and guides online? And how will I be able to
>>> tell whether VT-d is actually working? The fact that my system freezes
>>> shortly after decrypting my boot disk and before the dom0 login prompt
>>> shows that VT-d isn't working with Qubes (presumably because the netvm is
>>> being started at that point; see above). But I presume that I cannot just
>>> enable VT-d (flag 'iommu=1' or something) in a Xen installation and

>>> infer from the system *not* freezing that it's working. I presume I'd
>>> have to have it *do* something roughly equivalent to whatever Qubes does

>>> when it starts the netvm, right?
>>>

>>> On a slightly different note, reading this page<http://qubes-os.org/trac/wiki/HCLR1>and reading about how several Supermicro motherboard models in the past
>>> required a BIOS <http://wiki.xen.org/wiki/VTd_HowTo> update<http://www.newegg.com/Product/Product.aspx?Item=N82E16813182164>in order to get VT-d working correctly makes me fear that it might be a

>>> hardware/BIOS problem. If anyone has any thoughts on how to tell, please
>>> let me know! Knowing that it's a hardware/BIOS issue would greatly help me
>>> to re-evaluate my options. I would really prefer not to reflash my BIOS
>>> (for the reasons Joanna has been talking about for years), but I'm not sure
>>> if I have much of a choice if that's the only path to VT-d support on this
>>> board...
>>>
>>

>> After reading this post<https://groups.google.com/d/msg/qubes-devel/6I07Bbzn5M4/njBqjFXAku4J> I

>> tried re-enabled VT-d but disabling TXT in the bios. No change.
>>
> Tried selecting the Xen 4.1.2 option under the "advanced" options in the
> bootloader. No change.

You can try 4.1.5:
sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing xen

If you have physical serial port in this hardware (USB converted isn't
enough), you can try to get console output:
1. in grub menu, edit the entry, find xen.gz line and append "com1=115200,8n1
console=com1,vga"
2. Connect serial console to other computer ("null modem" cable).
3. Run minicom on the other computer.

Besides that IMHO this worth a mail to xen-users or xen-devel mailing list.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[Qubes Fan]

Thank you for the timely post, Marek. I was just trying to figure out the command to get Xen 4.1.5. Unfortunately, the problem persists even after this.

If you have physical serial port in this hardware (USB converted isn't
enough), you can try to get console output:
1. in grub menu, edit the entry, find xen.gz line and append "com1=115200,8n1
console=com1,vga"
2. Connect serial console to other computer ("null modem" cable).
3. Run minicom on the other computer.

Hm, interesting. No actual ports on the motherboard, but the manual states that it has two serial (COM) headers. I will have to try to figure out the hardware for this.

Besides that IMHO this worth a mail to xen-users or xen-devel mailing list.

Thanks. I will do that. Should I just tell them all the things I posted here? 

[Qubes Fan]

The message showed up for a long time as Qubes was shutting down. The full message is:

[    2.81055] pciback 0000:03:00.0: MSI-X preparation failed (-38)

Googling parts of the message suggests that the message has to do with my video card vis-a-vis Xen. See this discussion on xen-devel, and here, and here. 

[Marek Marczykowski-Górecki]

Yes, perhaps even forward this message/part of messages.

[Qubes Fan]

On Monday, June 17, 2013 4:27:52 AM UTC-7, Qubes Fan wrote:

Hi Zrubi,

I tested R1 so that you can have more information for the HCL. Results:

Installation fails with IGP: 'firewire_ohci: inconsistent self IDs.
Installation fails with discrete GPU: 'No driver found'.

[Qubes Fan]

Correction: Installation with IGP only hangs on 'firewire_ohci: inconsistent self IDs' for a long time. Then it's 'No driver found' (same as with discrete GPU).

[Qubes Fan]

Ok, I've installed the latest Xen on this hardware with Fedora 18 as the host OS. Here is the relevant output of sudo xl dmesg:

(XEN)  Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN)  Intel VT-d Snoop Control enabled.
(XEN)  Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN)  Intel VT-d Queued Invalidation enabled.
(XEN)  Intel VT-d Interrupt Remapping enabled.
(XEN)  Intel VT-d Shared EPT tables enabled.
(XEN)  I/O virtualisation enabled

So, what's the deal here? Shouldn't it be working? Or do I need 'Dom0 DMA Passthrough' to be enabled? (If other outputs are relevant, let me know, and I will post them.)

[Qubes Fan]

Based on this post, I tried xl info | grep ^virt_caps.*hvm_directio. Not sure if this is only supposed to be done on a Qubes system or if it's helpful at all, but the output is:

virt_caps             : hvm hvm_directio

[Qubes Fan]

After a little bit of searching and reading, it sounds like this means that VT-d is, in fact, working with Xen on this system. (Unless I am missing something, which is always possible.) Given that this is Xen 4.2.2 with Fedora 18, does this mean that Qubes with Xen 4.2.x would probably allow me to enable VT-d? :D

[Qubes Fan]

Oh, I also meant to link to this thread, in which someone else has 'Dom0 DMA Passthrough not enabled' and Joanna says she thinks it's okay!

[Qubes Fan]

I think the recent update fixed it! Now I no longer experience a freeze/crash when booting into Qubes with VT-d enabled in the BIOS.

Furthermore,

sudo xl dmesg outputs:

(XEN)  Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN)  Intel VT-d Snoop Control enabled.
(XEN)  Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN)  Intel VT-d Queued Invalidation enabled.
(XEN)  Intel VT-d Interrupt Remapping enabled.

(XEN)  Intel VT-d Shared EPT tables not enabled.
(XEN)  I/O virtualisation enabled.

sudo xl info outputs:

virt_caps           : hvm hvm_directio

---------------

Does this mean success? The only thing I'm not sure about is why Shared EPT tables is not enabled in the Qubes installation while it is enabled in the regular Xen installation. Does Qubes use it?

[Qubes Fan]

 
Oh, and of course there's this! (Forgot to add to previous post):

Qubes release 2 (R2)
Model Name:    Supermicro X10SAE

Chipset:    00:00.0 Host bridge [0600]: Intel Corporation Haswell DRAM Controller [8086:0c08] (rev 06)
VGA:        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Juniper [Radeon HD 5700 Series] [1002:68b8]
CPU:        Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
BIOS:        1.00
VT-x:        Active

VT-d:        Active

So, the only mystery left is 'Shared EPT tables'. Since other people have also had this not enabled, I'm not going to worry about it, and hope that I don't somehow get DMA attacked or something.

[cprise]

On 06/17/2013 07:27 AM, Qubes Fan wrote:
> _*HCL Report*_

>
> Qubes release 2 (R2)
> Model Name: Supermicro X10SAE
>
> Chipset: 00:00.0 Host bridge [0600]: Intel Corporation Haswell
> DRAM Controller [8086:0c08] (rev 06)
> VGA: 01:00.0 VGA compatible controller [0300]: Advanced Micro
> Devices [AMD] nee ATI Juniper [Radeon HD 5700 Series] [1002:68b8]
> CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
> BIOS: 1.00
> VT-x: Active
> VT-d: Not Active
>
>
> After several days of troubleshooting and around twenty reinstallations,
> here is what I've come up with:
>

> *The MOBO and CPU both support VT-d, but I had to disable it in the BIOS
> in order to get Qubes working.*
>
> *Explanation:* If VT-d is enabled in the BIOS, then installation of

> Qubes proceeds normally up the last step (VM creation). At this stage,
> the user is presented with three options. The first two options create
> some VMs, while the third option creates no VMs (and leaves the user to
> do it manually later). If either of the first two options is selected,
> the system completely freezes when installation tries to create the
> netvm. A hard reboot is necessary. If the third option is chosen (i.e.,
> create no VMs during installation), then the installation completes
> successfully, and the user can log in. But the system again freezes as
> soon as a netvm is manually created (with 'qvm-create --net --label red
> netvm' in dom0), and again a hard reboot is required. (I tried pretty
> much every possible combination of assigning different devices to the
> netvm before starting it up, but the system keeps freezing. I also tried
> updating (i.e., qubes-dom0-update), but no dice.)
>

> *Question/Hypothesis: *Is this happening because the new Haswell

> implementation of VT-d (if there is such a thing) is not yet supported
> by the version of the Linux kernel we're using? Or maybe it's not yet
> supported by (the version of) Xen we're using? After all, this MOBO/CPU
> just came out in the last week or so. Maybe I just need to wait for an
> update?
>

> *The CPU has integrated Intel HD 4600 (GT2) graphics, but I had to use a
> discrete GPU in order to install Qubes.*
>
> *Explanation:* I actually chose the Xeon E3-1245v3 over the E3-1230v3

> because the former has Intel graphics, and the HCL page recommends Intel
> graphics over nVidia/AMD. But, as it turns out (in my case, at least),
> using an AMD Radeon HD 5770 was necessary. At first, I tried installing
> Qubes with just the MOBO and CPU (no discrete graphics, no other PCI/e
> devices of any kind). This caused several errors during the
> installation, which I have seen others mention (in different
> situations). The first major thing that happened is that the graphical
> install failed (I believe it said, 'X startup failed'), and it kicked
> back to an anaconda text-only installation. I was able to select my
> options (time zone, storage device target for installation, etc.), but
> then I received an error which said, 'luks device has no key'. (I
> gathered from a post by Marek in an old thread that this is probably due
> to shortcomings in anaconda.) The installation could not proceed. So
> then I slotted in the Radeon HD 5770, and everything was fine.
>

> *Question/Hypothesis:* Same as above. Maybe this is a kernel/Xen issue?

> I'm not personally worried about it, as the HD 5770 is working fine, but

> this may be important to others. By contrast, I /really/ want VT-d to work.
>
> *Request for assistance:* Is there anything else I can try to do to get
> VT-d working?
>
> --

I just wanted to point out Supermicro indicates[1] this motherboard is
incompatible with versions of fedora after 18. Maybe some kind of caveat
should be added to this entry in the Qubes HCL.

-
[1] http://www.supermicro.com/support/resources/OS/C226.cfm