HCL: Supermicro AMD server: success!

Bill Wether

unread,

Jan 21, 2016, 12:35:44 PM1/21/16

to qubes-users

With all the doom and gloom in the Qubes docs about AMD IOMMU and incorrect APIC
entries, I thought that this might be useful to some.

I put this machine together from used eBay parts mostly, with a new chassis and PSU ($239 +
shipping). I have a production machine that's closely similar that I want to migrate from RHEL to Qubes at some point.

Chassis: CSE-733TQ-665B (665 W PSU)

Mobo: Supermicro H8DGi-F (p/n MBD-H8DGi-F-0), dual Socket G34
Processors: dual AMD Opteron 6128 2 GHz Eight Core (OS6128WKT8EGO)
CPU Fans: Dynatron A6 Socket G34 (2U height)
RAM: 14 pcs MT18JSF25672PZ-1G4F1AB 2GB 1Rx4 DDR3 PC3-10600 1333 REG MEMORY
Disc: SAMSUNG 850 EVO 2.5" 500GB SATA III SSD MZ-75E500B/AM
Video: ATI Radeon HD3470 Dual Display Port PCIe Video Card Dell W459D C120D
CD/DVD: Samsung SH-224FB/RSBS 24X CD DVD Internal burner
TPM: Supermicro AOM-TPM-9655V

Total cost for 150 Gflops peak: $880 plus some patience and a bit of hair loss.

However, the BIOS was ancient: V. 2.0a, and the mobo has a bug: Xen refuses to
boot because there's a security hole relating to SATA/IDE combined mode.
Disabling that in the BIOS enabled Qubes to boot.

However, due to the notorious APIC table errors, Xen turned off IOMMU.

Also, thie BIOS itself allowed me to configure the TPM but then couldn't find it
on reboot.

After a _lot_ of screwing around, I reflashed the latest BIOS, V. 3.5,
Septermber 2015, and it apparently fixed everything: xl dmesg says IOMMU0 and
IOMMU1 enabled, and TPM enabled.

So here's one datapoint for a working AMD system with Qubes 3.1rc2, hooray.

I've attached the qubes-hcl output (for some reason it couldn't save it to a file).

I've also attached the output of xl dmesg for both the bad old BIOS and the new one.

Cheers

BillW

hcl.yml

iommufail.txt

iommufixed.txt

Chris Laprise

unread,

Jan 24, 2016, 6:50:16 PM1/24/16

to Bill Wether, qubes-users

> --

Excellent report... Thanks!

ohhe...@gmail.com

unread,

Feb 21, 2016, 11:08:33 AM2/21/16

to qubes-users

Bill Wether

unread,

Feb 21, 2016, 12:19:03 PM2/21/16

to qubes-users, ohhe...@gmail.com

The fan is set to "power save" mode in the BIOS, which keeps the fan speed way down unless it's really crunching on something. I have two of them right next to my desk, one with a RAID 5 array and one with a SSD, and they're pretty quiet. Really a sweet system. We recently upgraded it to a pair of 6176es and 64GB of memory for another $200 total, and it runs like a champ.

The noise level is nothing remotely comparable to the same board in a 1U pizza box--those sound like a jet engine.

Cheers

BillW

ohhe...@gmail.com

unread,

Feb 21, 2016, 1:35:23 PM2/21/16

to qubes-users, ohhe...@gmail.com

Do you use them as servers or as desktops?

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWygNWAAoJELl89Kfp5xgBU7oP/iMxsbqldVF98cJS6+Jh/SLV
H9JFd5TUh6Or2KwHY5bb0ExZIlblrT1lmlHCe/TRqS7sDxB8gyG0+naf7XA7bRZF
SEdDWv0L1L+p64GN31qcVQiqp7ShGA0pyHEYD4qc7+GsawpZkgMTlume8PnZF9bt
gUlUbB3IlAa8+UDr8n6YvDyXBhV/18G+Go4Nne/ezafb8yyxp9kbG0h2LxHZrvZD
kF/MPnolFsh7LFhhVETPSUEbBU2mov1f4O1VDwApJIwGQFXDvmAKG2re7BfCIUW1
kYG+AbqDFrB8W4+G/xedYYcbAmtDtkBxTn9xkJ4rXZbmKuQnDQW1wUxt4kEsEvHd
N4s4+nunTwSebMaLrZx2JEX4IWVGNvRKElVLICZofvAGDc4w5DDOlZHMLdTPZN8c
HMC1Gft73BSwh+bLCnCSC1duDeeJ/qYpI1aGdbbhASOWAx3T77ilq60kPG1viAFM
o+ujAT/H1dDsxi2FkBeOC1b6/X5hgkFYT7aTpcRhuW5c713g+UO3AfYyCwaNPPxC
MBlBfsQLe0yxx51/u5qiAknkWaKQxpJxtEYSuJ5FZVJt8yxSZPPgivWCRC+OkKkX
/6W0PPqqaVyZPblJti80YARj5PR2zIg18u2nrmHQfo2qh9kMV7j0eEmCn7cgc6rO
8GaQAa9le35nOGdfUDkU
=H2/V
-----END PGP SIGNATURE-----

Bill Wether

unread,

Feb 21, 2016, 2:13:14 PM2/21/16

to qubes-users, ohhe...@gmail.com

Do you use them as servers or as desktops?

The RHEL 6 one is a server, the Qubes one is a desktop, but the boxes are nearly identical hardware-wise. We run heavy-duty multithreaded simulations on both. I'm impressed with how much faster the Xen PVMs are than Virtualbox VMs.

We may try doing remote access via a Tor hidden service. It doesn't make much sense SSHing into a Qubes box remotely at this point, I don't think. If we were clever Xen/Linux hackers, we could probably make a dedicated VM to serve files from, but there are a bunch of security issues to think through, what with moving files to and from more trusted VMs, and besides, it's inconvenient by design.

As long as nobody runs a browser on the server box, it's probably fine on RHEL 6 for awhile.

Cheers

BillW

dlme...@gmail.com

unread,

Oct 9, 2016, 4:03:49 AM10/9/16

to qubes-users, ohhe...@gmail.com

Tried this Bill? https://www.coreboot.org/Board:supermicro/h8qgi

DM.

dlme...@gmail.com

unread,

Oct 9, 2016, 4:09:04 AM10/9/16

to qubes-users, ohhe...@gmail.com, dlme...@gmail.com

On Sunday, 9 October 2016 19:03:49 UTC+11, dlme...@gmail.com wrote:
> Tried this Bill? https://www.coreboot.org/Board:supermicro/h8qgi
>
> DM.

Oops, just noticed you're on H8DGI :O|

Reply all

Reply to author

Forward