HCL: Supermicro AMD server: success!

1684 views
Skip to first unread message

Bill Wether

unread,
Jan 21, 2016, 12:35:44 PM1/21/16
to qubes-users
With all the doom and gloom in the Qubes docs about AMD IOMMU and incorrect APIC
entries, I thought that this might be useful to some.

 I put this machine together from used eBay parts mostly, with a new chassis and PSU ($239 +
shipping).  I have a production machine that's closely similar that I want to migrate from RHEL to Qubes at some point.

Chassis: CSE-733TQ-665B  (665 W PSU)

Mobo: Supermicro H8DGi-F (p/n MBD-H8DGi-F-0), dual Socket G34
Processors: dual AMD Opteron 6128  2 GHz Eight Core (OS6128WKT8EGO)
CPU Fans: Dynatron A6 Socket G34 (2U height)
RAM: 14 pcs MT18JSF25672PZ-1G4F1AB 2GB 1Rx4 DDR3 PC3-10600 1333 REG MEMORY
Disc:  SAMSUNG 850 EVO 2.5" 500GB SATA III SSD MZ-75E500B/AM
Video: ATI Radeon HD3470 Dual Display Port PCIe Video Card Dell W459D C120D
CD/DVD:  Samsung SH-224FB/RSBS 24X CD DVD Internal burner
TPM: Supermicro  AOM-TPM-9655V

Total cost for 150 Gflops peak: $880 plus some patience and a bit of hair loss.

However, the BIOS was ancient: V. 2.0a, and the mobo has a bug: Xen refuses to
boot because there's a security hole relating to SATA/IDE combined mode.
Disabling that in the BIOS enabled Qubes to boot.

However, due to the notorious APIC table errors, Xen turned off IOMMU.

Also, thie BIOS itself allowed me to configure the TPM but then couldn't find it
on reboot.

After a _lot_ of screwing around, I reflashed the latest BIOS, V. 3.5,
Septermber 2015, and it apparently fixed everything: xl dmesg says IOMMU0 and
IOMMU1 enabled, and TPM enabled.

So here's one datapoint for a working AMD system with Qubes 3.1rc2, hooray.

I've attached the qubes-hcl output (for some reason it couldn't save it to a file).

I've also attached the output of xl dmesg for both the bad old BIOS and the new one.

Cheers

BillW

hcl.yml
iommufail.txt
iommufixed.txt

Chris Laprise

unread,
Jan 24, 2016, 6:50:16 PM1/24/16
to Bill Wether, qubes-users
> --

Excellent report... Thanks!

ohhe...@gmail.com

unread,
Feb 21, 2016, 11:08:33 AM2/21/16
to qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Is this setup loud? Would the noise level be noticeable if only using SSD’s?
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWyeDxAAoJELl89Kfp5xgBQNQP+wXqVZXUeqY0kR5V6W14eWaA
+V27sMWPZhLq2UhWsdqHVv9vtkxdKwvU/PFTmDOZTAYr03JoZ0t42uSQNS8EGL55
EtDgdkccSVnbsHvShVU/vVfgiiPPwr1u6NXPHpZJklCXA88ydF3RHpa0096xgd02
aGD61Is2skPmhcgIt0YzZeFo8JhYFd5gtVMdMA7h8FGivB0Pxk9tzrNjIvAwvvvL
VG3+qJdkgnsEe57E0CX5UV+uHXFxwM1wzlYnSon+e47bTFBAOBsdGXJNQu0S3Ff0
pTAKIWdkdFeTEgXxkuM0Tk/TSIKacEyPq8siUSuZGWtzkjuNwtj/ZX0Ov5JBH5jI
VTbN6WwRXk1HkYNOW+09UiFCTao4YKsqnLafpB9BGRFK5Uax3WzwlhDjgToQ+ScB
Y9yQuKNG5wCtNaTZ/AZfHBueuCCpQMGKetmIwhYweHMEJ3YqLfqV1jgPCXisLNV0
9rMO7BHJdNxHaT1N109gdY/QFYNZ6uPJv4XVpP2nxnE6IBN8AcFnINLj3pfJ8zRw
6Xlyik+NNHgwaJK5GSge1qQWAurl5JzPo0W7RJtQHrxlG0O734GFKYtKR8dv5Raq
Jrc6wooZguyOMqrQYVLvnBPIj9heR8dNQI72XfhDrgUY7NIMfRB1P2rLtl6IclWW
vH1H4u/SLtXbhNpnqp2X
=QvCA
-----END PGP SIGNATURE-----

Bill Wether

unread,
Feb 21, 2016, 12:19:03 PM2/21/16
to qubes-users, ohhe...@gmail.com

The fan is set to "power save" mode in the BIOS, which keeps the fan speed way down unless it's really crunching on something.   I have two of them right next to my desk, one with a RAID 5 array and one with a SSD, and they're pretty quiet.   Really a sweet system.  We recently upgraded it to a pair of 6176es and 64GB of memory for another $200 total, and it runs like a champ.

The noise level is nothing remotely comparable to the same board in a 1U pizza box--those sound like a jet engine.

Cheers

BillW

ohhe...@gmail.com

unread,
Feb 21, 2016, 1:35:23 PM2/21/16
to qubes-users, ohhe...@gmail.com
Do you use them as servers or as desktops?
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWygNWAAoJELl89Kfp5xgBU7oP/iMxsbqldVF98cJS6+Jh/SLV
H9JFd5TUh6Or2KwHY5bb0ExZIlblrT1lmlHCe/TRqS7sDxB8gyG0+naf7XA7bRZF
SEdDWv0L1L+p64GN31qcVQiqp7ShGA0pyHEYD4qc7+GsawpZkgMTlume8PnZF9bt
gUlUbB3IlAa8+UDr8n6YvDyXBhV/18G+Go4Nne/ezafb8yyxp9kbG0h2LxHZrvZD
kF/MPnolFsh7LFhhVETPSUEbBU2mov1f4O1VDwApJIwGQFXDvmAKG2re7BfCIUW1
kYG+AbqDFrB8W4+G/xedYYcbAmtDtkBxTn9xkJ4rXZbmKuQnDQW1wUxt4kEsEvHd
N4s4+nunTwSebMaLrZx2JEX4IWVGNvRKElVLICZofvAGDc4w5DDOlZHMLdTPZN8c
HMC1Gft73BSwh+bLCnCSC1duDeeJ/qYpI1aGdbbhASOWAx3T77ilq60kPG1viAFM
o+ujAT/H1dDsxi2FkBeOC1b6/X5hgkFYT7aTpcRhuW5c713g+UO3AfYyCwaNPPxC
MBlBfsQLe0yxx51/u5qiAknkWaKQxpJxtEYSuJ5FZVJt8yxSZPPgivWCRC+OkKkX
/6W0PPqqaVyZPblJti80YARj5PR2zIg18u2nrmHQfo2qh9kMV7j0eEmCn7cgc6rO
8GaQAa9le35nOGdfUDkU
=H2/V
-----END PGP SIGNATURE-----

Bill Wether

unread,
Feb 21, 2016, 2:13:14 PM2/21/16
to qubes-users, ohhe...@gmail.com
Do you use them as servers or as desktops?

The RHEL 6 one is a server, the Qubes one is a desktop, but the boxes are nearly identical hardware-wise.  We run heavy-duty multithreaded simulations on both.  I'm impressed with how much faster the Xen PVMs are than Virtualbox VMs.

We may try doing remote access via a Tor hidden service.  It doesn't make much sense SSHing into a Qubes box remotely at this point, I don't think.  If we were clever Xen/Linux hackers, we could probably make a dedicated VM to serve files from, but there are a bunch of security issues to think through, what with moving files to and from more trusted VMs, and besides, it's inconvenient by design.

As long as nobody runs a browser on the server box, it's probably fine on RHEL 6 for awhile.

Cheers

BillW

dlme...@gmail.com

unread,
Oct 9, 2016, 4:03:49 AM10/9/16
to qubes-users, ohhe...@gmail.com

dlme...@gmail.com

unread,
Oct 9, 2016, 4:09:04 AM10/9/16
to qubes-users, ohhe...@gmail.com, dlme...@gmail.com
On Sunday, 9 October 2016 19:03:49 UTC+11, dlme...@gmail.com wrote:
> Tried this Bill? https://www.coreboot.org/Board:supermicro/h8qgi
>
> DM.

Oops, just noticed you're on H8DGI :O|

Reply all
Reply to author
Forward
0 new messages