Sata pci-e card for ARM device

[Michael Howard]

On the off chance, has anybody (Gordan?) got any Sata pci-e card working
on the Gigabye MP30 board?

Cheers,

Mike.

[Gordan Bobic]

Hi Michael,

I haven't had to try, the on-board SATA has been sufficient for me. I had an issue a while back where upstream kernels would throw a panic regardless of what was plugged into any PCIe ports. The CentOS distro kernel worked fine, though. This was 3 years ago and a few LTS kernel releases ago.

I'll see if I can find a SATA card lying around and try it. Give me a day or two.

Gordan

P.S.

Congrats on your MP30 acquisition, it is an amazing board, it makes my bulk recompiles much more livable with. :-)

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/85e6e7fc-8c52-b15e-281b-75685f5050db%40dewberryfields.co.uk.

[Gordan Bobic]

OK, to answer your question - seems to work fine now, with my 4.9.x kernel series:

http://ftp.redsleeve.org/pub/misc/kernel/aarch64/RPMS/

Disclaimer: I only tested with the latest one, 4.9.196.

# lspci
0000:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)
0000:01:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
0002:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)
0002:01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
0002:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
0003:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)

The Silicon Image line is the PCIe SATA controller.

It looks like some drivers are more problematic than others, though. A Marvell based SATA card still causes a kernel panic:

[   40.732955] sata_mv 0000:01:00.0: enabling device (0000 -> 0003)
[   40.739066] Unhandled fault: synchronous external abort (0x96000010) at 0xffffff8046401d64
[   40.747310] Internal error: : 96000010 [#1] SMP
[   40.751823] Modules linked in: sysimgblt(E) sata_mv(E+) xgene_enet(E) gpio_xgene_sb(E) sg(E) mdio_xgene(E) gpio_generic(E) xgene_rng(E) uio_pdrv_genirq(E) uio(E) lz4(E) lz4_compress(E) zram(E) binfmt_misc(E) ip_tables(E) zfs(POE) zunicode(POE) zlua(POE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) i2c_xgene_slimpro(E) ahci_xgene(E) i2c_core(E) libahci_platform(E)
[   40.784821] CPU: 1 PID: 1726 Comm: systemd-udevd Tainted: P           OE   4.9.196-1.el7.aarch64 #1
[   40.793826] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Feb 22 2016
[   40.801102] task: ffffffdea620c880 task.stack: ffffffdea62a4000
[   40.807002] PC is at mv_init_host+0xe8/0x6d8 [sata_mv]
[   40.812124] LR is at mv_pci_init_one+0x300/0x544 [sata_mv]
[   40.817585] pc : [<ffffff80018b8b28>] lr : [<ffffff80018b99dc>] pstate: 60000145
[   40.824949] sp : ffffffdea62a7970
[   40.828248] x29: ffffffdea62a7970 x28: 0000000000000018
[   40.833557] x27: ffffff80018bc068 x26: ffffff80018bc070
[   40.838865] x25: ffffff80018bd500 x24: 0000000000028000
[   40.844172] x23: ffffffdf7948c0a0 x22: ffffff8046400000
[   40.849479] x21: ffffffdea6fa4518 x20: ffffffdea6fa4a18
[   40.854786] x19: 0000000000000710 x18: 0000007fcf7054e0
[   40.860093] x17: 0000007f89ce72e8 x16: 0000000000000000
[   40.865401] x15: ffffffffffffffff x14: 0000000000000000
[   40.870709] x13: 00000000000001ee x12: 0000000000000030
[   40.876014] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[   40.881321] x9 : feff6b61735e6672 x8 : ffffffffffffffff
[   40.886629] x7 : 000000000000000f x6 : 00000000fffffff4
[   40.891936] x5 : ffffffdea62a77e8 x4 : 000000000000000a
[   40.897244] x3 : ffffff80018b8ac0 x2 : 0000000000000006
[   40.902550] x1 : ffffff8046401d60 x0 : ffffff8046401d64
[   40.907858]
[   40.909342] Process systemd-udevd (pid: 1726, stack limit = 0xffffffdea62a4020)
[   40.916618] Stack: (0xffffffdea62a7970 to 0xffffffdea62a8000)
[   40.922339] 7960:                                   ffffffdea62a79c0 ffffff80018b99dc
[   40.930133] 7980: ffffffdea6fa4a18 0000000000000004 ffffffdea46f0000 ffffffdea6fa4518
[   40.937928] 79a0: ffffffdf7948c000 0000000000028000 ffffff80018bd500 ffffff80018bc070
[   40.945723] 79c0: ffffffdea62a7a50 ffffff80084519dc ffffffdf7948c0a0 ffffffdf7948c000
[   40.953517] 79e0: 0000000000000000 ffffff80018bc858 ffffff80018bc7f0 ffffff80018bbb08
[   40.961313] 7a00: 0000000000000000 ffffffdea62a7dd8 ffffff80018c1080 0000000000000140
[   40.969108] 7a20: ffffffdea62a7a50 ffffff80018bbb08 ffffffdf7948c0a0 ffffffdf7948c000
[   40.976905] 7a40: ffffff80018bb890 0000000000000000 ffffffdea62a7a90 ffffff8008544fe0
[   40.984699] 7a60: ffffffdf7948c0a0 0000000000000000 ffffff8008e21000 ffffff800941e000
[   40.992495] 7a80: ffffff80018bc858 0000000000000018 ffffffdea62a7ad0 ffffff80085453ec
[   41.000290] 7aa0: ffffffdf7948c0a0 ffffff80018bc858 ffffffdf7948c100 0000000000000000
[   41.008084] 7ac0: ffffff8008ddf000 ffffff8008e21000 ffffffdea62a7b00 ffffff8008542c6c
[   41.015878] 7ae0: 0000000000000000 ffffff80018bc858 ffffff80085452ec ffffff8008e21000
[   41.023673] 7b00: ffffffdea62a7b50 ffffff8008544994 ffffff80018bc858 ffffffdea6fa5600
[   41.031468] 7b20: ffffff8008db0a50 ffffff80087fe824 ffffffdef3e238d0 0000000000000000
[   41.039263] 7b40: ffffffdef3e238a8 ffffffdf79684d68 ffffffdea62a7b70 ffffff8008544408
[   41.047058] 7b60: ffffff80018bc858 ffffff80018bc858 ffffffdea62a7bb0 ffffff8008546028
[   41.054852] 7b80: ffffff80018bc858 ffffff80018bc558 0000000000000000 ffffff8008d26258
[   41.062648] 7ba0: ffffff8008d26000 0000000000000000 ffffffdea62a7be0 ffffff8008450704
[   41.070443] 7bc0: ffffff80018bc7f0 ffffff80018bc558 ffffff80018bc7f0 ffffffded0116a00
[   41.078237] 7be0: ffffffdea62a7c10 ffffff80018c102c ffffff80018bd200 000000018020001f
[   41.086032] 7c00: ffffff80018bc040 ffffff80018bd200 ffffffdea62a7c40 ffffff8008083b6c
[   41.093827] 7c20: ffffff80018c1000 ffffffdea62a4000 0000000000000000 ffffffdea6fa4500
[   41.101623] 7c40: ffffffdea62a7cb0 ffffff80081c7848 ffffff80018bd200 ffffffdea5b02000
[   41.109416] 7c60: ffffff8008d26000 ffffff8008d26258 ffffff80018bd200 ffffff80018bd200
[   41.117211] 7c80: ffffffdea62a7e58 ffffff8008d26258 ffffff8008d26000 0000000000000000
[   41.125005] 7ca0: 0000000000000000 ffffffdea62a7dd8 ffffffdea62a7ce0 ffffff800814b374
[   41.132798] 7cc0: ffffff80018bd218 ffffff80018bd200 ffffffdea62a7e58 ffffff8008d26258
[   41.140593] 7ce0: ffffffdea62a7e10 ffffff800814b88c ffffffdea62a7e58 0000000000000000
[   41.148387] 7d00: 0000000000000008 0000007f89cc7910 0000000080000000 0000000000000015
[   41.156181] 7d20: 0000000000000123 0000000000000111 ffffff8008822000 ffffffdea62a4000
[   41.163975] 7d40: 0000000000000072 000000000000006e ffffff8008a62720 000000000000003f
[   41.171771] 7d60: 000000000000feff 000000000000fff1 ffffff80018bd510 ffffff8008a62678
[   41.179564] 7d80: ffffff8008e1bab0 ffffff8008e1b830 0000000000000008 ffffffdea62a7eb4
[   41.187358] 7da0: ffffffdea62a7e68 0000000000000015 0000000000000123 ffffff80080d001c
[   41.195151] 7dc0: ffffffdea62a7e10 ffffff800814b854 ffffffdea62a7e58 0000000000000000
[   41.202947] 7de0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   41.210740] 7e00: 0000000000000000 0000000000000000 0000000000000000 ffffff8008083940
[   41.218535] 7e20: fffffffffffffeee 0000007f89cc7910 ffffffffffffffff 0000007f89b83774
[   41.226329] 7e40: 0000000000000123 0000000000011388 ffffff80463dd000 ffffff80463dd000
[   41.234125] 7e60: 0000000000011388 ffffff80463edb48 ffffff80463eda08 ffffff80463e8608
[   41.238536] [drm] Initialized
[   41.244872] 7e80: 0000000000007510 0000000000008998 0000000000000000 0000000000000000
[   41.252668] 7ea0: 0000000000003428 0000001f0000001e 0000001600000019 0000000000000010
[   41.260464] 7ec0: 0000000000000008 0000007f89cc7910 0000000000000000 0000000000000008
[   41.268258] 7ee0: 0000000000000000 0000000000000000 000000558ff55880 000000558ff55880
[   41.276052] 7f00: 0000000000000111 0000007fcf7056e0 00000000ffffffff 0000000000000007
[   41.283846] 7f20: 0000000000000005 ffffffffffffffff 000000558ff53229 000000000000009b
[   41.291641] 7f40: 0000007f89b83750 0000007f89ce72e8 0000007fcf7054e0 000000558ff55880
[   41.299435] 7f60: 0000007f89cc7910 000000558ff51ff0 0000000000000000 0000000000020000
[   41.307230] 7f80: 000000558ff55030 0000000000000000 0000000000000000 0000000000000000
[   41.315023] 7fa0: 00000055811f7ae0 0000007fcf705710 0000007f89cbfdbc 0000007fcf705710
[   41.322817] 7fc0: 0000007f89b83774 0000000080000000 0000000000000008 0000000000000111
[   41.330613] 7fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   41.338406] Call trace:
[   41.340844] Exception stack(0xffffffdea62a77a0 to 0xffffffdea62a78d0)
[   41.347254] 77a0: 0000000000000710 0000007fffffffff ffffffdea62a7970 ffffff80018b8b28
[   41.355047] 77c0: 0000000000000007 ffffff8000000000 ffffff8046401d64 ffffff80083f2ae0
[   41.362842] 77e0: ffffffdea46f21d6 3030303238303030 ffffffdea62a6632 ffffff8008ad4850
[   41.370636] 7800: ffffffdea62a7880 ffffff80083f3100 0000000000000035 000000000000001b
[   41.378430] 7820: ffffff8008ad4848 ffffffdea6fa4518 ffffffdf7948c000 0000000000028000
[   41.386225] 7840: ffffff8046401d64 ffffff8046401d60 0000000000000006 ffffff80018b8ac0
[   41.394020] 7860: 000000000000000a ffffffdea62a77e8 00000000fffffff4 000000000000000f
[   41.401814] 7880: ffffffffffffffff feff6b61735e6672 7f7f7f7f7f7f7f7f 0101010101010101
[   41.409609] 78a0: 0000000000000030 00000000000001ee 0000000000000000 ffffffffffffffff
[   41.417402] 78c0: 0000000000000000 0000007f89ce72e8
[   41.422265] [<ffffff80018b8b28>] mv_init_host+0xe8/0x6d8 [sata_mv]
[   41.428420] [<ffffff80018b99dc>] mv_pci_init_one+0x300/0x544 [sata_mv]
[   41.434921] [<ffffff80084519dc>] pci_device_probe+0xa0/0x114
[   41.440555] [<ffffff8008544fe0>] driver_probe_device+0xe0/0x3ec
[   41.446447] [<ffffff80085453ec>] __driver_attach+0x100/0x118
[   41.452080] [<ffffff8008542c6c>] bus_for_each_dev+0x6c/0xac
[   41.457628] [<ffffff8008544994>] driver_attach+0x30/0x38
[   41.462914] [<ffffff8008544408>] bus_add_driver+0x1f0/0x294
[   41.468462] [<ffffff8008546028>] driver_register+0x70/0x110
[   41.474008] [<ffffff8008450704>] __pci_register_driver+0x60/0x6c
[   41.479993] [<ffffff80018c102c>] mv_init+0x2c/0x80 [sata_mv]
[   41.485626] [<ffffff8008083b6c>] do_one_initcall+0x44/0x138
[   41.491174] [<ffffff80081c7848>] do_init_module+0x68/0x1cc
[   41.496635] [<ffffff800814b374>] load_module+0xf48/0x11d8
[   41.502008] [<ffffff800814b88c>] SyS_finit_module+0xb8/0xe0
[   41.507554] [<ffffff8008083940>] el0_svc_naked+0x34/0x38
[   41.512842] Code: 91008001 91009000 f9003681 f9003a80 (b9400000)
[   41.518910] ---[ end trace 28b77eee2f184519 ]---
[   41.523507] Kernel panic - not syncing: Fatal exception
[   41.528707] SMP: stopping secondary CPUs
[   41.532614] Kernel Offset: disabled
[   41.536086] Memory Limit: none
[   41.539125] ---[ end Kernel panic - not syncing: Fatal exception

I suggest you try the standard CentOS kernel if mine aren't working for you.

Did you have a specific card in mind?

Gordan

[Michael Howard]

On 16/10/2019 16:50, Gordan Bobic wrote:

OK, to answer your question - seems to work fine now, with my 4.9.x kernel series:

http://ftp.redsleeve.org/pub/misc/kernel/aarch64/RPMS/

Disclaimer: I only tested with the latest one, 4.9.196.

Ok, great, I'll give that a go.

# lspci
0000:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)
0000:01:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
0002:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)
0002:01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
0002:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
0003:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge (rev 04)

The Silicon Image line is the PCIe SATA controller.

It looks like some drivers are more problematic than others, though. A Marvell based SATA card still causes a kernel panic:

I have tried a number of cards over the last couple of years but never got the combination of card/kernel right. Just tried a new Marvell based card and that panics, as did a few other cards in the past, along with an LSI based Dell H800. Some cards I tried in the past would not panic the kernel but still not host any disks.

I'll dig some out and document what does what.

Thanks for the info, appreciated.

Mike.

--

[Gordan Bobic]

I suggest you try the official CentOS kernel. For whatever reason, that wasn't panicking for me on most cards. I never quite got to the bottom of why that worked when my kernels didn't.

I have a suspicion it could be due to a compound issue (more than one setting playing part), and a part of it could be that my kernels use a 4KB page size while the CentOS kernels use 64KB page size, but changing just that one option didn't make my mainline kernels stop panicking.

Please do post back with any insight you gain into the issue and if you need me to cross-check anything on mine.

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/9bd30dda-64ee-e336-faee-5b710c1ed347%40dewberryfields.co.uk.

[Michael Howard]

On 16/10/2019 17:24, Gordan Bobic wrote:
> I suggest you try the official CentOS kernel. For whatever reason,
> that wasn't panicking for me on most cards. I never quite got to the
> bottom of why that worked when my kernels didn't.
> I have a suspicion it could be due to a compound issue (more than one
> setting playing part), and a part of it could be that my kernels use a
> 4KB page size while the CentOS kernels use 64KB page size, but
> changing just that one option didn't make my mainline kernels stop
> panicking.
> Please do post back with any insight you gain into the issue and if
> you need me to cross-check anything on mine.

I haven't managed to get anything to work as yet. The system boots fine
with most all cards I've tried so far, even the Dell H800. They just
need to be in the bottom slot. However, no card yet is prepared to
actually serve a drive to me.

On a different note, in my trials I have tried a number of kernels and
even flavours of kernels and I've noticed that the mmc slot is not
detected after 4.5. Anybody else seen this?

Cheers,

Mike.

[Gordan Bobic]

On Sun, Oct 20, 2019 at 7:57 PM 'Michael Howard' via redsleeve-users <redslee...@googlegroups.com> wrote:

On 16/10/2019 17:24, Gordan Bobic wrote:
> I suggest you try the official CentOS kernel. For whatever reason,
> that wasn't panicking for me on most cards. I never quite got to the
> bottom of why that worked when my kernels didn't.
> I have a suspicion it could be due to a compound issue (more than one
> setting playing part), and a part of it could be that my kernels use a
> 4KB page size while the CentOS kernels use 64KB page size, but
> changing just that one option didn't make my mainline kernels stop
> panicking.
> Please do post back with any insight you gain into the issue and if
> you need me to cross-check anything on mine.

I haven't managed to get anything to work as yet. The system boots fine
with most all cards I've tried so far, even the Dell H800. They just
need to be in the bottom slot. However, no card yet is prepared to
actually serve a drive to me.

If you have tried it with CentOS and the CentOS distro kernel, and you still can't get it working, I suggest you raise the issue on the CentOS list.

 

On a different note, in my trials I have tried a number of kernels and
even flavours of kernels and I've noticed that the mmc slot is not
detected after 4.5. Anybody else seen this?

I have not noticed, but then again I haven't looked. I'll check later when I'm on the network I can get to mine from.

Gordan 

[Michael Howard]

On 16/10/2019 17:24, Gordan Bobic wrote:

> I suggest you try the official CentOS kernel. For whatever reason,
> that wasn't panicking for me on most cards. I never quite got to the
> bottom of why that worked when my kernels didn't.
> I have a suspicion it could be due to a compound issue (more than one
> setting playing part), and a part of it could be that my kernels use a
> 4KB page size while the CentOS kernels use 64KB page size, but
> changing just that one option didn't make my mainline kernels stop
> panicking.
> Please do post back with any insight you gain into the issue and if
> you need me to cross-check anything on mine.
>

Ok, finally have something that works. I have a Supermicro AOC-SASLP-MV8
PCIe SAS Card (bought used off fleabay) installed in the lower pcie
slot, which gives me an extra 8 sata/sas ports.

# lspci
0000:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge
(rev 04)

0001:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge
(rev 04)
0001:01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI
Bridge (rev 03)
0001:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED
Graphics Family (rev 30)

0002:00:00.0 PCI bridge: Applied Micro Circuits Corp. X-Gene PCIe bridge
(rev 04)

0002:01:00.0 SCSI storage controller: Marvell Technology Group Ltd.
MV64460/64461/64462 System Controller, Revision B (rev 01)

Mike.

--
Michael Howard

[Gordan Bobic]

Great info, thanks for sharing. Interesting that it works.

I just got a new Epyc workstation and discovered that the support for all of my SAS controllers (LSI and 3ware) was dropped in EL8. Elrepo seem to have kmods for some of the removed SAS controllers, but modprobing the driver insta-kills the machine to the point where hard reset is required.

It makes me wonder if the issue is a large scale abandonment of various hardware support (e.g. all non-UEFI hardware regardless of CPU architecture).

I think I'll give that SAS controller a shot if my experiment with PMPs turns into a bust.

Just out of interest, you mention you are using it in the low PCIe slot - is that to say it's the only one you tried (e.g. position convenience) or does it fail to work in the other slots?

Gordan

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/438dd8a0-c74b-6c33-f3e9-154b195e2320%40dewberryfields.co.uk.

[Michael Howard]

On 23/11/2019 17:12, Gordan Bobic wrote:
> Great info, thanks for sharing. Interesting that it works.
> I just got a new Epyc workstation and discovered that the support for
> all of my SAS controllers (LSI and 3ware) was dropped in EL8. Elrepo
> seem to have kmods for some of the removed SAS controllers, but
> modprobing the driver insta-kills the machine to the point where hard
> reset is required.
> It makes me wonder if the issue is a large scale abandonment of
> various hardware support (e.g. all non-UEFI hardware regardless of CPU
> architecture).
>
> I think I'll give that SAS controller a shot if my experiment with
> PMPs turns into a bust.
>
> Just out of interest, you mention you are using it in the low PCIe
> slot - is that to say it's the only one you tried (e.g. position
> convenience) or does it fail to work in the other slots?
>
>

Haven't tried it in the upper slot as when I started testing, the first
card (can't remember which card now) caused a panic in the upper slot
but not in the lower one. Still wouldn't pass disks through though.

I will give this a card a go in the upper slot for completeness.

Mike.

--
Michael Howard

[Gordan Bobic]

What driver does that SAS controller use?

Reason I ask is because I am finding that when running on a pure 64-bit UEFI system (and our aarch64 boards fit the description), all of my SATA and SAS controllers result in kernel panics or other inexplicable insta-crashes.

It looks like the problem is much broader than just aarch64, as I'm seeing it on a brand new x86-64 pure-UEFI system.

If this is a systematic issue, it would also go a long way to explaining why EL8 x86-64 dropped nearly all SATA and SAS controllers from the shipped driver set.

I'd be curious to find out if the driver for the SAS controler you got working is in fact one of the very few that remain supported in EL8.

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/038073cf-55e4-cf5c-f65f-871eb456b2a2%40dewberryfields.co.uk.

[Gordan Bobic]

I've been struggling a bit with older (i.e. vaguely sensibly priced) SAS controllers on pure-UEFI x86-64 systems. I suspect the reasons for this are similar to why various storage controllers aren't working quite right on aarch64, too.

I got a Supermicro AOC-SAS2LP-MV8 (note the "2" in there, next generation up from the one you said works for you), and with legacy ROMs completely disabled, it would randomly error out - random disks would fall off the bus at random times, causing the ZFS pool to suspend. I haven't been able to find any obvious pattern to it, it seems to happen when the virtual machines are doing a lot of I/O via both Samba shares on that pool or when there is a zvol on the pool exported to VMs. I couldn't trigger it by doing ZFS scrubbing which I would expect to generate way more disk I/O.

For an unrelated troubleshooting reason I enabled legacy ROM support in the BIOS, and to my surprise, the problem of disks falling off the bus seems to have gone away.

I'm going to hazard a guess that the card's BIOS does something at boot time that initialises the hardware in a way that the driver alone does not. With the BIOS not running at boot time, the card ends up in a slightly dodgy state. It isn't all that surprising considering these "legacy" devices and their drivers were probably never tested with their ROMs disabled (because on a non-UEFI system, why would they be).

I guess the main point is that there is likely to be a lot of similar pain going on with various add-in cards on anything but legacy x86 systems, and the most likely ones to work are the ones that don't have an option ROM to be hooked at all (by either legacy BIOS or UEFI).

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/438dd8a0-c74b-6c33-f3e9-154b195e2320%40dewberryfields.co.uk.

[Gordan Bobic]

I finally had to take the plunge myself in getting PCIe cards and disk controller specifically working on my MP30-AR0, and I have some successes to report.

1) MP30-AR0 can easily be flashed into MP30-AR1

Update the BMC firmware to the latest version. (3.62)

Download the MP30-AR1 BIOS from Gigabyte, extract and make sure the MD5 checksum matches the file.

Go to BMC web interface, select to update the ROM (as opposed to BMC), and select the MP30-AR1 BIOS.

Upload, confirm, wait.

It will, worryingly, ask for "part 2 of the ROM" (!!). While I was thinking about it I just hit the "upload" button, without selecting any files (there were no files to select anyway). I left it for half an hour just to make sure, before I reconnected to the BMC, powered it on, and it "just worked".

Note: The AMI firmware from the AR1 takes _forever_ to boot. We are talking 5 minutes from power on to the point where it comes up with a splash screen on the VGA output. It outputs tons and tons of what looks like debug output on the IPMI serial console, but it will eventually boot. Don't panic. I thought I bricked it with the firmware update.

2) With the AMI firmware, I no longer have problems with _ANY_ PCIe cards. Before I was chainloading Tianocore from u-boot, and any PCIe card plugged in resulted in an insta-crash of the kernel the moment the driver for the card loads. With the AMI firmware, it all "just works".

I am using a Marvell 88SX7042 4-port SATA controller with the sata_mv driver, with no issues. Before I would reliably get a hard crash as soon as the driver loads, with any PCIe card I ever tried to use.

The main downside is that the boot time approximately tripled. The AMI firmware POST takes _forever_.

I haven't been able to get my hands on the flashable Tianocore firmware image for the MP30-AR0 - the only link I can find is a dropbox, so I haven't tried flashing Tianocore directly instead of chain loading it.

Another thing to note - flashing the AMI firmware will reset the MAC addresses on all the NICs to 00:11:22:33:44:56-59 instead of what they were originally. Make sure you note down what they were before if you want to preserve them (they are also written on the sticker on the port cover, but that can be hard to get to, in my case under a PSU).

Anyway, now ALL of my SATA and SAS controllers work perfectly in the MP30-AR0/1.

HTH.

Gordan

--
You received this message because you are subscribed to the Google Groups "redsleeve-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redsleeve-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/redsleeve-users/038073cf-55e4-cf5c-f65f-871eb456b2a2%40dewberryfields.co.uk.