Dell branded storage controller for ZFS file-server | Advice requested

[Ahmed Kamal]

Hi everyone,

I'm looking to build a ZFS file-server, unfortunately I'm locked into Dell
branded hardware only. I know Dell sometimes ships LSI branded cards, but
that too probably won't work for me, as I live outside the US and the local
Dell rep is clueless (so whatever I want to buy, has to be on Dell's
website). I am considering Dell R720xd or R730xd. I will use Intel i350
NICs, and ECC RAM, but the biggest questions are around the SAS HBA

Per my research, I have the following notes and questions. Appreciating
your comments on them

* H310 Card supports syspd (jbod) mode per
http://svnweb.freebsd.org/base?view=revision&revision=254906 so it is
probably the best card for what I want to do, right?

* H310 is a low end card, it has very limited queue depth (25 I believe) ..
Is this actually a problem when using the disks (20 x SAS 10k disks) in
syspd/jbod mode? what about if I attach a couple of SSD disks?

* H310 can be flashed to IT mode (LSI firmware), and is supported by
Illumos kernel (if I'll ever need that) .. so overall it looks like a good
option. Any "watch-outs" I should be aware of ?

* H710/p have no way of exposing JBOD/syspd in any way .. If using those
cards, the best I can do, is to create a raid-0 per disk, and put zfs on
top of that. How bad is this? Is the only problem that I'm locked into H710
firmware? Would the nvram and bbu on H710 improve performance significantly
vs H310?

* H330 and H730, seem supported by mrsas driver. Is this driver able to
expose jbod/syspd mode? If yes, what's the overall advise, should I go with
one of those, or the "trusty" H310 ?

Your help and advice is most appreciated
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
To unsubscribe, send any mail to "freebsd-scsi...@freebsd.org"

[David Gwynne]

> On 23 Jan 2015, at 7:16 pm, Ahmed Kamal <email.ah...@googlemail.com> wrote:
>
> Hi everyone,
>
> I'm looking to build a ZFS file-server, unfortunately I'm locked into Dell
> branded hardware only. I know Dell sometimes ships LSI branded cards, but
> that too probably won't work for me, as I live outside the US and the local
> Dell rep is clueless (so whatever I want to buy, has to be on Dell's
> website). I am considering Dell R720xd or R730xd. I will use Intel i350
> NICs, and ECC RAM, but the biggest questions are around the SAS HBA
>
> Per my research, I have the following notes and questions. Appreciating
> your comments on them
>
> * H310 Card supports syspd (jbod) mode per
> http://svnweb.freebsd.org/base?view=revision&revision=254906 so it is
> probably the best card for what I want to do, right?

no.

> * H310 is a low end card, it has very limited queue depth (25 I believe) ..
> Is this actually a problem when using the disks (20 x SAS 10k disks) in
> syspd/jbod mode? what about if I attach a couple of SSD disks?

there are two queue limits here, one for the controller and one for each disk attached to it.

h310s have a controller queue depth of 31, ie, they can only support 31 commands at a time. if you have 24 disks and distribute the controllers command slots between them, thats about 1 command per disk.

> * H310 can be flashed to IT mode (LSI firmware), and is supported by
> Illumos kernel (if I'll ever need that) .. so overall it looks like a good
> option. Any "watch-outs" I should be aware of ?

yes. if you flash the h310 to become an sas hba, the pci product/vendor and subproduct/subvendor ids on the controller change. the dell bios will refuse to boot if it detects an unsupported controller in its storage slot. you'll brick the machine until you remove the h310. because it is on a non standard mezzanine connector you will not be able to flash it back to a h310, and you'll have a useless chunk of silicon to put on your desk as a monument to how annoying vendors are.

> * H710/p have no way of exposing JBOD/syspd in any way .. If using those
> cards, the best I can do, is to create a raid-0 per disk, and put zfs on
> top of that. How bad is this? Is the only problem that I'm locked into H710
> firmware? Would the nvram and bbu on H710 improve performance significantly
> vs H310?

i wouldnt run zfs on that.

> * H330 and H730, seem supported by mrsas driver. Is this driver able to
> expose jbod/syspd mode? If yes, what's the overall advise, should I go with
> one of those, or the "trusty" H310 ?

thats a workable option. h330 should be fine if all you want to do is jbod.

[Ahmed Kamal]

Thanks a lot for your reply .. I've got a couple of questions inline

My initial line of thought was that in pass-through (jbod) mode, the HBA
does *not* buffer commands and proxy disk responses internally. Which meant
that in jbod mode, the HBA basically becomes a communications channel
connecting the OS to disks, making its internal buffering limitations
not-applicable. Is this not correct ?

>
> > * H310 can be flashed to IT mode (LSI firmware), and is supported by
> > Illumos kernel (if I'll ever need that) .. so overall it looks like a
> good
> > option. Any "watch-outs" I should be aware of ?
>
> yes. if you flash the h310 to become an sas hba, the pci product/vendor
> and subproduct/subvendor ids on the controller change. the dell bios will
> refuse to boot if it detects an unsupported controller in its storage slot.
> you'll brick the machine until you remove the h310. because it is on a non
> standard mezzanine connector you will not be able to flash it back to a
> h310, and you'll have a useless chunk of silicon to put on your desk as a
> monument to how annoying vendors are.
>

pff nightmare scenario .. thanks for pointing this out. I stumbled across
web posts where people successfully flashed h310, but meh, now I'm scared
to buy that configuration

>
> > * H710/p have no way of exposing JBOD/syspd in any way .. If using those
> > cards, the best I can do, is to create a raid-0 per disk, and put zfs on
> > top of that. How bad is this? Is the only problem that I'm locked into
> H710
> > firmware? Would the nvram and bbu on H710 improve performance
> significantly
> > vs H310?
>
> i wouldnt run zfs on that.
>
> > * H330 and H730, seem supported by mrsas driver. Is this driver able to
> > expose jbod/syspd mode? If yes, what's the overall advise, should I go
> with
> > one of those, or the "trusty" H310 ?
>
> thats a workable option. h330 should be fine if all you want to do is jbod.
>

Now that you're pointing me to H330 being my best option given the
situation, got some more questions:
* "if all you want is jbod" .. with zfs, that's all we need right ? just
wondering if I'm missing something important
* Any idea if jbod configuration on h330 is done on bios level, or using
some freebsd tool (how?)
* mrsas man page, mentions the driver appeared in 10.1 .. Does that mean if
I decide to use FreeNAS (9.x based) it won't be there ?
* If I go the h330 path, it'll be a r730xd server. Worst case if the h330
gives me trouble, can I throw it out and plug a LSI 9207-8i in its place
(same pci slot, cables ..etc) ?

Thanks!

[David Gwynne]

> On 23 Jan 2015, at 9:41 pm, Ahmed Kamal <email.ah...@googlemail.com> wrote:
>
> Thanks a lot for your reply .. I've got a couple of questions inline
>
> On Fri, Jan 23, 2015 at 12:21 PM, David Gwynne <da...@gwynne.id.au> wrote:
>
> > On 23 Jan 2015, at 7:16 pm, Ahmed Kamal <email.ah...@googlemail.com> wrote:
> >
> > Hi everyone,
> >
> > I'm looking to build a ZFS file-server, unfortunately I'm locked into Dell
> > branded hardware only. I know Dell sometimes ships LSI branded cards, but
> > that too probably won't work for me, as I live outside the US and the local
> > Dell rep is clueless (so whatever I want to buy, has to be on Dell's
> > website). I am considering Dell R720xd or R730xd. I will use Intel i350
> > NICs, and ECC RAM, but the biggest questions are around the SAS HBA
> >
> > Per my research, I have the following notes and questions. Appreciating
> > your comments on them
> >
> > * H310 Card supports syspd (jbod) mode per
> > http://svnweb.freebsd.org/base?view=revision&revision=254906 so it is
> > probably the best card for what I want to do, right?
>
> no.
>
> > * H310 is a low end card, it has very limited queue depth (25 I believe) ..
> > Is this actually a problem when using the disks (20 x SAS 10k disks) in
> > syspd/jbod mode? what about if I attach a couple of SSD disks?
>
> there are two queue limits here, one for the controller and one for each disk attached to it.
>
> h310s have a controller queue depth of 31, ie, they can only support 31 commands at a time. if you have 24 disks and distribute the controllers command slots between them, thats about 1 command per disk.
>
> My initial line of thought was that in pass-through (jbod) mode, the HBA does *not* buffer commands and proxy disk responses internally. Which meant that in jbod mode, the HBA basically becomes a communications channel connecting the OS to disks, making its internal buffering limitations not-applicable. Is this not correct ?

the h310 physically doesnt have cache, so it cant buffer your I/O even if it wanted to.

so yes, in your use case it would just be a hba, but a very constrained one. even most sata controllers (eg, ahci or the silicon image ones) can keep more commands up in the air since they can have up 32 command slots per port instead of 31 for the whole controller.

im also not sure if dell will let you order an r720xd with a h310. i dont think they consider that a supported config.

>
>
> > * H310 can be flashed to IT mode (LSI firmware), and is supported by
> > Illumos kernel (if I'll ever need that) .. so overall it looks like a good
> > option. Any "watch-outs" I should be aware of ?
>
> yes. if you flash the h310 to become an sas hba, the pci product/vendor and subproduct/subvendor ids on the controller change. the dell bios will refuse to boot if it detects an unsupported controller in its storage slot. you'll brick the machine until you remove the h310. because it is on a non standard mezzanine connector you will not be able to flash it back to a h310, and you'll have a useless chunk of silicon to put on your desk as a monument to how annoying vendors are.
>
> pff nightmare scenario .. thanks for pointing this out. I stumbled across web posts where people successfully flashed h310, but meh, now I'm scared to buy that configuration

there are h310s that are normal pci-e cards that go into dells lower end servers. the ones in the r720s sit on the custom mezzanine connector.

> > * H710/p have no way of exposing JBOD/syspd in any way .. If using those
> > cards, the best I can do, is to create a raid-0 per disk, and put zfs on
> > top of that. How bad is this? Is the only problem that I'm locked into H710
> > firmware? Would the nvram and bbu on H710 improve performance significantly
> > vs H310?
>
> i wouldnt run zfs on that.
>
> > * H330 and H730, seem supported by mrsas driver. Is this driver able to
> > expose jbod/syspd mode? If yes, what's the overall advise, should I go with
> > one of those, or the "trusty" H310 ?
>
> thats a workable option. h330 should be fine if all you want to do is jbod.
>
> Now that you're pointing me to H330 being my best option given the situation, got some more questions:
> * "if all you want is jbod" .. with zfs, that's all we need right ? just wondering if I'm missing something important

nope.

> * Any idea if jbod configuration on h330 is done on bios level, or using some freebsd tool (how?)

you can do it in the bios.

> * mrsas man page, mentions the driver appeared in 10.1 .. Does that mean if I decide to use FreeNAS (9.x based) it won't be there ?

i have no idea :)

> * If I go the h330 path, it'll be a r730xd server. Worst case if the h330 gives me trouble, can I throw it out and plug a LSI 9207-8i in its place (same pci slot, cables ..etc) ?

the h330 sits on a mezzanine connector. you probably can get that card and drive the disks with it, but i havent done it with an r730 or 730xd so i cant say for sure. i have done that with an r720 though, and it mostly works.

we had to order those systems with a h310 and add the lsi card ourselves, and then we discovered that the disks they shipped in the system do not spin up when theyre powered on. the raid controller (h310) explicitly spins them up before issuing io to them.

if you attach those disks to a straight sas hba or use them via syspd on the h310, you cant boot off them because the machines bios (not the raid controller) blindly issues io to them without spinning them up first.

fun times.

dlg

[Steven Hartland]

Dells c6000 chassis have the option of LSI controllers, however their
midplane is troublesome and will cause issues at 6Gbps speed, so be careful.

Regards
Steve

[Ahmed Kamal]

comments down below

scary stuff .. so just to be clear:

- LSI SAS hba => disks DONT spin up
- h310 in jbod mode => disks DONT spin up
- h330 in jbod mode => disks work fine ?

Note that currently dell's web configurator, only sells h330 with r730(xd)
not r720(xd) .. The r720 comes with h310 or h710 afaict

Note 2, Illumos people are able to pass a driver parameter to spin up
crappy dell disks .. check
http://www.listbox.com/member/archive/182179/2013/06/sort/from/page/4/entry/15:430/20130617155143:52F0F7EA-D787-11E2-84A9-87C6F7D228D8/

[David Gwynne]

> On 23 Jan 2015, at 11:02 pm, Ahmed Kamal <email.ah...@googlemail.com> wrote:
>
> scary stuff .. so just to be clear:
>
> - LSI SAS hba => disks DONT spin up
> - h310 in jbod mode => disks DONT spin up
> - h330 in jbod mode => disks work fine ?

h330 doing jbod is a supported config from dell, so if it doesnt work should be able to have a little tantrum and get them to fix it or replace bits till it does work.

>
> Note that currently dell's web configurator, only sells h330 with r730(xd) not r720(xd) .. The r720 comes with h310 or h710 afaict

yes.

> Note 2, Illumos people are able to pass a driver parameter to spin up crappy dell disks .. check
> http://www.listbox.com/member/archive/182179/2013/06/sort/from/page/4/entry/15:430/20130617155143:52F0F7EA-D787-11E2-84A9-87C6F7D228D8/

you can patch your kernels as much as you want to, but you cant fix the bios in the machine. if you want to boot off disks like that in a jbod config you'll need ones that spin up when they get power.

dlg

[Steven Hartland]

I believe h330 is a LSI 3008 underneath with 256 QD which is classed my
mrsas as Invader/Fury 12Gb/s MR controller so you should get decent
performance from it, as with anything though only testing will answer
the question categorically ;-)

[kpn...@pobox.com]

On Fri, Jan 23, 2015 at 02:45:56PM +0000, Steven Hartland wrote:
> I believe h330 is a LSI 3008 underneath with 256 QD which is classed my
> mrsas as Invader/Fury 12Gb/s MR controller so you should get decent
> performance from it, as with anything though only testing will answer
> the question categorically ;-)

Just curious: Are jbod disks all bootable? Or does only one show up in the
bios' boot list?

If only one, what happens if that drive mostly fails but still shows up
to the bios? How about if it totally fails and doesn't show up? How about
if the boot process starts but fails after control is passed to the boot
code loaded from disk?

These questions are part of the reason I boot my Dells from the memory
card.

Personally, I wish Dell would stop screwing around with the firmware used
with LSI chips.

--
Kevin P. Neal http://www.pobox.com/~kpn/

"What is mathematics? The age-old answer is, of course, that mathematics
is what mathematicians do." - Donald Knuth

[Steven Hartland]

On 23/01/2015 15:13, kpn...@pobox.com wrote:
> On Fri, Jan 23, 2015 at 02:45:56PM +0000, Steven Hartland wrote:
>> I believe h330 is a LSI 3008 underneath with 256 QD which is classed my
>> mrsas as Invader/Fury 12Gb/s MR controller so you should get decent
>> performance from it, as with anything though only testing will answer
>> the question categorically ;-)
> Just curious: Are jbod disks all bootable? Or does only one show up in the
> bios' boot list?

This is BIOS dependent, most provide support for a number of boot disks.

> If only one, what happens if that drive mostly fails but still shows up
> to the bios? How about if it totally fails and doesn't show up? How about
> if the boot process starts but fails after control is passed to the boot
> code loaded from disk?
>
> These questions are part of the reason I boot my Dells from the memory
> card.
>
> Personally, I wish Dell would stop screwing around with the firmware used
> with LSI chips.

Generally we'll use the on-board controller to provide the boot disk
pool in a simple mirror then configure a large RAIDZ for data, however
that depends on how the physical disks are attached.

This avoids the issue where the disk being chosen to boot from in the
BIOS is not actually bootable.