Hardware RAID or software for ZFS

[b...@todoo.biz]

Hi,

I would need your help in order to figure out which will be the best in order to setup a server which will be used as a Poudriere server.

My question is related to the usage of hardware RAID controller (JBOD) or software based controller (graid)

Server we are targeting is a SuperMicro 1018R-WC0R - 1U

I used to build my server using an HBA such as LSI SAS 9207-8i or LSI SAS 9201-16i
Then I’ll configure the HBA as JBOD so that disks can be presented correctly to the system so that we can build our ZFS FS on top of that.

My sysadmin suggested me to get rid of the HBA and use direct connectivity and Graid.

I would like to have some feedback on various aspects :

1. performance
2. usability
3. problem one might have encountered

Thanks for your feedback.

G.B.

________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Your provider of OpenSource Appliances

www.osnet.eu
________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
PGP ID --> 0x1BA3C2FD

_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Arthur Chance]

On 18/11/2014 15:28, b...@todoo.biz wrote:
> Hi,
>
>
> I would need your help in order to figure out which will be the best
> in order to setup a server which will be used as a Poudriere server.
>
> My question is related to the usage of hardware RAID controller (JBOD)
> or software based controller (graid)
>
> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>
> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI
> SAS 9201-16i Then I’ll configure the HBA as JBOD so that disks can be
> presented correctly to the system so that we can build our ZFS FS on
> top of that.
>
>
> My sysadmin suggested me to get rid of the HBA and use direct
> connectivity and Graid.
>
>
> I would like to have some feedback on various aspects :
>
> 1. performance
> 2. usability
> 3. problem one might have encountered

If you're intending to use ZFS don't use JBOD or graid, just tell zpool
to create a pool out of individual disks using raidz (or raidz[23] if
you have enough disks).

As for your hardware choice, I'll let others with more experience than
me answer.

[Andrei Brezan]

On 11/18/14 16:28, b...@todoo.biz wrote:
> Hi,
>
>
> I would need your help in order to figure out which will be the best in order to setup a server which will be used as a Poudriere server.
>
> My question is related to the usage of hardware RAID controller (JBOD) or software based controller (graid)
>
> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>
> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI SAS 9201-16i
> Then I’ll configure the HBA as JBOD so that disks can be presented correctly to the system so that we can build our ZFS FS on top of that.
>
>
> My sysadmin suggested me to get rid of the HBA and use direct connectivity and Graid.

If your disks are SATA you can use the disks in a similar fashion as
JBOD from a PCI-E controller as the onboard controller and backplane
should recognize them. If they are SAS than you need a controller for
them. For ZFS I don't think that anything than exposing the disk as is
to ZFS gives any benefits, ZFS is designed to write directly on disk
without any raid between it and the hardware layer. The caching
algorithm of a raid system might tell ZFS that data was written to disk
when data is still in the controller's cache for example.

I've never heard of a setup with ZFS on top of gmirror or gstripe but my
gut feeling is that they might not play well together as they try to
address similar, if not identical, issues.

>
> I would like to have some feedback on various aspects :
>
> 1. performance
> 2. usability
> 3. problem one might have encountered
>
>
>
> Thanks for your feedback.
>
> G.B.

Regards,

Andrei

[Andrew Berg]

On 2014.11.18 09:28, b...@todoo.biz wrote:
> Hi,
>
>
> I would need your help in order to figure out which will be the best in order to setup a server which will be used as a Poudriere server.
>
> My question is related to the usage of hardware RAID controller (JBOD) or software based controller (graid)

Use neither. ZFS does its own RAID and it works much, much more efficiently if
it doesn't have some other system below it lying about what is going on with
the disks. At best, your other RAID solution will waste cycles, and at worst,
ZFS will constantly try to fix all the changes that happen underneath and the
two will constantly be fighting over what the correct state of the disks is. In
that case, ZFS often loses and you will be unable to access your data because,
as far as ZFS knows, gremlins are flipping bits at random and destroying your data.

[Andrei Brezan]

On 11/18/14 17:01, Arthur Chance wrote:
> On 18/11/2014 15:28, b...@todoo.biz wrote:
>> Hi,
>>
>>
>> I would need your help in order to figure out which will be the best
> > in order to setup a server which will be used as a Poudriere server.
>>
>> My question is related to the usage of hardware RAID controller (JBOD)
> > or software based controller (graid)
>>
>> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>>
>> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI
> > SAS 9201-16i Then I’ll configure the HBA as JBOD so that disks can be
> > presented correctly to the system so that we can build our ZFS FS on
> > top of that.
>>
>>
>> My sysadmin suggested me to get rid of the HBA and use direct
> > connectivity and Graid.
>>
>>
>> I would like to have some feedback on various aspects :
>>
>> 1. performance
>> 2. usability
>> 3. problem one might have encountered
>
> If you're intending to use ZFS don't use JBOD or graid, just tell
> zpool to create a pool out of individual disks using raidz (or
> raidz[23] if you have enough disks).
>
> As for your hardware choice, I'll let others with more experience than
> me answer.
>

JBOD is presenting the disk directly to the OS so it is well suited for
a ZFS setup. There are still raid controllers that have both raid
capabilities and JBOD although some manufacturers try really hard to
separate them for $ome reason. Most of the times you can re-write the
firmware on such a device and have either raid capabilities or JBOD.

[b...@todoo.biz]

> Le 18 nov. 2014 à 17:10, Andrei Brezan <andr...@gmail.com> a écrit :
>
>
> On 11/18/14 17:01, Arthur Chance wrote:
>> On 18/11/2014 15:28, b...@todoo.biz wrote:
>>> Hi,
>>>
>>>
>>> I would need your help in order to figure out which will be the best
>> > in order to setup a server which will be used as a Poudriere server.
>>>
>>> My question is related to the usage of hardware RAID controller (JBOD)
>> > or software based controller (graid)
>>>
>>> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>>>
>>> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI
>> > SAS 9201-16i Then I’ll configure the HBA as JBOD so that disks can be
>> > presented correctly to the system so that we can build our ZFS FS on
>> > top of that.
>>>
>>>
>>> My sysadmin suggested me to get rid of the HBA and use direct
>> > connectivity and Graid.
>>>
>>>
>>> I would like to have some feedback on various aspects :
>>>
>>> 1. performance
>>> 2. usability
>>> 3. problem one might have encountered
>>
>> If you're intending to use ZFS don't use JBOD or graid, just tell zpool to create a pool out of individual disks using raidz (or raidz[23] if you have enough disks).
>>
>> As for your hardware choice, I'll let others with more experience than me answer.
>>
>
> JBOD is presenting the disk directly to the OS so it is well suited for a ZFS setup. There are still raid controllers that have both raid capabilities and JBOD although some manufacturers try really hard to separate them for $ome reason. Most of the times you can re-write the firmware on such a device and have either raid capabilities or JBOD.
>

Ok.

So with SAS disks, I’ll most probably need a controller / HBA with JBOD capabilities ?
Like that disks will be presented directly to the OS where I’ll handle the RAIDZ as needed.

Is that correct ?

Or is there any other « native » option ?

Thx very much.

________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Your provider of OpenSource Appliances

www.osnet.eu
________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
PGP ID --> 0x1BA3C2FD

_______________________________________________

[Matthew Seaman]

On 11/18/14 15:28, b...@todoo.biz wrote:
> I would need your help in order to figure out which will be the best in order to setup a server which will be used as a Poudriere server.
>
> My question is related to the usage of hardware RAID controller (JBOD) or software based controller (graid)
>
> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>
> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI SAS 9201-16i
> Then I’ll configure the HBA as JBOD so that disks can be presented correctly to the system so that we can build our ZFS FS on top of that.
>
>
> My sysadmin suggested me to get rid of the HBA and use direct connectivity and Graid.
>
>
> I would like to have some feedback on various aspects :
>
> 1. performance
> 2. usability
> 3. problem one might have encountered
>

OK -- Your sysadmin is correct about the direct connectivity thing, but
graid? Seriously? That's an option for really cheap and some might say
quite nasty pseudo-raid things you get on some desktop hardware. If
you're going to be building a significant number of packages with any
regularity, then you're going to need something better than that.

The LSI HBAs you suggest are known to work well with FreeBSD. If you're
going to be using ZFS -- which is really a no-brainer for a poudriere
box, as poudriere has a lot of functionality built around cloning and
snapshotting and other things that come naturally to ZFS[*] -- well, in
that case, ideally you want ZFS to have direct access to the disk devices.

With LSI, when they say 'JBOD' this direct access is generally what they
mean -- although depending on the precise model you get, you may need to
flash the device with different firmware to get the characteristics you
want. Some other RAID controller manufacturers essentially give you a
load of single-disk RAID-0 devices, so there's still a layer of RAID
controller stuff between ZFS and the disk. That isn't the kiss of death
to the idea of building a zpool from it, but it's less optimal.

On the three aspecs you mention:

performance depends very much on the precise type of disk hardware
you specify, how you configure the zpool (for instance, will you be
using ZIL or ARC devices?) and on having enough RAM in the system to
provide an effective buffer-cache for all those ZFS filesystems.

usability. I'd say ZFS is pretty usable. Once it's set up, you
don't really need to interact with things at the level of the HBA. It
just works.

problems: this depends very much on the details of what you're trying
to do. But IMHO you're on the right track with what you've described so
far.

Cheers,

Matthew

[*] Although poudriere does now work on UFS as well and has done for a
year or so.

[b...@todoo.biz]

Thank you very much for this documented and precise answer.

I think that the hardware we were targeting is a bit too new and is not yet marked as « ok » for FreeBSD by SuperMicro.
We will probably go for a little bit older hardware that’s 100% known to work ok with FreeBSD.

I have already been building quite large NAS using this setup (JBOD Controller + ZFS on top) and I had very interesting perfs.
I didn’t know about the problem with native JBOD and controller SW update/upgrade…

> SuperMicro 1018R-WC0R - 1U

This is the one we wanted to target but It doesn’t seem to be on the hardware list marked as FreeBSD compatible.

Thanks all - long leave FBSD.

;-)

>
>
>
>

________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

BSD - BSD - BSD - BSD - BSD - BSD - BSD - BSD -

[Warren Block]

On Tue, 18 Nov 2014, Matthew Seaman wrote:

> On 11/18/14 15:28, b...@todoo.biz wrote:
>> I would need your help in order to figure out which will be the best in order to setup a server which will be used as a Poudriere server.
>>
>> My question is related to the usage of hardware RAID controller (JBOD) or software based controller (graid)
>>
>> Server we are targeting is a SuperMicro 1018R-WC0R - 1U
>>
>> I used to build my server using an HBA such as LSI SAS 9207-8i or LSI SAS 9201-16i

>> Then I?ll configure the HBA as JBOD so that disks can be presented correctly to the system so that we can build our ZFS FS on top of that.

>>
>>
>> My sysadmin suggested me to get rid of the HBA and use direct connectivity and Graid.
>>
>>
>> I would like to have some feedback on various aspects :
>>
>> 1. performance
>> 2. usability
>> 3. problem one might have encountered
>>
>
> OK -- Your sysadmin is correct about the direct connectivity thing, but
> graid? Seriously? That's an option for really cheap and some might say
> quite nasty pseudo-raid things you get on some desktop hardware.

The implementations from different vendors no doubt vary in quality
quite a bit. However, if a motherboard has a "software RAID device"
(the euphemism we picked for the Handbook), those additional ports can
almost certainly be used as plain non-RAID ports also. Essentially,
it's the same as JBOD mode on a real RAID controller.

So one way to look at this is based on how many ports are available on
the motherboard. If there are not enough ports for the drive array,
another controller is needed anyway. If there are enough ports, and the
onboard controller can be set to non-RAID, then the option to use just
the motherboard is available.