Switching from MFI to MRSAS

[Tim Gustafson]

As I mentioned in my previous post last week, I have a new Dell R630
server that uses a Perc controller, and I have installed 4 SSD drives
and configured the PERC controller in pass-through mode. However, the
PERC controller and its drives are being handled by the "mfi" driver,
which apparently does not support TRIM.

I have been reading up on all this, and I see that there is also an
"mrsas" driver that (according to the documentation) appears to be
compatible with my PERC controller, which is identified as follows:

mfi0: <Fury> port 0x2000-0x20ff mem
0x92000000-0x9200ffff,0x91f00000-0x91ffffff irq 26 at device 0.0 on
pci2
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
mfi0: FW MaxCmds = 240, limiting to 128
mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0

I see that there's a tunable for /boot/device.hints that I can set to
prefer mrsas over mfi:

hw.mfi.mrsas_enable="1"

and then add this to /boot/loader.conf:

mrsas_load="yes"

So, my question is: if I add those options and then reboot, will my
drives magically re-appear as /dev/da* devices? Will the data on them
still be accessible as-is, or would I need to re-install the OS to get
this to go? Would switching to this other driver enable the TRIM
command, as the devices will now be /dev/da* rather than
/dev/mfisyspd*? Or am I just being naive here? :)

--

Tim Gustafson
t...@ucsc.edu
831-459-5354
Baskin Engineering, Room 313A
_______________________________________________
freeb...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Michael Fuckner]

On 9/23/2015 1:26 AM, Tim Gustafson wrote:
> As I mentioned in my previous post last week, I have a new Dell R630
> server that uses a Perc controller, and I have installed 4 SSD drives
> and configured the PERC controller in pass-through mode. However, the
> PERC controller and its drives are being handled by the "mfi" driver,
> which apparently does not support TRIM.
>
> I have been reading up on all this, and I see that there is also an
> "mrsas" driver that (according to the documentation) appears to be
> compatible with my PERC controller, which is identified as follows:
>

Hi,

mfi is the old driver for 3GBit and 6GBit (Single Core) Raid Cards
(102108 based)
mrsas is the newer driver for 6GBit (Dual Core) and 12GBit Cards (2208/
3008 based)

Different driver for different LSI Chipsets

Sorry to disappoint you, regards,
Michael

[Tim Gustafson]

> Different driver for different LSI Chipsets

That's not what the man page says. The man page says:

"Older MegaRAID controllers are supported by mfi(4) and will not work
with mrsas, but both the mfi(4) and mrsas drivers can detect and
manage the LSI MegaRAID SAS 2208/2308/3008/3108 series of
controllers."

And goes on to say:

"The device.hints(5) option is provided to tune the mrsas driver's
behavior for LSI MegaRAID SAS 2208/2308/3008/3108 controllers. By
default, the mfi(4) driver will detect these controllers. See the
PRIORITY section to know more about driver priority for MR-Fusion
devices. mrsas will provide a priority of (-30) (between
BUS_PROBE_DEFAULT and BUS_PROBE_LOW_PRIORITY) at probe call for device
id's 0x005B, 0x005D, and 0x005F so that mrsas does not take control of
these devices without user intervention."

The FreeBSD hardware support page specifically says that the "mrsas"
driver will handle the Dell PERC H330 controller (which is what I
have):

https://www.freebsd.org/relnotes/CURRENT/hardware/support.html

So that's what I'm asking about. I have a newer card, which according
to the man page is supported by the mrsas driver. So my question is,
if I "flip the switch" so that the mrsas driver takes over, will
everything "just" work, or will I need to re-format my disks because
the "mfi" driver configures the disks in a way that is somehow
incompatible with the "mrsas" driver.

--

Tim Gustafson
t...@ucsc.edu
831-459-5354
Baskin Engineering, Room 313A

[Michael Fuckner]

On 9/23/2015 5:55 PM, Tim Gustafson wrote:
>> Different driver for different LSI Chipsets
>
> That's not what the man page says. The man page says:
>
> "Older MegaRAID controllers are supported by mfi(4) and will not work
> with mrsas, but both the mfi(4) and mrsas drivers can detect and
> manage the LSI MegaRAID SAS 2208/2308/3008/3108 series of
> controllers."
>

OK, haven't done this before, perhaps this works, but... you are
replacing a raid driver with another raid driver, so probably TRIM will
not work either.

> So that's what I'm asking about. I have a newer card, which according
> to the man page is supported by the mrsas driver. So my question is,
> if I "flip the switch" so that the mrsas driver takes over, will
> everything "just" work, or will I need to re-format my disks because
> the "mfi" driver configures the disks in a way that is somehow
> incompatible with the "mrsas" driver.

the on disk format should be compatible.

Good luck,
Michael!

[Tim Gustafson]

> OK, haven't done this before, perhaps this works, but... you are replacing a
> raid driver with another raid driver, so probably TRIM will not work either.

The mrsas driver man page says that it reports the disks as "da0",
"da1" and so on, and the man pages about TRIM support all say that it
works with "ada" and "da" drives. So, I was hoping that the change in
driver technology would enable the TRIM command to work.

I suppose I could update device.hints and loader.conf as describe,
reboot, and if the system doesn't come back up, just boot off a thumb
drive and re-edit those files back to their defaults...

--

Tim Gustafson
t...@ucsc.edu
831-459-5354
Baskin Engineering, Room 313A

[Freddie Cash]

On Wed, Sep 23, 2015 at 9:59 AM, Tim Gustafson <t...@ucsc.edu> wrote:

> > OK, haven't done this before, perhaps this works, but... you are
> replacing a
> > raid driver with another raid driver, so probably TRIM will not work
> either.
>
> The mrsas driver man page says that it reports the disks as "da0",
> "da1" and so on, and the man pages about TRIM support all say that it
> works with "ada" and "da" drives. So, I was hoping that the change in
> driver technology would enable the TRIM command to work.
>
> I suppose I could update device.hints and loader.conf as describe,
> reboot, and if the system doesn't come back up, just boot off a thumb
> drive and re-edit those files back to their defaults...

​Or, reboot the system, and drop to the loader prompt. Then manually type
in those loader.conf entries at the loader prompt. And continue the boot
process.

If it fails, then just reboot, and everything will come up normally.

If it succeeds, then you can edit /boot/loader.conf to make the change
permanent.

You'll want to boot to single-user mode as well, as /etc/fstab will point
to the wrong devices (unless you are using some kind of generic labels for
the entries instead of the physical device node names).

In single-user mode, you can update /etc/fstab and /boot/loader.conf as
needed if the new driver works.​

--
Freddie Cash
fjw...@gmail.com

[Matthew Seaman]

On 23/09/2015 16:55, Tim Gustafson wrote:
> The FreeBSD hardware support page specifically says that the "mrsas"
> driver will handle the Dell PERC H330 controller (which is what I
> have):
>
> https://www.freebsd.org/relnotes/CURRENT/hardware/support.html
>

> So that's what I'm asking about. I have a newer card, which according
> to the man page is supported by the mrsas driver. So my question is,
> if I "flip the switch" so that the mrsas driver takes over, will
> everything "just" work, or will I need to re-format my disks because
> the "mfi" driver configures the disks in a way that is somehow
> incompatible with the "mrsas" driver.

We've a bunch of Dell machines of the H730 cards, which are (I think)
the same chipset but with various additional bits like a BBU and nvram
and so forth.

Yes, if mfi(8) detects the controller but doesn't work (or doesn't
perform) then the mrsas driver is the thing to use. It works fine as a
RAID controller, but there is no equivalent of mfiutil(8) for these cards.

Trim support -- or the lack of it -- is apparently a feature of the
firmware on Dell-branded SSD drives.

Cheers,

Matthew

[Ryan Stone]

On Wed, Sep 23, 2015 at 12:59 PM, Tim Gustafson <t...@ucsc.edu> wrote:

> The mrsas driver man page says that it reports the disks as "da0",
> "da1" and so on, and the man pages about TRIM support all say that it
> works with "ada" and "da" drives. So, I was hoping that the change in
> driver technology would enable the TRIM command to work.
>
> I suppose I could update device.hints and loader.conf as describe,
> reboot, and if the system doesn't come back up, just boot off a thumb
> drive and re-edit those files back to their defaults...
>

The mrsas driver is interesting. Your RAID volumes show up as da0, da1,
etc, but they really shouldn't. I'm doubtful that TRIM will work here
because the drives are not truly directly attached, even though the mrsas
driver is misleading the kernel into thinking that they are.

You can always give a try, though.

[Tim Gustafson]

> The mrsas driver is interesting. Your RAID volumes show up as da0, da1,
> etc, but they really shouldn't. I'm doubtful that TRIM will work here
> because the drives are not truly directly attached, even though the mrsas
> driver is misleading the kernel into thinking that they are.
>
> You can always give a try, though.

Just to be clear: we have the PERC controller configured in
"pass-through" mode, which I understand to mean "HBA" mode. We are
not using any of the RAID features of the PERC card. Instead, the
server is configured as ZFS-on-root in a RAID 0+1 configuration:

pool: tank
state: ONLINE
scan: scrub repaired 14.5K in 0h13m with 0 errors on Thu Sep 17 20:47:17 2015
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
mfisyspd0p3 ONLINE 0 0 0
mfisyspd1p3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
mfisyspd3p3 ONLINE 0 0 0
mfisyspd2p3 ONLINE 0 0 0

where mfisyspd0, mfisyspd1, mfisyspd2 and mfisyspd3 are Samsung SSDs,
not RAID volumes.

So in this case, I believe it would be "correct" for the RAID
controller to report the devices as "da0", "da1" and so on.

--

Tim Gustafson
t...@ucsc.edu
831-459-5354
Baskin Engineering, Room 313A

[Steven Hartland]

On 23/09/2015 22:32, Ryan Stone wrote:
> On Wed, Sep 23, 2015 at 12:59 PM, Tim Gustafson <t...@ucsc.edu> wrote:
>
>> The mrsas driver man page says that it reports the disks as "da0",
>> "da1" and so on, and the man pages about TRIM support all say that it
>> works with "ada" and "da" drives. So, I was hoping that the change in
>> driver technology would enable the TRIM command to work.
>>
>> I suppose I could update device.hints and loader.conf as describe,
>> reboot, and if the system doesn't come back up, just boot off a thumb
>> drive and re-edit those files back to their defaults...
>>

> The mrsas driver is interesting. Your RAID volumes show up as da0, da1,
> etc, but they really shouldn't. I'm doubtful that TRIM will work here
> because the drives are not truly directly attached, even though the mrsas
> driver is misleading the kernel into thinking that they are.
>
> You can always give a try, though.

camcontrol identify da0 will give you an indication

ZFS will tell you straight away after a file delete:
sysctl kstat.zfs.misc.zio_trim

Regards
Steve

[Marco.re]

On 09/23/2015 01:26, Tim Gustafson wrote:
> mfi0: <Fury> port 0x2000-0x20ff mem
> 0x92000000-0x9200ffff,0x91f00000-0x91ffffff irq 26 at device 0.0 on
> pci2
> mfi0: Using MSI
> mfi0: Megaraid SAS driver Ver 4.23
> mfi0: FW MaxCmds = 240, limiting to 128
> mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0
>
> I see that there's a tunable for /boot/device.hints that I can set to
> prefer mrsas over mfi:
>
> hw.mfi.mrsas_enable="1"
>
> and then add this to /boot/loader.conf:
>
> mrsas_load="yes"
>
> So, my question is: if I add those options and then reboot, will my
> drives magically re-appear as /dev/da* devices? Will the data on them
> still be accessible as-is, or would I need to re-install the OS to get
> this to go? Would switching to this other driver enable the TRIM
> command, as the devices will now be /dev/da* rather than
> /dev/mfisyspd*? Or am I just being naive here? :)

I'm not sure, I've installed one Dell R730 with a new controller H730/P,
I'm not sure if the mfi driver was seeing the raid, I don't remember :-(
if you have this controller
# pciconf -lv
mrsas0@pci0:2:0:0: class=0x010400 card=0x1f491028 chip=0x005d1000
rev=0x02 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'MegaRAID SAS-3 3108 [Invader]'
class = mass storage
subclass = RAID

it works with the mrsas driver.

# cat /boot/loader.conf
hw.mfi.mrsas_enable=1
mrsas_load="yes"



# dmesg | grep -i mrsas
module_register: module pci/mrsas already exists!
Module pci/mrsas failed to register: 17
LSI MegaRAID SAS FreeBSD mrsas driver version: 06.704.01.01-fbsd
mrsas0: <LSI Invader SAS Controller> port 0x2000-0x20ff mem

0x92000000-0x9200ffff,0x91f00000-0x91ffffff irq 26 at device 0.0 on pci2

mrsas0: FW supports: UnevenSpanSupport=1
mrsas0: Internal command timed out after 180 seconds.
da0 at mrsas0 bus 0 scbus0 target 0 lun 0


Cioa Marco

[Michael Fuckner]

On 9/28/2015 4:00 PM, Marco.re wrote:

> # dmesg | grep -i mrsas
> module_register: module pci/mrsas already exists!
> Module pci/mrsas failed to register: 17
> LSI MegaRAID SAS FreeBSD mrsas driver version: 06.704.01.01-fbsd
> mrsas0: <LSI Invader SAS Controller> port 0x2000-0x20ff mem
> 0x92000000-0x9200ffff,0x91f00000-0x91ffffff irq 26 at device 0.0 on pci2
> mrsas0: FW supports: UnevenSpanSupport=1
> mrsas0: Internal command timed out after 180 seconds.
> da0 at mrsas0 bus 0 scbus0 target 0 lun 0
>

OK, we see Devices da instead of /dev/mfispd... but does it really do
trim now?