LSI 9260: is there a way to configure it JBOD like mps?

[Dmitry Morozovsky]

Dear colleagues,

I'm currently in process of making new backup server, based on LSI 9260
controller. I'm planning to use ZFS over disks, hence the most natural way
seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
neither in BIOS utilities nor via MegaCli

Any hints?

Thanks!

--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: ma...@FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru ***
------------------------------------------------------------------------
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Tom Evans]

On Tue, Mar 19, 2013 at 5:45 PM, Dmitry Morozovsky <ma...@rinet.ru> wrote:
> Dear colleagues,
>
> I'm currently in process of making new backup server, based on LSI 9260
> controller. I'm planning to use ZFS over disks, hence the most natural way
> seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
> neither in BIOS utilities nor via MegaCli
>
> Any hints?
>
> Thanks!
>

9260 should be SAS-2008 based, so mps(4) not mfi(4).

The internet[1] suggests that this card should be flashable to a
9211-8i with IT mode firmware, which is just about the ultimate ZFS
card, instant-JBOD on inserting a disk, passthru for SMART, high
performance, etc.

Cheers

Tom

[1] http://blog.grem.de/sysadmin/LSI-SAS2008-Flashing-2012-04-12-22-17.html

[Michael Fuckner]

Am 3/19/2013 7:00 PM, schrieb Tom Evans:
> On Tue, Mar 19, 2013 at 5:45 PM, Dmitry Morozovsky <ma...@rinet.ru> wrote:
>> Dear colleagues,
>>
>> I'm currently in process of making new backup server, based on LSI 9260
>> controller. I'm planning to use ZFS over disks, hence the most natural way
>> seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
>> neither in BIOS utilities nor via MegaCli

there is no such way :(

> 9260 should be SAS-2008 based, so mps(4) not mfi(4).

IMHO 9260 is a megaraid card (with Cache and BBU Option) based on 2108,
not 2008 Chipset. It is not flashable to an HBA. This wouldn't make
sense anyway since HBAs are way cheaper than Raid Cards.

I also tried to configure megaraid for ZFS- the good thing is that you
can make use of the controller cache. The bad thing is: it is very nasty
to configure: every disk needs to be an own raidset with a Raid0 array.
When a disk fails- the controller complains about unrecoverable
raids/volumes. There is nothing like an HBA mode.

I saw Adaptec claims to hace Controllers behaving like an HBA with
Cache, but I think it still takes some weeks until I have my Card.

Regards,
Michael!

[Dmitry Morozovsky]

On Tue, 19 Mar 2013, Tom Evans wrote:

> > I'm currently in process of making new backup server, based on LSI 9260
> > controller. I'm planning to use ZFS over disks, hence the most natural way
> > seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
> > neither in BIOS utilities nor via MegaCli
>

> 9260 should be SAS-2008 based, so mps(4) not mfi(4).

Well, it at least detected by stable/9 GENERIC as mfi

> The internet[1] suggests that this card should be flashable to a
> 9211-8i with IT mode firmware, which is just about the ultimate ZFS
> card, instant-JBOD on inserting a disk, passthru for SMART, high
> performance, etc.

Will check, thanks for the reference.

> [1] http://blog.grem.de/sysadmin/LSI-SAS2008-Flashing-2012-04-12-22-17.html

>

--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: ma...@FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru ***
------------------------------------------------------------------------

[Jason Keltz]

Hi.
I hope to soon put into production a new file server hosting many ZFS
filesystem with FreeBSD. The system has 2 x 9205-8e cards, and 1 x
9207-8i card and 24 x 900 GB 10K RPM drives. I'm trying to figure out
what is ultimately the "best" version of FreeBSD to run on a production
file server. I believe that it doesn't make sense to stick directly to
the 9.1/release because there have already been many ZFS problems that
were solved in 9.1/stable. On the other hand, stable doesn't
necessarily have to be "stable"! Of course "release" might not be
"stable" either if there's a bug that say, causes a hang on my
controller card, and it's not fixed in anything but "stable"! Yet,
"stable" might "break" something else. I'm wondering what people who
are running FreeBSD file servers in production do -- do you track
individual changes, and compile release + individual bug fixes that
likely affect you, or, in my case, if I run "stable", do all my testing
with "stable", do I run that version of stable, and only attempt to
upgrade to the next "stable" release while very carefully reviewing the
bug list, then holding my breath when the server comes up? Any
recommendations would be appreciated. I know there are a lot of people
who are happily running FreeBSD file servers. :)

Jason.

On 03/19/2013 03:04 PM, Dmitry Morozovsky wrote:
> On Tue, 19 Mar 2013, Tom Evans wrote:
>
>>> I'm currently in process of making new backup server, based on LSI 9260
>>> controller. I'm planning to use ZFS over disks, hence the most natural way
>>> seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
>>> neither in BIOS utilities nor via MegaCli
>> 9260 should be SAS-2008 based, so mps(4) not mfi(4).
> Well, it at least detected by stable/9 GENERIC as mfi
>
>> The internet[1] suggests that this card should be flashable to a
>> 9211-8i with IT mode firmware, which is just about the ultimate ZFS
>> card, instant-JBOD on inserting a disk, passthru for SMART, high
>> performance, etc.
> Will check, thanks for the reference.
>
>> [1] http://blog.grem.de/sysadmin/LSI-SAS2008-Flashing-2012-04-12-22-17.html
>>

[Freddie Cash]

9-STABLE after r248369. That enables:
- feature flags
- lz4 compression
- deadman thread
- various other bug fixes, tweaks, updates, etc

If you don't want to run the very latest stable, the following command is
useful:
svn co -r 248369 svn://svn0.us-west.freebsd.org/base/stable/9 /usr/src

I read through the svn-src-stable-9 mailing list archives to see if there
are any interesting commits for devices or services that I use, and update
to those revisions as needed.

For example, there's some nice mps(4) and zfs-related commits listed here:
http://lists.freebsd.org/pipermail/svn-src-stable-9/2013-March/thread.html

--
Freddie Cash
fjw...@gmail.com

[Ronald Klop]

On Tue, 19 Mar 2013 21:32:02 +0100, Jason Keltz <j...@cse.yorku.ca> wrote:

> Hi.
> I hope to soon put into production a new file server hosting many ZFS
> filesystem with FreeBSD. The system has 2 x 9205-8e cards, and 1 x
> 9207-8i card and 24 x 900 GB 10K RPM drives. I'm trying to figure out
> what is ultimately the "best" version of FreeBSD to run on a production
> file server. I believe that it doesn't make sense to stick directly to
> the 9.1/release because there have already been many ZFS problems that
> were solved in 9.1/stable. On the other hand, stable doesn't
> necessarily have to be "stable"! Of course "release" might not be
> "stable" either if there's a bug that say, causes a hang on my
> controller card, and it's not fixed in anything but "stable"! Yet,
> "stable" might "break" something else. I'm wondering what people who
> are running FreeBSD file servers in production do -- do you track
> individual changes, and compile release + individual bug fixes that
> likely affect you, or, in my case, if I run "stable", do all my testing
> with "stable", do I run that version of stable, and only attempt to
> upgrade to the next "stable" release while very carefully reviewing the
> bug list, then holding my breath when the server comes up? Any
> recommendations would be appreciated. I know there are a lot of people
> who are happily running FreeBSD file servers. :)

I would run 9-RELEASE until there is a really (really really) good reason
to do otherwise.
There is no reason to get h*rny about every feature or every additional
commit if just serving files works really well.

Ronald.

[Jason Keltz]

Hi Ronald,

I'm not at all concerned about new functionality, or even minor bug
fixes to general O/S commands that I likely won't be using on the server
anyway. That obviously leaves "current" out of the question, especially
for a production server. That being said, stability changes with
respect to ZFS (of which there are already several in 9.1-stable as
Freddie pointed out) are what I'm after. I wish there was a better
separation in FreeBSD between the "critical" versus "not so critical"
patches ... something like "release" that I can always download and know
that I can't really "go wrong"... the difference between adding
functionality, and fixing critical bugs in existing functionality. I
might be misunderstanding the whole concept, but it's probably what
puzzles me more than anything about FreeBSD. I'm coming from the RHEL
world where I can rely on vendor binary kernels to fix serious bugs
without "adding" new functionality. Sometimes, things break, but in
general, it's all pretty good...

Jason.



The truth is, on my other machines, I'm using to running RHEL where
there are frequent binary ker

[Rainer Duffner]

> I'm not at all concerned about new functionality, or even minor bug
> fixes to general O/S commands that I likely won't be using on the
> server anyway. That obviously leaves "current" out of the question,
> especially for a production server. That being said, stability
> changes with respect to ZFS (of which there are already several in
> 9.1-stable as Freddie pointed out) are what I'm after. I wish there
> was a better separation in FreeBSD between the "critical" versus "not
> so critical" patches ... something like "release" that I can always
> download and know that I can't really "go wrong"... the difference
> between adding functionality, and fixing critical bugs in existing
> functionality. I might be misunderstanding the whole concept, but
> it's probably what puzzles me more than anything about FreeBSD. I'm
> coming from the RHEL world where I can rely on vendor binary kernels
> to fix serious bugs without "adding" new functionality. Sometimes,
> things break, but in general, it's all pretty good...

AFAIK, the reason why this "stable-stable" version of FreeBSD does not
exist is purely due to the lack of resources (money mostly).
If somebody would pay for it, it would happen.

Personally, I'm glad FreeBSD get's out one release per year but it
would be interesting to know if somebody from the Foundation has done
the maths on this and could come up with an estimate about the sort of
(financial) commitment this would require.

[Steven Hartland]

----- Original Message -----
From: "Dmitry Morozovsky" <ma...@rinet.ru>

> I'm currently in process of making new backup server, based on LSI 9260
> controller. I'm planning to use ZFS over disks, hence the most natural way
> seems to configure mfi to JBOD mode - but I can't find easy way to reach this,
> neither in BIOS utilities nor via MegaCli
>
> Any hints?

I don't remember the model number, but you may find there's an
IT mode FW which will make it report under mps and not mfi.

If that's one of the versions which has no IT mode FW then you
can try configuring JBOD under mfi with something like:-

MegaCli -AdpSetProp -EnableJBOD -1 -aALL

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.

[Dmitry Morozovsky]

On Wed, 20 Mar 2013, Steven Hartland wrote:

> ----- Original Message ----- From: "Dmitry Morozovsky" <ma...@rinet.ru>
>
> > I'm currently in process of making new backup server, based on LSI 9260
> > controller. I'm planning to use ZFS over disks, hence the most natural way
> > seems to configure mfi to JBOD mode - but I can't find easy way to reach
> > this, neither in BIOS utilities nor via MegaCli
> >
> > Any hints?
>
> I don't remember the model number, but you may find there's an
> IT mode FW which will make it report under mps and not mfi.
>
> If that's one of the versions which has no IT mode FW then you
> can try configuring JBOD under mfi with something like:-
>
> MegaCli -AdpSetProp -EnableJBOD -1 -aALL

I did. No luck :(

Well, it seems I'll have to create 12 RAOI0 volumes :-/

--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: ma...@FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru ***
------------------------------------------------------------------------

[Peter Maloney]

On 2013-03-21 11:31, Dmitry Morozovsky wrote:
> Well, it seems I'll have to create 12 RAOI0 volumes :-/

If you do that, you still have vendor lock and all the rest of the
problems. Why not just sell/trade your 1 raid card for 2 or 3 HBAs? If
that's an 8 port one, it's something like $480 and an HBA is only around
$230 (9211-8i). If you plan on permanently using this machine for pure
ZFS, this is the best option.

[Rich]

On Thu, Mar 21, 2013 at 6:31 AM, Dmitry Morozovsky <ma...@rinet.ru> wrote:
> On Wed, 20 Mar 2013, Steven Hartland wrote:
>
>> ----- Original Message ----- From: "Dmitry Morozovsky" <ma...@rinet.ru>
>>
>> > I'm currently in process of making new backup server, based on LSI 9260
>> > controller. I'm planning to use ZFS over disks, hence the most natural way
>> > seems to configure mfi to JBOD mode - but I can't find easy way to reach
>> > this, neither in BIOS utilities nor via MegaCli
>> >
>> > Any hints?
>>
>> I don't remember the model number, but you may find there's an
>> IT mode FW which will make it report under mps and not mfi.
>>
>> If that's one of the versions which has no IT mode FW then you
>> can try configuring JBOD under mfi with something like:-
>>
>> MegaCli -AdpSetProp -EnableJBOD -1 -aALL
>
> I did. No luck :(
>
> Well, it seems I'll have to create 12 RAOI0 volumes :-/

924x will do JBOD, 926x will not.

One of those annoying little caveats for no clear reason.

But yes, I'd imagine you can destructively reflash it from DOS into a
9211-8i if it's the 2008 chipset. [If it's 2108/2208, that's a
different FW but same concept.]

- Rich

[devnullius]

I hope some guru knows? Reviving this thread for I too have a 9260-4i 6GB/S
SATA+SAS MEGARAID RAID PCI Card (L3-25121-61A)

I want to use FreeNAS / ZFS on it and for that, I need JBOD 'raid'. As far
as I understand, I'll need IT Firmware. Which isn't available for 2960-4i
(right?).

So now I'm wondering... (How) could I re-flash it with
http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9211-4i#downloads
- 9211_4i_Package_P20_IR_IT_Firmware_BIOS_for_MSDOS_Windows ? It kinda feels
some expert voodoo would achieve what I want: my own NAS managed with ZFS...

Anyone here that can help?

Peace!

:) Devnullius




--
View this message in context: http://freebsd.1045724.n5.nabble.com/LSI-9260-is-there-a-way-to-configure-it-JBOD-like-mps-tp5796967p6062095.html
Sent from the freebsd-fs mailing list archive at Nabble.com.
_______________________________________________
freeb...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs

[devnullius]

I also got this info from Michelle (thanks!)


Michelle Sullivan wrote
> Well depends on your RAID level of zfs...
>
> On my system I have 16 x 3T drives and all are config'd as RAID0 then I
> have zraid2 for 15 of the disks and the remainder is set to be a spare.
>
> ...and that all looks like:
>
> $ zpool status -v
> pool: storage
> state: ONLINE
> scan: scrub in progress since Sun Dec 20 00:00:01 2015
> 8.13T scanned out of 28.5T at 28.5M/s, 208h38m to go
> 0 repaired, 28.49% done
> config:
>
> NAME STATE READ WRITE CKSUM
> storage ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> mfid10 ONLINE 0 0 0
> mfid8 ONLINE 0 0 0
> mfid12 ONLINE 0 0 0
> mfid0 ONLINE 0 0 0
> mfid14 ONLINE 0 0 0
> mfid15 ONLINE 0 0 0
> mfid1 ONLINE 0 0 0
> mfid7 ONLINE 0 0 0
> mfid2 ONLINE 0 0 0
> mfid9 ONLINE 0 0 0
> mfid3 ONLINE 0 0 0
> mfid4 ONLINE 0 0 0
> mfid5 ONLINE 0 0 0
> mfid6 ONLINE 0 0 0
> mfid11 ONLINE 0 0 0
> spares
> mfid13 AVAIL
>
> errors: No known data errors

It still feels like I should anticipate a lot of problems, assuming worst
case scenarios... So I'm gonna look for one of these cards instead and rule
out all risks...:
https://forums.freenas.org/index.php?threads/it-mode-firmware-for-lsi-megaraid-sas-9260-4i.40093/
(currently in moderation)



--
View this message in context: http://freebsd.1045724.n5.nabble.com/LSI-9260-is-there-a-way-to-configure-it-JBOD-like-mps-tp5796967p6063032.html