Functional RAID controller?
Barrett Lyon
are completely non-functional. I had my team working directly with
3ware for two weeks and they offered a tip here and there.
Ultimately we ended up with a non-functioning systems and a good idea
what's wrong with the twa driver.
I offered to help fund and provide hardware for a development effort
to update their driver to support HEAD but they refused to help. I
am worried about 3ware's commitment to the future of FreeBSD and the
twa driver at this point, 3ware is not the least bit concerned (even
with a large customer):
> From: x...@amcc.com
> Subject: RE: 3ware driver issues-Machine to be dropped at 3ware
> Date: May 7, 2007 5:14:04 PM PDT
>
> Good Afternoon. Sorry for the delay in getting back to everyone. We
> have been discussing this internally, and ultimately BSD 7 is
> unreleased, unstable OS at this time.
>
> I am sorry, but this is not something we can put resources in at this
> time and if you bring a systems here, it could not be worked on right
> away either. It would be weeks/months before we can work on this...
I would like to find a SATA RAID controller that plays nice by a
company that is FreeBSD friendly and progressively working on their
drivers. Does anyone have any suggestions other than to not use the
7.0 branch?
-Barrett
Rainer Duffner
I don't think there are a lot of companies who do care about FreeBSD7
(or FreeBSD at all).
For many, it's rather a nuisance (from my perspective). They only
support it, because a significant portion of their sales depends on it.
Can anybody make a statement about the status of the Areca driver?
Because that's the only other non-SCSI controller I know that is worth
having.
I think some of the higher-end MSI-controllers are also good. And of
course, the HP P600 and P800 controllers - but to fully make use of
those two, you need a special enclosure anyway....
cheers,
Rainer
Rink Springer
> I would like to find a SATA RAID controller that plays nice by a
> company that is FreeBSD friendly and progressively working on their
> drivers. Does anyone have any suggestions other than to not use the
> 7.0 branch?
I'd suggest you look at the Areca series of controllers, supported by
arcmsr(4). They are actively maintained, and have been rock solid for
me. Performance is outstanding. I must say I have not tried the
controller under CURRENT, but as the driver seems to be MFC'ed every now
and then.
--
Rink P.W. Springer - http://rink.nu
"It is such a quiet thing, to fall. But yet a far more terrible thing,
to admit it." - Darth Traya
Mike Tancsa
>>From: x...@amcc.com
>>Subject: RE: 3ware driver issues-Machine to be dropped at 3ware
>>Date: May 7, 2007 5:14:04 PM PDT
>>
>>Good Afternoon. Sorry for the delay in getting back to everyone. We
>>have been discussing this internally, and ultimately BSD 7 is
>>unreleased, unstable OS at this time.
>>
>>I am sorry, but this is not something we can put resources in at this
>>time and if you bring a systems here, it could not be worked on right
>>away either. It would be weeks/months before we can work on this...
>
>I would like to find a SATA RAID controller that plays nice by a
>company that is FreeBSD friendly and progressively working on their
>drivers. Does anyone have any suggestions other than to not use the
>7.0 branch?
Are they saying FreeBSD in general, or HEAD specifically ? i.e.
would they commit resources to fixing issues in a RELEASE branch, but
not a development branch ?
---Mike
adam radford
If you have "a good idea what's wrong with the twa driver", would you mind
sharing a stack trace or other information? So far I have only been told that
"system hangs when I do heavy I/O". This is _not_ reproducable here.
Have you run memtest86 on the machine? Have you run a PCI analyzer on
your machine to see who is on the PCI bus before/during the hang?
When I talked to your associates, they didn't have options DDB, KDB, or WITNESS
even compiled in.
You claim the hang doesn't happen on the 6.2 series twa driver,
the driver changes between the 6.x and 7.x twa driver are _very_ minimal,
some simple time keeping changes, and some XPT_* path inquiry handling
changes.
3ware/AMCC is still very much so supporting BSD and we have many customers.
I have sent a patch to Scott Long for the 6.X tree very recently (like
2 weeks ago)
to support the 9650SE series controllers.
I am sorry that you are frustrated. There are plans to make a patch
update to the 7.0
FreeBSD tree, but we cannot send it at this time.
I am really surprised that you are trying to design servers around the
FreeBSD un-stable
kernel.
-Adam
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>
Scott Long
The following companies have SATA RAID controllers that work well with
FreeBSD. Most of these companies also have staff that actively work on
FreeBSD drivers and apps, and are thus motivated to help people with
problems:
Areca
LSI
HP (CISS)
Adaptec
Highpoint
Scott
Scott Long
That was true in the past, but is much less true now. Areca, HighPoint,
and Adaptec are showing a high amount of interesting FreeBSD these days.
> For many, it's rather a nuisance (from my perspective). They only
> support it, because a significant portion of their sales depends on it.
> Can anybody make a statement about the status of the Areca driver?
It works very well.
> Because that's the only other non-SCSI controller I know that is worth
> having.
See my previous email. I would recommend each of those companies; some
have strengths in certain attributes over others, but they all tend to
work well with FreeBSD.
Scott
Barrett Lyon
> Are they saying FreeBSD in general, or HEAD specifically ? i.e.
> would they commit resources to fixing issues in a RELEASE branch,
> but not a development branch ?
Mike, they are working on the 6.x RELEASE branch code actively but it
sounds like they are over committed on those resources and are
refusing to work on the development branch support.
-Barrett
Barrett Lyon
> you mind
> sharing a stack trace or other information? So far I have only
> been told that
> "system hangs when I do heavy I/O". This is _not_ reproducable here.
> Have you run memtest86 on the machine? Have you run a PCI analyzer on
> your machine to see who is on the PCI bus before/during the hang?
We have done everything including asking to bring the machines that
are crashing to AMCC's offices which are down the street. I have not
been doing the technical debugging but a few members of AMCC's staff
have been trying to help. We've been running memtest, etc. When the
machines hang there are no debugging options, it's completely frozen
without any details pointing to why. Its not clear from that
condition whether the problem is due to an unacknowledged interrupt
or a mutex deadlock of some sort. We are assuming that in this case
it is due to the driver trying to do work assuming the interrupt is
valid and getting stuck or returning early before the interrupt is
acknowledged, causing it to trigger over and over and over.
If you want to see it reproduced, we are more than happy to provide
you two machines that both have this condition.
> You claim the hang doesn't happen on the 6.2 series twa driver,
> the driver changes between the 6.x and 7.x twa driver are _very_
> minimal,
> some simple time keeping changes, and some XPT_* path inquiry handling
> changes.
Under 6.x the systems as built function completely stable.
> I am really surprised that you are trying to design servers around the
> FreeBSD un-stable kernel.
There are other reasons for this which I don't want to discuss here,
but the other components we are using work very well within 7.0 and
we have a lot of performance gains that make it worth using a
development kernel. The 10GbE drivers like mxge are having a lot of
development work done in HEAD and as a result the 6.x is getting left
behind on some of the work we are doing. At the very least, I want
to make sure I deploy hardware that will function beyond 6.x.
-Barrett
Scott Long
The biggest difference between 7-CURRENT and 6-STABLE right now in this
space is the MPSAFE work in CAM. It should have been a complete NO-OP
for the 3ware driver, but it's always possible that either I overlooked
something, or the driver was doing something screwy before that was
unsafe, and it's now being caught.
I'll look at this tonight, as well as look at committing the update that
Adam mentioned (sorry Adam!). My 3ware hardware inventory is very
limited, so if I can't spot the problem by code inspection then I'll
need to work with you and Adam to help narrow it down.
Scott
Kip Macy
> The biggest difference between 7-CURRENT and 6-STABLE right now in this
> space is the MPSAFE work in CAM. It should have been a complete NO-OP
> for the 3ware driver, but it's always possible that either I overlooked
> something, or the driver was doing something screwy before that was
> unsafe, and it's now being caught.
>
> I'll look at this tonight, as well as look at committing the update that
> Adam mentioned (sorry Adam!). My 3ware hardware inventory is very
> limited, so if I can't spot the problem by code inspection then I'll
> need to work with you and Adam to help narrow it down.
In fairness, if you care about network bandwidth more than stability,
HEAD is the place to be. On my hardware if_mxge can get 9.3Gbps and
if_cxgb can get full line rate. if_mxge isn't even in RELENG_6 and
if_cxgb performance is at least 25% worse on RELENG_6.
-Kip
Barrett Lyon
> HEAD is the place to be. On my hardware if_mxge can get 9.3Gbps and
> if_cxgb can get full line rate. if_mxge isn't even in RELENG_6 and
> if_cxgb performance is at least 25% worse on RELENG_6.
I can concur, that's why there is so much pressure to use HEAD, it's
a substantial difference and all the network performance is found in
HEAD, but it's useless if my disk arrays crash after writing some
logs. :)
-Barrett
Julian Elischer
write them to a 6.2 machine using NFS :-)
Scott Long
That statement is a bit overly broad and can lead to bad rumors :-)
Most storage drivers work pretty well right now. So I can understand your
feustration with your case, and hopefully Adam and I will have a resolution soon.
Scott
Scott Long
> I've been using HEAD with 3ware adapters and all of my test machines are
> completely non-functional. I had my team working directly with 3ware
> for two weeks and they offered a tip here and there. Ultimately we
> ended up with a non-functioning systems and a good idea what's wrong
> with the twa driver.
>
> I offered to help fund and provide hardware for a development effort to
> update their driver to support HEAD but they refused to help. I am
> worried about 3ware's commitment to the future of FreeBSD and the twa
> driver at this point, 3ware is not the least bit concerned (even with a
> large customer):
>
I have a pretty good idea of what is wrong, and it's partially my fault.
A quick work-around would be to edit /sys/dev/twa/tw_osl_freebsd.c and
remove the INTR_MPSAFE flag as so:
--- tw_osl_freebsd.c 9 May 2007 04:16:32 -0000 1.7
+++ tw_osl_freebsd.c 9 May 2007 04:54:24 -0000
@@ -359,7 +359,7 @@
return(ENXIO);
}
if ((error = bus_setup_intr(sc->bus_dev, sc->irq_res,
- INTR_TYPE_CAM | INTR_MPSAFE,
+ INTR_TYPE_CAM,
#ifdef TW_OSLI_DEFERRED_INTR_USED
twa_pci_intr_fast, NULL,
#else
If that works for you then I'll check it into CVS and work with
AMCC on a real fix.
Scott
Boris Samorodov
Hi Scott, All!
On Tue, 08 May 2007 22:55:06 -0600 Scott Long wrote:
> I have a pretty good idea of what is wrong, and it's partially my fault.
> A quick work-around would be to edit /sys/dev/twa/tw_osl_freebsd.c and
> remove the INTR_MPSAFE flag as so:
I have today's current (with sys/cam/cam_xpt.c 1.187):
-----
FreeBSD 7.0-CURRENT #0: Thu May 17 16:56:58 MSD 2007
bs...@tinderbox.amd64.ipt.ru:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
ACPI APIC Table: <PTLTD APIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (2000.01-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,<b15>,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
usable memory = 8576622592 (8179 MB)
avail memory = 8289337344 (7905 MB)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
...
3ware device driver for 9000 series storage controllers, version: 3.70.03.006
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem 0xd8000000-0xd9ffffff,0xda300000-0xda300fff irq 16 at device 0.0 on pci8
twa0: [FILTER]
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 ports, Firmware FE9X 3.06.00.005, BIOS BE9X 3.06.00.002
...
panic: blockable sleep lock (sleep mutex) CAM SIMQ lock @ /usr/src/sys/cam/cam_xpt.c:4847
cpuid = 0
KDB: enter: panic
[thread pid 17 tid 100007 ]
Stopped at kdb_enter+0x2f: nop
db> where
Tracing pid 17 tid 100007 td 0xffffff021eeaa540
kdb_enter() at kdb_enter+0x2f
panic() at panic+0x225
witness_checkorder() at witness_checkorder+0x5db
_mtx_lock_flags() at _mtx_lock_flags+0x75
xpt_done() at xpt_done+0xca
tw_osl_complete_io() at tw_osl_complete_io+0x10c
tw_cli_complete_io() at tw_cli_complete_io+0x80
tw_cli_process_complete_queue() at tw_cli_process_complete_queue+0xa3
tw_cli_process_resp_intr() at tw_cli_process_resp_intr+0x1e3
twa_pci_intr_fast() at twa_pci_intr_fast+0x2b
intr_execute_handlers() at intr_execute_handlers+0x126
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff8065a9b6, rsp = 0xffffffffac2a7be0, rbp = 0xffffffffac2a7bf0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a0
sched_idletd() at sched_idletd+0x35
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2a7d30, rbp = 0 ---
db>
-----
Is that a known problem you are working on? If it may help I can
arrange an access to comconsole on that server.
Thanks!
WBR
--
Boris Samorodov (bsam)
Research Engineer, http://www.ipt.ru Telephone & Internet SP
FreeBSD committer, http://www.FreeBSD.org The Power To Serve
adam radford
This should be fixed with the latest patch I sent to Scott Long to
turn off deferred
interrupt handling in the twa driver. CVS up until you get twa driver
v3.70.03.007.
-Adam
Boris Samorodov
On Mon, 21 May 2007 15:37:37 -0700 you wrote:
> This should be fixed with the latest patch I sent to Scott Long to
> turn off deferred
> interrupt handling in the twa driver. CVS up until you get twa driver
> v3.70.03.007.
Got, thanks!
Barrett Lyon
changes in place that has stabilized the use of the 'twa' 3ware
driver on HEAD. We've been running a lot of performance tests and we
really like what we see, both on the performance and on stability.
Cheers,
-Barrett