mps0-troubles
Joachim Tingvold
I'm not sure if this is the proper place to ask for help regarding
this, but here it goes;
I've got 17 disks connected to a HP SAS expander, which again is
connected to a LSI SAS 9211-8i HBA. I also have 1 system-disk that's
connected directly to the SATA-controller on the motherboard. This is
running on FreeBSD 9.0-CURRENT-201012.
I'm running ZFS on root (referred to as "zroot"), and also on the 17
disks connected to the LSI-controller (6x2TB raid-z2 + 10x1TB raid-z +
1 hot-spare, referred to as "storage").
This setup has been running fine since around christmas, but today,
when I was moving some files from the zroot to storage, it failed.
First, the moving went just fine (I was looking at gstat while it was
copying), but then no activity (even though I knew it wasn't done --
there was a lot of large files). Trying to list any files on the
storage-volume didn't work (CTRL+C didn't work either, I had to quit
the terminal). The mv-process was still running, even though there was
no disk-activity;
[jocke@filserver ~]$ ps aux | grep mv
root 33698 0,0 0,1 10048 2132 0- D+ 11:35am
0:01,66 mv -PRp -- JAG /storage/series/JAG (cp)
I've extracted the relevant lines from dmesg since the machine booted
on sunday; <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi.txt>.
After a while (couple of minutes), I could list files on the storage-
volume, and ZFS reported no problems. Then, after a few new minutes, I
could not list anything on the storage-volume, and it's been like that
since (ZFS and dmesg reports no further errors, though).
I mentioned that the mv-process is still running; it won't die, but I
guess that's because it has the D-flag (disk wait).
[root@filserver ~]# kill -9 33698
[root@filserver ~]# ps aux | grep mv
root 33698 0,0 0,1 10048 2132 0- D+ 11:35am 0:01,66 mv -PRp -- JAG /
storage/series/JAG (cp)
This isn't really my field of expertise, so I'm hoping that someone
here on the list might enlighten me. (-:
--
Joachim
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
To unsubscribe, send any mail to "freebsd-scsi...@freebsd.org"
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Alexander Motin
> I'm not sure if this is the proper place to ask for help regarding this,
> but here it goes;
>
> I've got 17 disks connected to a HP SAS expander, which again is
> connected to a LSI SAS 9211-8i HBA. I also have 1 system-disk that's
> connected directly to the SATA-controller on the motherboard. This is
> running on FreeBSD 9.0-CURRENT-201012.
>
> I'm running ZFS on root (referred to as "zroot"), and also on the 17
> disks connected to the LSI-controller (6x2TB raid-z2 + 10x1TB raid-z + 1
> hot-spare, referred to as "storage").
>
> This setup has been running fine since around christmas, but today, when
> I was moving some files from the zroot to storage, it failed. First, the
> moving went just fine (I was looking at gstat while it was copying), but
> then no activity (even though I knew it wasn't done -- there was a lot
> of large files). Trying to list any files on the storage-volume didn't
> work (CTRL+C didn't work either, I had to quit the terminal). The
> mv-process was still running, even though there was no disk-activity;
>
> [jocke@filserver ~]$ ps aux | grep mv
> root 33698 0,0 0,1 10048 2132 0- D+ 11:35am
> 0:01,66 mv -PRp -- JAG /storage/series/JAG (cp)
>
> I've extracted the relevant lines from dmesg since the machine booted on
> sunday; <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi.txt>.
>
> After a while (couple of minutes), I could list files on the
> minutes, I could not list anything on the storage-volume, and it's been
> like that since (ZFS and dmesg reports no further errors, though).
>
> I mentioned that the mv-process is still running; it won't die, but I
> guess that's because it has the D-flag (disk wait).
>
> [root@filserver ~]# kill -9 33698
> [root@filserver ~]# ps aux | grep mv
> root 33698 0,0 0,1 10048 2132 0- D+ 11:35am 0:01,66 mv -PRp -- JAG
>
> This isn't really my field of expertise, so I'm hoping that someone here
> on the list might enlighten me. (-:
dmesg you've shown shown many command timeouts on multiple devices. As
soon as default ATA timeout is about 30 seconds - it may cause
significant delays before recovery sequence will manage it. That could
result in delays you observed.
What's more suspicious is that timeouts happened same time on
AHCI-attached disk and several disks on mps controller. I can hardly
assume that two completely different controllers and drivers triggered
some unrelated problems simultaneously. I would suggest to check your
power supplies, cables, backplanes and other mechanical things.
--
Alexander Motin
Joachim Tingvold
> dmesg you've shown shown many command timeouts on multiple devices. As
> soon as default ATA timeout is about 30 seconds - it may cause
> significant delays before recovery sequence will manage it. That could
> result in delays you observed.
I let the computer stay on for about 7-8 hours after I sent my
previous email, and it was still frozen. I had to physically reset the
computer to get it operational again.
> What's more suspicious is that timeouts happened same time on
> AHCI-attached disk and several disks on mps controller. I can hardly
> assume that two completely different controllers and drivers triggered
> some unrelated problems simultaneously.
If I were copying from the AHCI-attached disk to the mps controller,
and the AHCI-attached disk timeouts, wouldn't this cause the disks on
the mps controller to timeout as well?
--
Joachim
Joachim Tingvold
> If I were copying from the AHCI-attached disk to the mps controller,
> and the AHCI-attached disk timeouts, wouldn't this cause the disks
> on the mps controller to timeout as well?
Now it happened again (while copying from 'zroot' to 'storage'). This
time only mps0 produced errors; <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_2.txt
>. As the timeout seem to be over quickly, I find it strange that
whatever process that accessed the disks (in my case, 'mv'), doesn't
continue once the disks are available -- or is this some kind of
safeguard against corrupted data?
--
Joachim
Alexander Motin
> On Wed, Jan 12, 2011, at 14:19:01PM GMT+01:00, Alexander Motin wrote:
>> soon as default ATA timeout is about 30 seconds - it may cause
>> significant delays before recovery sequence will manage it. That could
>> result in delays you observed.
>
> email, and it was still frozen. I had to physically reset the computer
> to get it operational again.
>
>> AHCI-attached disk and several disks on mps controller. I can hardly
>> assume that two completely different controllers and drivers triggered
>> some unrelated problems simultaneously.
>
> the AHCI-attached disk timeouts, wouldn't this cause the disks on the
> mps controller to timeout as well?
No. Timeout means that disk doesn't respond to request. If you just
stuck on AHCI disk -- mps disks should be just idle.
Alexander Motin
> On Wed, Jan 12, 2011, at 23:29:53PM GMT+01:00, Joachim Tingvold wrote:
>> If I were copying from the AHCI-attached disk to the mps controller,
>> and the AHCI-attached disk timeouts, wouldn't this cause the disks on
>> the mps controller to timeout as well?
>
> Now it happened again (while copying from 'zroot' to 'storage'). This
> time only mps0 produced errors;
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_2.txt>. As the
> timeout seem to be over quickly, I find it strange that whatever process
> that accessed the disks (in my case, 'mv'), doesn't continue once the
> disks are available -- or is this some kind of safeguard against
> corrupted data?
I don't know such "safeguards". I would suppose there is some bug in
recovery process, making some request(s) never finish. As result,
application initiated that request keeps waiting for it.
Kenneth D. Merry
> On Wed, Jan 12, 2011, at 23:29:53PM GMT+01:00, Joachim Tingvold wrote:
> >If I were copying from the AHCI-attached disk to the mps controller,
> >and the AHCI-attached disk timeouts, wouldn't this cause the disks
> >on the mps controller to timeout as well?
>
> Now it happened again (while copying from 'zroot' to 'storage'). This
> time only mps0 produced errors;
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_2.txt >. As the
> timeout seem to be over quickly, I find it strange that whatever process
> that accessed the disks (in my case, 'mv'), doesn't continue once the
> disks are available -- or is this some kind of safeguard against corrupted
> data?
Did the system recover this time?
The 'out of chain frames' messages are somewhat worrysome. From looking at
the logic in the driver (mpssas_action_scsiio() and mps_data_cb()), it
looks like if it runs out of chain frames, it won't cancel the timeout on
the command. So you'll wind up getting timeouts. But sending an abort for
a command that hasn't gone down to the chip is rather pointless.
Did you see any other messages before the 'out of chain frames' messages
popped up?
Try editing sys/dev/mps/mpsvar.h, and change MPS_CHAIN_FRAMES from 1024 to
2048 and see if that helps things any.
That won't fix the underlying problem, but it may help you avoid running
out of that resource.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> Did the system recover this time?
Yes and no. The mv-process is still running, however I can use 'zroot'
without issues. I can list some folders on 'storage', but not all --
most of them makes my terminal freeze. And as mentioned, a warm reboot
doesn't work (I have to physically power-cycle it).
> Did you see any other messages before the 'out of chain frames'
> messages
> popped up?
No, not in my last error-log (only boot-entries on the lines above).
> Try editing sys/dev/mps/mpsvar.h, and change MPS_CHAIN_FRAMES from
> 1024 to
> 2048 and see if that helps things any.
>
> That won't fix the underlying problem, but it may help you avoid
> running
> out of that resource.
This won't "conflict" with what you've written here[1]? The 9211-8i
controller I have, use the SAS2008-chip.
Would it be possible that low amounts of free memory (<100MB, the
system has a total of 4GB) could cause this? (since I'm using ZFS,
which relies heavily on memory).
[1] <http://www.mail-archive.com/svn-sr...@freebsd.org/msg06214.html>
--
Joachim
Kenneth D. Merry
> On Thu, Jan 13, 2011, at 21:37:50PM GMT+01:00, Kenneth D. Merry wrote:
>
> without issues. I can list some folders on 'storage', but not all --
> most of them makes my terminal freeze. And as mentioned, a warm reboot
> doesn't work (I have to physically power-cycle it).
Hmm.
> >Did you see any other messages before the 'out of chain frames'
> >messages
> >popped up?
>
> No, not in my last error-log (only boot-entries on the lines above).
>
> >Try editing sys/dev/mps/mpsvar.h, and change MPS_CHAIN_FRAMES from
> >1024 to
> >2048 and see if that helps things any.
> >
> >That won't fix the underlying problem, but it may help you avoid
> >running
> >out of that resource.
>
> This won't "conflict" with what you've written here[1]? The 9211-8i
> controller I have, use the SAS2008-chip.
No, this affects the number of buffers available for scatter/gather
entries. Most commands will just have one physical memory buffer attached,
since system buffer memory is physically contiguous.
It doesn't affect the number of commands the driver sends down to the
chip.
> Would it be possible that low amounts of free memory (<100MB, the
> system has a total of 4GB) could cause this? (since I'm using ZFS,
> which relies heavily on memory).
I don't think so. If anything, I think that this is likely triggered by
a large number of outstanding commands, or perhaps a leak somewhere. If
it's the former, hopefully this will fix it. If it's the latter, you'll
eventually run into the problem again.
> [1] <http://www.mail-archive.com/svn-sr...@freebsd.org/msg06214.html>
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
>> This won't "conflict" with what you've written here[1]? The 9211-8i
>> controller I have, use the SAS2008-chip.
>
> No, this affects the number of buffers available for scatter/gather
> entries. Most commands will just have one physical memory buffer
> attached,
> since system buffer memory is physically contiguous.
>
> It doesn't affect the number of commands the driver sends down to the
> chip.
I see.
The strange thing is that I copied around 7-8TB at around 105MB/s (via
samba) to 'storage', without ever having this issue. There has been no
hardware-change of any kind since this was done.
>> Would it be possible that low amounts of free memory (<100MB, the
>> system has a total of 4GB) could cause this? (since I'm using ZFS,
>> which relies heavily on memory).
>
> I don't think so. If anything, I think that this is likely
> triggered by
> a large number of outstanding commands, or perhaps a leak
> somewhere. If
> it's the former, hopefully this will fix it. If it's the latter,
> you'll
> eventually run into the problem again.
Okay. I've changed the value. I'll keep you posted.
--
Joachim
Joachim Tingvold
>> I don't think so. If anything, I think that this is likely
>> triggered by
>> a large number of outstanding commands, or perhaps a leak
>> somewhere. If
>> it's the former, hopefully this will fix it. If it's the latter,
>> you'll
>> eventually run into the problem again.
>
> Okay. I've changed the value. I'll keep you posted.
Some more errors; <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_3.txt
>. Same as earlier, but without the "out of chain"-messages (probably
just because I increased the number, so that this specific error-
message isn't printed?).
This happened without direct user-interaction (it's a cron-job that
copies files from 'zroot' to 'storage', and deletes some stuff on
'zroot' afterwards). However, it seems as if I can access the files/
folders on 'storage' without the terminal freezing (based on the log-
messages, I was able to pinpoint the last 2-3 copied folders, and
those I could list/copy files from without the terminal freezing). I'm
not sure why, though. Maybe because the error/crash occurred when no
files where being copied/moved?
--
Joachim
Alexander Motin
> On Fri, Jan 14, 2011, at 01:38:27AM GMT+01:00, Joachim Tingvold wrote:
>>> a large number of outstanding commands, or perhaps a leak somewhere. If
>>> it's the former, hopefully this will fix it. If it's the latter, you'll
>>> eventually run into the problem again.
>>
>> Okay. I've changed the value. I'll keep you posted.
>
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_3.txt>. Same as
> earlier, but without the "out of chain"-messages (probably just because
> printed?).
>
> This happened without direct user-interaction (it's a cron-job that
> copies files from 'zroot' to 'storage', and deletes some stuff on
> 'zroot' afterwards). However, it seems as if I can access the
> log-messages, I was able to pinpoint the last 2-3 copied folders, and
> those I could list/copy files from without the terminal freezing). I'm
> not sure why, though. Maybe because the error/crash occurred when no
> files where being copied/moved?
I would say because error recovery now managed to what it should. If
problem causing timeouts is not permanent, CAM should correctly recover
from them, and system should work further just with some delays for
recovery. In any case processes should not freeze forever as they did --
they should either work (even if slow) or terminate with I/O errors.
Kenneth D. Merry
> On Fri, Jan 14, 2011, at 01:38:27AM GMT+01:00, Joachim Tingvold wrote:
> >>triggered by
> >>a large number of outstanding commands, or perhaps a leak
> >>somewhere. If
> >>it's the former, hopefully this will fix it. If it's the latter,
> >>you'll
> >>eventually run into the problem again.
> >
> >Okay. I've changed the value. I'll keep you posted.
>
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_3.txt >. Same as
> earlier, but without the "out of chain"-messages (probably just because I
> increased the number, so that this specific error- message isn't printed?).
>
> This happened without direct user-interaction (it's a cron-job that
> copies files from 'zroot' to 'storage', and deletes some stuff on
> 'zroot' afterwards). However, it seems as if I can access the files/
> folders on 'storage' without the terminal freezing (based on the log-
> messages, I was able to pinpoint the last 2-3 copied folders, and
> those I could list/copy files from without the terminal freezing). I'm
> not sure why, though. Maybe because the error/crash occurred when no
> files where being copied/moved?
So did the system freeze or crash this time?
Do you have a serial console on the machine?
If it hangs or crashes, perhaps we can get a stack trace.
It does look like the out of chain problem was fixed by increasing the
number, so that's good at least.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> So did the system freeze or crash this time?
It happened while I was not actively using the machine. When I found
out about it, everything seemed to be fine.
> Do you have a serial console on the machine?
> If it hangs or crashes, perhaps we can get a stack trace.
You mean normal RS-232?
> It does look like the out of chain problem was fixed by increasing the
> number, so that's good at least.
Yes. For now, at least. (-:
--
Joachim
Kenneth D. Merry
> On Thu, Jan 20, 2011, at 16:57:46PM GMT+01:00, Kenneth D. Merry wrote:
> >So did the system freeze or crash this time?
>
> It happened while I was not actively using the machine. When I found
> out about it, everything seemed to be fine.
Okay, so that means it didn't panic and didn't lock up permanently, so
things are probably okay.
> >Do you have a serial console on the machine?
> >If it hangs or crashes, perhaps we can get a stack trace.
>
> You mean normal RS-232?
Yes.
> >It does look like the out of chain problem was fixed by increasing the
> >number, so that's good at least.
>
> Yes. For now, at least. (-:
Yes. It looks like you're just getting some timeouts, followed by aborts
and retries. It's not optimal, but at least the system is recovering and
continuing on.
FWIW, there is some much more sophisticated error recovery code coming
for the driver, written by the guys at Isilon. That'll come in when LSI
gets done with their version of the driver.
The error recovery that is in there now is basic, but it works in most of
the common cases. There is a case related to device removal that Justin
fixed that I'm planning to commit in the near future.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
>> It does look like the out of chain problem was fixed by increasing
>> the
>> number, so that's good at least.
>
> Yes. For now, at least. (-:
So, it didn't last for long; <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_4.txt
>.
[jocke@filserver ~]$ cat /sys/dev/mps/mpsvar.h|grep "MPS_CHAIN_FRAMES"
#define MPS_CHAIN_FRAMES 2048
It seems to be happening when copying larger files from 'zroot' to
'storage' (that is, files over 1.5GB in size), or when moving files
right after each other (f.ex. using scripts, or moving folders with
many files of medium size (300MB+)). Moving a file now-and-then (of
sizes no larger than 1.5GB) doesn't seem to trigger it.
--
Joachim
Kenneth D. Merry
> On Fri, Jan 21, 2011, at 20:15:15PM GMT+01:00, Joachim Tingvold wrote:
> >>It does look like the out of chain problem was fixed by increasing
> >>the
> >>number, so that's good at least.
> >
> >Yes. For now, at least. (-:
>
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_4.txt >.
>
> [jocke@filserver ~]$ cat /sys/dev/mps/mpsvar.h|grep
> "MPS_CHAIN_FRAMES"
> #define MPS_CHAIN_FRAMES 2048
>
> It seems to be happening when copying larger files from 'zroot' to
> 'storage' (that is, files over 1.5GB in size), or when moving files
> right after each other (f.ex. using scripts, or moving folders with
> many files of medium size (300MB+)). Moving a file now-and-then (of
> sizes no larger than 1.5GB) doesn't seem to trigger it.
That's strange that you're still running into the problem with that many
chain frames. In my tests, I haven't seen more than 80 chain frames used.
You can probably work around it again by bumping up the number of chain
frames (e.g. to 3072 or 4096), but it would probably be good to make sure
there isn't a leak in the driver somewhere.
I've attached a patch that has a number of debugging sysctls, a change from
gibbs@ that has to do with device removal, and some other extra debugging
cruft. (i.e. this patch won't go into the tree as-is, it's just for
debugging.)
Try running this, and then do 'sysctl hw.mps' and let's see what your low
water mark is for free chain elements. We'll also want to make sure your
chain_free value is about equal to MPS_CHAIN_FRAMES when the system is
idle. On my system with a LSI 9201-16i controller, I see:
hw.mps.1.debug_level: 0
hw.mps.1.allow_multiple_tm_cmds: 0
hw.mps.1.io_cmds_active: 24
hw.mps.1.io_cmds_highwater: 252
hw.mps.1.chain_free: 1024
hw.mps.1.chain_free_lowwater: 948
hw.mps.1.chain_alloc_fail: 0
I know what the root cause is for this bug, I just haven't had time to fix
it. Unfortunately I've been chasing bugs in the driver that are a little
higher priority for $REAL_JOB. The good news at least is that any fixes we
make will go back into FreeBSD.
Joachim Tingvold
> I've attached a patch that has a number of debugging sysctls, a
> change from
> gibbs@ that has to do with device removal, and some other extra
> debugging
> cruft. (i.e. this patch won't go into the tree as-is, it's just for
> debugging.)
Index: mps.c
|===================================================================
|--- mps.c (revision 218241)
|+++ mps.c (working copy)
--------------------------
Patching file mps.c using Plan A...
Hunk #1 succeeded at 386 (offset -1 lines).
Hunk #2 succeeded at 734 (offset -7 lines).
Hunk #3 succeeded at 812 (offset -9 lines).
Hunk #4 failed at 847.
The patch is looking for
SYSCTL_ADD_UINT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
[...]
SYSCTL_ADD_UINT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
[...]
but my mps.c has
SYSCTL_ADD_INT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree), [...]
SYSCTL_ADD_INT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree), [...]
That, and all the offsets; maybe my mps-driver is a bit outdated?
Maybe this is a stupid question, but I'm no FreeBSD-expert. (-:
Kenneth D. Merry
> On Thu, Feb 03, 2011, at 23:10:56PM GMT+01:00, Kenneth D. Merry wrote:
> >I've attached a patch that has a number of debugging sysctls, a
> >change from
> >gibbs@ that has to do with device removal, and some other extra
> >debugging
> >cruft. (i.e. this patch won't go into the tree as-is, it's just for
> >debugging.)
>
> |===================================================================
> |--- mps.c (revision 218241)
> |+++ mps.c (working copy)
> --------------------------
> Patching file mps.c using Plan A...
> Hunk #1 succeeded at 386 (offset -1 lines).
> Hunk #2 succeeded at 734 (offset -7 lines).
> Hunk #3 succeeded at 812 (offset -9 lines).
> Hunk #4 failed at 847.
>
> The patch is looking for
>
> SYSCTL_ADD_UINT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
> [...]
> SYSCTL_ADD_UINT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
> [...]
>
> but my mps.c has
>
> SYSCTL_ADD_INT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
> [...]
> SYSCTL_ADD_INT(&sc->sysctl_ctx, SYSCTL_CHILDREN(sc->sysctl_tree),
> [...]
>
> That, and all the offsets; maybe my mps-driver is a bit outdated?
> Maybe this is a stupid question, but I'm no FreeBSD-expert. (-:
That change went in a few weeks ago. Try this patch, perhaps it will work.
Joachim Tingvold
> That change went in a few weeks ago.
Ahh, okay. I'll try updating it this weekend, then.
> Try this patch, perhaps it will work.
No errors this time, but it didn't seem to do anything;
[root@filserver ~]# sysctl hw.mps
hw.mps.disable_msi: 0
hw.mps.disable_msix: 0
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
(-:
Kenneth D. Merry
> On Fri, Feb 04, 2011, at 01:02:49AM GMT+01:00, Kenneth D. Merry wrote:
> >That change went in a few weeks ago.
>
>
>
>
> [root@filserver ~]# sysctl hw.mps
> hw.mps.disable_msi: 0
> hw.mps.disable_msix: 0
> hw.mps.0.debug_level: 0
> hw.mps.0.allow_multiple_tm_cmds: 0
Hmm. Lots of possiblilities, but a few things to look at:
- Did you do a buildkernel or a make cleandepend && make depend && make?
- Look at the timestamp on mps.o in your kernel build directory
- touch mps.c, rebuild your kernel, and look to see that it got built
- 'grep io_cmds_active *' in sys/dev/mps, and make sure the patch got
applied.
- make sure you installed the kernel in the right place
- make sure you booted the correct kernel (look at uname -a to see when it
was built)
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> Hmm. Lots of possiblilities, but a few things to look at:
>
> - Did you do a buildkernel or a make cleandepend && make depend &&
> make?
> - Look at the timestamp on mps.o in your kernel build directory
> - touch mps.c, rebuild your kernel, and look to see that it got built
> - 'grep io_cmds_active *' in sys/dev/mps, and make sure the patch got
> applied.
> - make sure you installed the kernel in the right place
> - make sure you booted the correct kernel (look at uname -a to see
> when it
> was built)
Okay, now I feel like an idiot; I haven't rebuilt any kernel at any
time, so the 1024->2048 change of 'MPS_CHAIN_FRAMES' probably had no
effect (literally).
Now, I've run into another problem;
[root@filserver /usr/src]# make buildworld && make buildkernel
[...]
/usr/src/sys/dev/ath/if_ath.c:3895:5: error: "NOTYET" is not defined
*** Error code 1
Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
Alexander Motin
> On Fri, Feb 04, 2011, at 01:25:37AM GMT+01:00, Kenneth D. Merry wrote:
>> Hmm. Lots of possiblilities, but a few things to look at:
>>
>> - Did you do a buildkernel or a make cleandepend && make depend && make?
>> - Look at the timestamp on mps.o in your kernel build directory
>> - touch mps.c, rebuild your kernel, and look to see that it got built
>> - 'grep io_cmds_active *' in sys/dev/mps, and make sure the patch got
>> applied.
>> - make sure you installed the kernel in the right place
>> - make sure you booted the correct kernel (look at uname -a to see
>> when it
>> was built)
>
> time, so the 1024->2048 change of 'MPS_CHAIN_FRAMES' probably had no
> effect (literally).
>
> Now, I've run into another problem;
>
> [root@filserver /usr/src]# make buildworld && make buildkernel
> [...]
> /usr/src/sys/dev/ath/if_ath.c:3895:5: error: "NOTYET" is not defined
> *** Error code 1
>
> Stop in /usr/obj/usr/src/sys/GENERIC.
> *** Error code 1
>
> Stop in /usr/src.
> *** Error code 1
>
> Stop in /usr/src.
It was already fixed. Update your sources and retry.
--
Alexander Motin
Joachim Tingvold
> It was already fixed. Update your sources and retry.
Ahhh, thanks for the heads up. I updated it when I sent the email, and
the fix came just an hour later.
Joachim Tingvold
> Try running this, and then do 'sysctl hw.mps' and let's see what
> your low
> water mark is for free chain elements. We'll also want to make sure
> your
> chain_free value is about equal to MPS_CHAIN_FRAMES when the system is
> idle. On my system with a LSI 9201-16i controller, I see:
While idle;
[jocke@filserver ~]$ sysctl hw.mps.0
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
hw.mps.0.io_cmds_active: 0
hw.mps.0.io_cmds_highwater: 53
hw.mps.0.chain_free: 2048
hw.mps.0.chain_free_lowwater: 2029
hw.mps.0.chain_alloc_fail: 0
After I copied a 22G file from 'storage' to 'zroot';
[jocke@filserver ~]$ sysctl hw.mps.0
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
hw.mps.0.io_cmds_active: 0
hw.mps.0.io_cmds_highwater: 99
hw.mps.0.chain_free: 2048
hw.mps.0.chain_free_lowwater: 2029
hw.mps.0.chain_alloc_fail: 0
After I copied a 22G file from 'zroot' to 'storage';
[jocke@filserver ~]$ sysctl hw.mps.0
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
hw.mps.0.io_cmds_active: 0
hw.mps.0.io_cmds_highwater: 897
hw.mps.0.chain_free: 2048
hw.mps.0.chain_free_lowwater: 2029
hw.mps.0.chain_alloc_fail: 0
Kenneth D. Merry
Looks like there were a lot of commands active at once.
> hw.mps.0.chain_free: 2048
> hw.mps.0.chain_free_lowwater: 2029
> hw.mps.0.chain_alloc_fail: 0
But no more than 19 chain elements used at any one time.
Perhaps it could depend on memory fragmentation somewhat. Over time you
may see the low water mark go down a bit.
The good news is that it doesn't look like we have a leak.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> Perhaps it could depend on memory fragmentation somewhat. Over time
> you
> may see the low water mark go down a bit.
>
> The good news is that it doesn't look like we have a leak.
<http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_5.txt>
[jocke@filserver ~]$ sysctl hw.mps.0
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
hw.mps.0.io_cmds_active: 1
hw.mps.0.io_cmds_highwater: 959
hw.mps.0.chain_free: 2048
hw.mps.0.chain_free_lowwater: 1721
hw.mps.0.chain_alloc_fail: 0
This time I did a recursive copy of a folder with no large files at
all (it contained only small documents), from 'storage' to 'storage'.
However, it recovered, so the copy just continued where it left of --
which is a change from previous crashes.
Joachim Tingvold
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_5.txt>
Updated; it happened again just a few minutes after the first one.
Kenneth D. Merry
> On Fri, Feb 04, 2011, at 19:00:11PM GMT+01:00, Kenneth D. Merry wrote:
> >Perhaps it could depend on memory fragmentation somewhat. Over time
> >you
> >may see the low water mark go down a bit.
> >
> >The good news is that it doesn't look like we have a leak.
>
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_5.txt>
>
This particular error is interesting:
mps0: (0:40:0) terminated ioc 804b scsi 0 state c xfer 0
mps0: (0:40:0) terminated ioc 804b scsi 0 state c xfer 0
It means that the chip terminated the command for some reason. I have been
talking to LSI about it. I'm working on getting an analyzer trace when it
happens, so I cn send that to LSI.
What kind of expander do you have in your system? How many expanders do
you have? How many drives do you have? Can you send 'camcontrol devlist
-v' output?
> [jocke@filserver ~]$ sysctl hw.mps.0
> hw.mps.0.debug_level: 0
> hw.mps.0.allow_multiple_tm_cmds: 0
> hw.mps.0.io_cmds_active: 1
> hw.mps.0.io_cmds_highwater: 959
> hw.mps.0.chain_free: 2048
> hw.mps.0.chain_free_lowwater: 1721
> hw.mps.0.chain_alloc_fail: 0
>
> This time I did a recursive copy of a folder with no large files at
> all (it contained only small documents), from 'storage' to 'storage'.
>
> However, it recovered, so the copy just continued where it left of --
> which is a change from previous crashes.
Yes, it looks like we're not running into the out of chain problem.
The timeouts could be due to all sorts of problems. The IOC terminated
errors I'm still not sure about. I need to get a trace and send that along
with a diagnostic ring buffer dump from the card to LSI to get some answers
about what is going on.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> What kind of expander do you have in your system? How many
> expanders do
> you have? How many drives do you have?
I have one HP SAS expander, running firmware 2.06. It's connected to
the 9211-8i with two mini-SAS cables. I currently have 17 disks
connected.
> Can you send 'camcontrol devlist -v' output?
<http://home.komsys.org/~jocke/dmesg_and_camcontrol-devlist.txt>
ada0 and cd0 is connected to the S-ATA controller on the motherboard.
> The timeouts could be due to all sorts of problems. The IOC
> terminated
> errors I'm still not sure about. I need to get a trace and send
> that along
> with a diagnostic ring buffer dump from the card to LSI to get some
> answers
> about what is going on.
Let me know if/when I can help. (-:
Joachim Tingvold
> I just committed a fix to -current for the out of chain buffers
> issue. If
> you've got time, pull that fix in and see whether it works for you.
[jocke@filserver /storage/tmp]$ i=0;while [ $i -le 10 ]; do dd if=/
dev/zero of=foo bs=1M count=204800; let "i += 1"; done
I let it run for a while, and checked back later.
<http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_6.txt>
hw.mps.0.debug_level: 0
hw.mps.0.allow_multiple_tm_cmds: 0
hw.mps.0.io_cmds_active: 0
hw.mps.0.io_cmds_highwater: 970
hw.mps.0.chain_free: 1024
hw.mps.0.chain_free_lowwater: 0
hw.mps.0.chain_alloc_fail: 786
As you can see, I didn't do anything about the 'MPS_CHAIN_FRAMES'-
value after I pulled the latest version. I could change it, but I
guess this only would get rid of the 'out of chain'-messages, and not
really solve the underlying problem.
Kenneth D. Merry
> On Fri, Feb 18, 2011, at 18:16:19PM GMT+01:00, Kenneth D. Merry wrote:
> >I just committed a fix to -current for the out of chain buffers
> >issue. If
> >you've got time, pull that fix in and see whether it works for you.
>
> [jocke@filserver /storage/tmp]$ i=0;while [ $i -le 10 ]; do dd if=/
> dev/zero of=foo bs=1M count=204800; let "i += 1"; done
>
> I let it run for a while, and checked back later.
>
> <http://home.komsys.org/~jocke/dmesg_mps0_freebsd-scsi_6.txt>
>
> hw.mps.0.allow_multiple_tm_cmds: 0
> hw.mps.0.io_cmds_active: 0
> hw.mps.0.chain_free: 1024
> hw.mps.0.chain_free_lowwater: 0
> hw.mps.0.chain_alloc_fail: 786
>
> As you can see, I didn't do anything about the 'MPS_CHAIN_FRAMES'-
> value after I pulled the latest version. I could change it, but I
> guess this only would get rid of the 'out of chain'-messages, and not
> really solve the underlying problem.
Okay, good. It looks like it is running as designed.
I'll probably just double the number of chain buffers and disable the
messages.
>From previous experiments, the problem is much less likely to occur when
you have 2048 chain buffers, correct?
What filesystem are you using by the way?
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> Okay, good. It looks like it is running as designed.
It is? It still terminating the commands, which I guess it shouldn't?
mps0: (0:40:0) terminated ioc 804b scsi 0 state c xfer 0
> I'll probably just double the number of chain buffers and disable the
> messages.
>
> From previous experiments, the problem is much less likely to occur
> when
> you have 2048 chain buffers, correct?
It just doesn't display the 'out of chain'-errors, that's all I think.
> What filesystem are you using by the way?
ZFS.
Kenneth D. Merry
> On Mon, Feb 21, 2011, at 16:50:41PM GMT+01:00, Kenneth D. Merry wrote:
> >Okay, good. It looks like it is running as designed.
>
>
Sorry, I missed that, I was just looking at the first part.
I'm still waiting for LSI to look at the SAS analyzer trace I sent them for
the "IOC terminated" bug.
It appears to be (at least on my hardware) a backend issue of some sort,
and probably not anything we can fix in the driver.
Since you've got an HP branded expander, that makes it a little more
difficult to determine whether it's an LSI, Maxim, or some other expander.
Can you try the following on your system? You'll need the sg3_utils port:
sg_inq -i ses0
(I need to update camcontrol to parse page 0x83 output.)
This is example output from a Supermicro chassis with an LSI 3Gb expander:
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 12
transport: Serial Attached SCSI (SAS)
designator_type: NAA, code_set: Binary
associated with the target port
NAA 5, IEEE Company_id: 0x3048
Vendor Specific Identifier: 0x45157f
[0x500304800045157f]
Maxim expanders seem to report LUN descriptors in VPD page 0x83 instead of
target port descriptors. We might get a slight clue from the output, but
it's hard to say for certain since HP could have customized the page 0x83
values in the expander firmware.
> >I'll probably just double the number of chain buffers and disable the
> >messages.
> >
> >From previous experiments, the problem is much less likely to occur
> >when
> >you have 2048 chain buffers, correct?
>
> It just doesn't display the 'out of chain'-errors, that's all I think.
Well, if you don't see the 'out of chain' errors with 2048 chain buffers,
that means the condition isn't happening.
The cost of going from 1024 to 2048 is only 32K of extra memory, which is
not a big deal, so I think I'll go ahead and bump the limit up and remove
the printfs. We've now proven the recovery strategy, so it'll just slow
things down slightly if anyone runs into that issue again.
> >What filesystem are you using by the way?
>
> ZFS.
Interesting. I haven't been able to run out of chain elements with ZFS,
but I can use quite a few with UFS. I had to artificially limit the number
of chain elements to test the change.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
>>> Okay, good. It looks like it is running as designed.
>>
No worries. (-:
> I'm still waiting for LSI to look at the SAS analyzer trace I sent
> them for
> the "IOC terminated" bug.
>
> It appears to be (at least on my hardware) a backend issue of some
> sort,
> and probably not anything we can fix in the driver.
I see. Good to know that you're able to reproduce it, since I with
good possibility can rule out that it's a hardware-issue on my
controller.
> Since you've got an HP branded expander, that makes it a little more
> difficult to determine whether it's an LSI, Maxim, or some other
> expander.
> Can you try the following on your system? You'll need the sg3_utils
> port:
>
> sg_inq -i ses0
>
> (I need to update camcontrol to parse page 0x83 output.)
>
> [...]
>
> Maxim expanders seem to report LUN descriptors in VPD page 0x83
> instead of
> target port descriptors. We might get a slight clue from the
> output, but
> it's hard to say for certain since HP could have customized the page
> 0x83
> values in the expander firmware.
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 12
transport: Serial Attached SCSI (SAS)
designator_type: NAA, code_set: Binary
associated with the target port
NAA 5, IEEE Company_id: 0x1438
Vendor Specific Identifier: 0x101a2865
[0x50014380101a2865]
Designation descriptor number 2, descriptor length: 8
transport: Serial Attached SCSI (SAS)
designator_type: Relative target port, code_set: Binary
associated with the target port
Relative target port: 0x1
>> It just doesn't display the 'out of chain'-errors, that's all I
>> think.
>
> Well, if you don't see the 'out of chain' errors with 2048 chain
> buffers,
> that means the condition isn't happening.
>
> The cost of going from 1024 to 2048 is only 32K of extra memory,
> which is
> not a big deal, so I think I'll go ahead and bump the limit up and
> remove
> the printfs. We've now proven the recovery strategy, so it'll just
> slow
> things down slightly if anyone runs into that issue again.
Good. It has such a small impact, yes, so it shouldn't trouble anyone.
>>> What filesystem are you using by the way?
>> ZFS.
> Interesting. I haven't been able to run out of chain elements with
> ZFS,
> but I can use quite a few with UFS. I had to artificially limit the
> number
> of chain elements to test the change.
Maybe it's because the amount of disks in the same pool that I have?
Or that I have two un-even raidz2 vdev's in the same pool? The latter
has to be forced when adding it to the pool, so I guess it's not an
"ideal" solution... (but "everyone" does it, it seems).
Kenneth D. Merry
> On Mon, Feb 21, 2011, at 22:45:44PM GMT+01:00, Kenneth D. Merry wrote:
> >>>Okay, good. It looks like it is running as designed.
> >>It is? It still terminating the commands, which I guess it shouldn't?
> >>
> >> mps0: (0:40:0) terminated ioc 804b scsi 0 state c xfer 0
> >Sorry, I missed that, I was just looking at the first part.
>
> No worries. (-:
>
> >I'm still waiting for LSI to look at the SAS analyzer trace I sent
> >them for
> >the "IOC terminated" bug.
> >
> >It appears to be (at least on my hardware) a backend issue of some
> >sort,
> >and probably not anything we can fix in the driver.
>
> I see. Good to know that you're able to reproduce it, since I with
> good possibility can rule out that it's a hardware-issue on my
> controller.
And if you're only seeing it occasionally, it's probably not a big worry.
I have the driver set up to retry IOC terminated errors without
decrementing the retry count, so you won't run into any filesystem or other
errors because of it.
The only issue is that if you get into a situation where you're getting
those errors continuously, you'll wind up in an endless retry loop. (I've
seen that on my test system, but 60 drives behind 8 expanders in multiple
levels is a bit of an exceptional case.)
Is this a 6Gb or a 3Gb expander? Since it has a target port descriptor, it
might be OEM LSI expander, but who knows.
> >>It just doesn't display the 'out of chain'-errors, that's all I
> >>think.
> >
> >Well, if you don't see the 'out of chain' errors with 2048 chain
> >buffers,
> >that means the condition isn't happening.
> >
> >The cost of going from 1024 to 2048 is only 32K of extra memory,
> >which is
> >not a big deal, so I think I'll go ahead and bump the limit up and
> >remove
> >the printfs. We've now proven the recovery strategy, so it'll just
> >slow
> >things down slightly if anyone runs into that issue again.
>
> Good. It has such a small impact, yes, so it shouldn't trouble anyone.
I just checked the change into -current, I'll merge it to -stable next
week.
> >>>What filesystem are you using by the way?
> >>ZFS.
> >Interesting. I haven't been able to run out of chain elements with
> >ZFS,
> >but I can use quite a few with UFS. I had to artificially limit the
> >number
> >of chain elements to test the change.
>
> Maybe it's because the amount of disks in the same pool that I have?
> Or that I have two un-even raidz2 vdev's in the same pool? The latter
> has to be forced when adding it to the pool, so I guess it's not an
> "ideal" solution... (but "everyone" does it, it seems).
I don't think that would affect the out of chain problem. It has a lot
more to do with memory fragmentation and therefore how long the
scatter/gather lists are that get generated by busdma.
Ken
--
Kenneth Merry
k...@FreeBSD.ORG
Joachim Tingvold
> I just checked the change into -current, I'll merge it to -stable
> next week.
I'm back! Missed me? :-D
After running fine for a while, I decided to do some more testing.
Usual 'dd' in a while-loop over the night, and woke up to this;
###
mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 0
mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
mps0: mpssas_remove_complete on target 0x0027, IOCStatus= 0x0
(da7:mps0:0:39:0): lost device
(da7:mps0:0:39:0): Invalidating pack
(da7:mps0:0:39:0): Invalidating pack
(da7:mps0:0:39:0): Invalidating pack
(da7:mps0:0:39:0): Invalidating pack
(da7:mps0:0:39:0): Synchronize cache failed, status == 0xa, scsi
status == 0x0
(da7:mps0:0:39:0): removing device entry
da7 at mps0 bus 0 scbus0 target 39 lun 0
da7: <ATA WDC WD10EACS-00Z 1B01> Fixed Direct Access SCSI-5 device
da7: 300.000MB/s transfers
da7: Command Queueing enabled
da7: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
###
Now, the disk was present at the time I checked, as camcontrol confirms;
[root@filserver /storage/tmp]# camcontrol devlist|grep da7
<ATA WDC WD10EACS-00Z 1B01> at scbus0 target 39 lun 0 (pass8,da7)
However, the disk was marked as "REMOVED" by 'zpool status';
###
[jocke@filserver /storage/tmp]$ zpool status
pool: storage
state: DEGRADED
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
da15 ONLINE 0 0 0
da16 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
da7 REMOVED 0 0 0
da12 ONLINE 0 0 0
da13 ONLINE 0 0 0
spares
da14 AVAIL
###
A quick 'zpool online storage da7' works fine, as suspected, and pool
is resilvering at the moment.
I find it a bit worrisome that a disk was removed like that. It
_could_ be that the disk isn't completely good, however, due to my
previous experiences with mps, I suspect the disk is fine (smartctl-
readouts on the disk seems to be good as well).
--
Joachim
Kenneth D. Merry
> On Fri, Feb 25, 2011, at 19:33:51PM GMT+01:00, Kenneth D. Merry wrote:
> >I just checked the change into -current, I'll merge it to -stable
> >next week.
>
>
> After running fine for a while, I decided to do some more testing.
> Usual 'dd' in a while-loop over the night, and woke up to this;
>
> ###
> mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
> mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
> mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
> mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 65536
> mps0: (0:39:0) terminated ioc 804b scsi 0 state c xfer 0
> mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
> mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
> mps0: (0:39:0) terminated ioc 804b scsi 0 state 0 xfer 0
> mps0: mpssas_remove_complete on target 0x0027, IOCStatus= 0x0
> (da7:mps0:0:39:0): lost device
> (da7:mps0:0:39:0): Invalidating pack
> (da7:mps0:0:39:0): Invalidating pack
> (da7:mps0:0:39:0): Invalidating pack
> (da7:mps0:0:39:0): Invalidating pack
> (da7:mps0:0:39:0): Synchronize cache failed, status == 0xa, scsi
> status == 0x0
> (da7:mps0:0:39:0): removing device entry
> da7 at mps0 bus 0 scbus0 target 39 lun 0
> da7: <ATA WDC WD10EACS-00Z 1B01> Fixed Direct Access SCSI-5 device
> da7: 300.000MB/s transfers
> da7: Command Queueing enabled
> da7: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
> ###
>
> Now, the disk was present at the time I checked, as camcontrol confirms;
>
> [root@filserver /storage/tmp]# camcontrol devlist|grep da7
> <ATA WDC WD10EACS-00Z 1B01> at scbus0 target 39 lun 0 (pass8,da7)
Yep, this looks like what I've seen with mps controllers talking to SATA
drives through an expander under high load.
I know I've asked this before, but what brand of expander do you have, and
is it 3Gb or 6Gb? It looks like the drive is probing at 3Gb in any case.
It looks like the drive went away and came back.
The disk is probably fine. That error tends to happen when you have a lot
of contention under high load. I wish I knew why. It is something that
LSI should fix, I was talking to them for a while trying to get an answer
on it, but got nowhere.
With some of the ZFS improvements that Justin is working on in -current,
I think the drive would have probably been automatically put back into
the pool when it came back.