Weird I/O block

[Anders Wallén]

Greetings, All!

I migrated over to O3X since MacZFS didn't work with Yosemite.

Now I have encountered a problem. I have already posted about it over at the O3X forum, but no-one there seems to know what to do and/or be interested.

The array "array_2" no longer shows its content, wether in Finder or in Terminal.

I tried the zpool scrub command, to see if that would resolve the problem. I got this return:

zpool scrub array_2
cannot scrub array_2: pool I/O is currently suspended

Some more info:

sudo zpool status -v array_2
pool: array_2
state: ONLINE
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://zfsonlinux.org/msg/ZFS-8000-HC
scan: resilvered 12,5K in 0h0m with 0 errors on Mon Nov 3 18:00:02 2014
config:

NAME STATE READ WRITE CKSUM
array_2 ONLINE 2 7 0
raidz2-0 ONLINE 6 30 0
disk11s2 ONLINE 3 26 0
disk12s2 ONLINE 4 22 0
disk13s2 ONLINE 4 22 0
disk14s2 ONLINE 4 22 0
disk15s2 ONLINE 3 22 0
disk16s2 ONLINE 3 28 0
disk17s2 ONLINE 3 26 0
disk18s2 ONLINE 3 24 0

errors: List of errors unavailable (insufficient privileges)

Even with "sudo" I am not allowed to view the error list. Hmmm...

after following the supplied link, I ran "zpool clear":

sudo zpool clear array_2
cannot clear errors for array_2: I/O error

So... It didn't work. What do I do now? Any suggestions are greatly appreciated.

[BelecMartin]

Cable connection failure. 

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

--

---
You received this message because you are subscribed to the Google Groups "zfs-macos" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-macos+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[BelecMartin]

On another note, never scrub if your not sure what's wrong. 

A little more detail. I have recently seen what your encountering several times.  It was at least one faulty cable in each case once it was two. Once fixed everything was perfect. 

Yes the -v fails, been that way for awhile now concerning permissions. 

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

[BelecMartin]

Anders, got it resolved?

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

On Nov 11, 2014, at 12:23 PM, Anders Wallén <anders...@mac.com> wrote:

--

[Anders Wallén]

Sorry about the delay.

Answer: No.

I have checked the cable at both ends, and found no errors.

Here is a transcript of today's exercises

anderswallen:~ anderswa$ sudo zpool list

NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

array_2  29,1T  14,8T  14,3T    50%  1.00x  UNAVAIL  -

anderswallen:~ anderswa$ sudo zpool status -v array_2

  pool: array_2

 state: ONLINE

status: One or more devices are faulted in response to IO failures.

action: Make sure the affected devices are connected, then run 'zpool clear'.

   see: http://zfsonlinux.org/msg/ZFS-8000-HC

  scan: resilvered 12,5K in 0h0m with 0 errors on Mon Nov  3 18:00:02 2014

config:

NAME          STATE     READ WRITE CKSUM

array_2       ONLINE       1     7     0

  raidz2-0    ONLINE       4    24     0

    disk5s2   ONLINE       4    12     0

    disk6s2   ONLINE       3    16     0

    disk7s2   ONLINE       3    18     0

    disk8s2   ONLINE       4    22     0

    disk9s2   ONLINE       4    22     0

    disk10s2  ONLINE       4    20     0

    disk11s2  ONLINE       4    16     0

    disk12s2  ONLINE       4    18     0

errors: List of errors unavailable (insufficient privileges)

anderswallen:~ anderswa$ sudo zpool clear -F array_2

[Anders Wallén]

Addendum:

To add insult to injury, the computer now refuses to accept both my older arrays (one ZFS Raid 6 and one HFS+ Raid 5).

The ZFS one doesn't even show up on "sudo zpool list", and the HFS+ one does not show up in App+le Disc Tools.

[BelecMartin]

Well the fact you are getting errors on every drive says you have either a cable(s) failed/failing or perhaps a controller that has gone south. If it's one or two drives then they would be in the possibles list but that is not what your showing. 

Visually checking a cable is only part of the process. Swapping out one by one and testing to see if errors clear up is the only way. If even one cable is faulty it can affect all the other data from the drives to the controller. 

It is a pain I know, however the only way to find your root problem. Once you find the culprit(s) everything just works again. 

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

[BelecMartin]

Yeah, expected this sooner, bad cable. Just went through almost exactly the same sequence for a client. It was a SATA cable, but still getting issues after replacing it and one drive that was seemingly bad. And then all drives disappeared. Replaced the ESATA cable but is was also crap. Replaced again with a quality known working cable from another system an voila! Everything perfect. Began scrub, all came out great. 

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

--

[Bjoern Kahl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Anders,

Am 17.11.14 um 17:28 schrieb Anders Wallén:

as Jason already said, it *looks* like a hardware fault. But if you
suspect a software problem, you can do a partial cross-check by asking
ZDB to scan your drives. Running ZDB in part exercises different code
paths in the ZFS system, so may behave differently than the kernel
level driver if you really have a software problem.

(I wrote "partial", because ZDB and the kernel level driver share
significant code parts (all the ZFS logic), but with ZDB being
userland, it hooks into the disk driver logic at a different point
than the kernel level driver.)

If I remember correctly, there was also a way to use ZDB to get to the
error log, though not sure at the moment and not in reach of a test
machine right now.


Another thought:

Have you rebooted recently? I suspect "yes", since you wrote you
inspected the cables which usually requires a power down.

In case not, reboot the box. It will probably render the pool offline
/ faulted, but in rare circumstances it could also clear the problem,
if caused by the kernel level driver getting confused. Should not
happen, but you never know ...


Best regards

Björn

> Den onsdagen den 12:e november 2014 kl. 19:14:29 UTC+1 skrev
> jasonbelec:
>>
>> Anders, got it resolved?
>>
>> Jason Belec Sent from my "It's an iPod, a Phone, and an Internet
>> Device..."
>>
>> On Nov 11, 2014, at 12:23 PM, Anders Wallén <anders...@mac.com

| Bjoern Kahl +++ Siegburg +++ Germany |
| "googlelogin@-my-domain-" +++ www.bjoern-kahl.de |
| Languages: German, English, Ancient Latin (a bit :-)) |
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQCVAgUBVGozhFsDv2ib9OLFAQJvLgP/cH660XLD3VXHvsAmEGVUP1vZfdVdH7kT
qL+ZoBDnD4MjjNb0EaIMQZ/0UZqm33KItlyKt3Kj1jWlcXbUSZO755I4brlDa1ln
+VmDn/REt2Q9//bLIn8VBHwAl2rgP9pppbWeITAd0gStr0GLGd5cm1JftMJ+69jN
7XlRZl55C8o=
=a7K1
-----END PGP SIGNATURE-----

[Anders Wallén]

New day, new cables.

Now the computer recognizes the two ZFS arrays, but once again I cannot read from or write to them.

Old array:

anderswallen:~ anderswa$ sudo zpool status -v

  pool: array

 state: ONLINE

status: One or more devices are faulted in response to IO failures.

action: Make sure the affected devices are connected, then run 'zpool clear'.

   see: http://zfsonlinux.org/msg/ZFS-8000-HC

  scan: none requested

config:

NAME          STATE     READ WRITE CKSUM

array         ONLINE       1     6     0

  raidz2-0    ONLINE       4    18     0

    disk10s2  ONLINE       1    14     0

    disk11s2  ONLINE       1    16     0

    disk12s2  ONLINE       1    16     0

    disk13s2  ONLINE       0    18     0

    disk14s2  ONLINE       0    14     0

    disk15s2  ONLINE       0    14     0

    disk16s2  ONLINE       0    14     0

errors: List of errors unavailable (insufficient privileges)

anderswallen:~ anderswa$ sudo zpool clear -F array

cannot clear errors for array: I/O error

New array:

  pool: array_2
 state: UNAVAIL

status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: resilvered 12,5K in 0h0m with 0 errors on Mon Nov  3 18:00:02 2014
config:
NAME         STATE     READ WRITE CKSUM

array_2      UNAVAIL      0     0     0  insufficient replicas
  raidz2-0   UNAVAIL      0     0     0  insufficient replicas
    disk2s2  FAULTED     30     0     0  too many errors
    disk3s2  FAULTED     27     0     0  too many errors
    disk4s2  FAULTED     24     0     0  too many errors
    disk5s2  FAULTED     21     0     0  too many errors
    disk6s2  FAULTED     18     0     0  too many errors
    disk7s2  FAULTED     15     0     0  too many errors
    disk8s2  FAULTED     12     0     0  too many errors
    disk9s2  FAULTED      9     0     0  too many errors

errors: List of errors unavailable (insufficient privileges)

anderswallen:~ anderswa$ sudo zpool clear -F array_2

cannot clear errors for array_2: I/O error

 Where do we go from here?

[BelecMartin]

Cool, getting somewhere. Can you give a quick rundown how things are connected in your system and what cables are between various elements to provide a better picture. What kind of controller for the drives, etc...?

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

--

[Anders Wallén]

The arrays are housed in cabinets of this type (one each).

Each cabinet is connected directly to a dedicated USB 3-port on the Mac Mini. The Mini itself is a quad core i7, running the latest version of Mac OS X.

I'm surprised that the cables failed in such rapid succession. They came with the cabinets, which are 0, 1 and 2 years old respectively.

Best regards,

Anders

[Daniel Becker]

Based on what you posted earlier, you’re still seeing lots of read and write errors after replacing the cables, so I highly doubt those were really the root of your problem.

[Daniel Becker]

Can you try connecting these to a different machine, and/or via either eSATA or USB 2.0?

[BelecMartin]

Yikes, did you say USB???? Oh I'm going to go back into my cave. 

Jason Belec

Sent from my "It's an iPod, a Phone, and an Internet Device..."

--

[Daniel Becker]

Really, though, it looks like there’s a bunch of reviews online (Amazon, Newegg, …) that mention stability issues with these boxes, drives dropping out under load and USB 3 flakiness. If these are indeed accurate (I don’t have any personal experience with them), I’m not surprised you’d run into issues with ZFS.

[Anders Wallén]

Hearing about others having problems with these cabinets makes me concerned too.

Oh, well! I connected the cabinets to a different Mac (an older Mini with USB 2 instead of 3), and the ZFS arrays now seem to be working. *holds thumbs*

I am now going to set up a copy-job to see if that works out.

Some output from the Terminal:

Anders-MC:~ anderswa$ sudo zpool list

no pools available

Anders-MC:~ anderswa$ sudo zpool import

   pool: array_2

     id: 7027660346350053619

  state: ONLINE

 status: The pool was last accessed by another system.

 action: The pool can be imported using its name or numeric identifier and

the '-f' flag.

   see: http://zfsonlinux.org/msg/ZFS-8000-EY

 config:

array_2       ONLINE

  raidz2-0    ONLINE

    disk12s2  ONLINE

    disk13s2  ONLINE

    disk14s2  ONLINE

    disk15s2  ONLINE

    disk16s2  ONLINE

    disk17s2  ONLINE

    disk18s2  ONLINE

    disk19s2  ONLINE

   pool: array

     id: 14634014075632557217

  state: ONLINE

 status: The pool was last accessed by another system.

 action: The pool can be imported using its name or numeric identifier and

the '-f' flag.

   see: http://zfsonlinux.org/msg/ZFS-8000-EY

 config:

array         ONLINE

  raidz2-0    ONLINE

    disk4s2   ONLINE

    disk10s2  ONLINE

    disk5s2   ONLINE

    disk6s2   ONLINE

    disk7s2   ONLINE

    disk8s2   ONLINE

    disk9s2   ONLINE

Anders-MC:~ anderswa$ sudo zpool import -f array

Anders-MC:~ anderswa$ sudo zpool import -f array_2

Anders-MC:~ anderswa$ sudo zpool list

NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

array    19,1T  13,0T  6,05T    68%  1.00x  ONLINE  -

array_2  29,1T  14,8T  14,3T    50%  1.00x  ONLINE  -

Non-ZFS stuff:

The older HFS+ array does not seem to be affected by the change, but I think that that one actually has a problem with one of its dics - it sounds that way when booted up. Still, I am confounded as to why it doesn't show up on Disk Tools. Also, after the initial noise, the cabinet indicates constant, never-ending access to disk # 7 in that array. Hm...

Does anyone know of some good disc tool to examine and (hopefully) repair an HFS+ raid?