Importing degraded pool...

[Charles Huber]

Background: I have four SATA drives in a raidz (no spares). One of
them started throwing up SMART read errors. I zpool export'd the
pool, shut down, and removed the drive for RMA prep.

Problem: Upon rebooting to access the pool read-only, zpool import
comes up with:
pool: tank
id: 10858732204425674861
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.
The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:

tank
DEGRADED
raidz1
DEGRADED
sdc1
ONLINE
disk/by-id/ata-WDC_WD10EACS-00ZJB0_WD-WCASJ1128671-part1
UNAVAIL cannot open
sdd1
ONLINE
sde1
ONLINE

zpool import tank returns:
cannot import 'tank': one or more devices is currently unavailable

8000-2Q seems to indicate that what I did should work. Re-adding the
drive (it's in that quasi-failed state where light read/write in some
places works) did not help importing, although the pool state was
upgraded to ONLINE. What can I do to get this pool re-imported?

Thanks!
-Charles

[Ricardo M. Correia]

Hi Charles,

Your scenario seems to work here.

I think your devices got renamed and zfs-fuse is getting confused
because of that.

In particular, my guess is that the
device /dev/disk/by-id/ata-WDC_WD10EACS-00ZJB0_WD-WCASJ1128671-part1 is
in fact refers to the same partition as one of the other devices
(/dev/sdc1, /dev/sdd1 or /dev/sde1), and therefore zfs-fuse may be
trying to open the same partition twice.

Anyway, can you try doing "zpool import -d /dev/disk/by-id tank"?
If my guess is correct, that should work fine.

Failing that, can you give me the output of:

1) strace zpool import tank -d /dev/disk/by-id 2>&1 | grep open
2) strace zpool import tank -d /dev 2>&1 | grep open
3) strace zpool import tank 2>&1 | grep open

Thanks,
Ricardo

[Ricardo M. Correia]

On Sex, 2008-10-03 at 14:14 +0100, Ricardo M. Correia wrote:
> Failing that, can you give me the output of:
>
> 1) strace zpool import tank -d /dev/disk/by-id 2>&1 | grep open
> 2) strace zpool import tank -d /dev 2>&1 | grep open
> 3) strace zpool import tank 2>&1 | grep open

In fact, forget the above commands - this is going to be a bit more
complicated.

If the "zpool import -d /dev/disk/by-id tank" command didn't work,
you're going to need to do this as root:

$ cd /proc/`pgrep zfs-fuse`/task
$ for i in *; do sh -c "strace -p $i 2> /tmp/zfs_debug.$i &"; done

$ zpool import -d /dev/disk/by-id tank
$ zpool import -d /dev tank
$ zpool import tank

$ killall strace
$ (send me the output of this:) grep open /tmp/zfs_debug.*
$ rm /tmp/zfs_debug.*

Thanks,
Ricardo

[GenPFault]

Thanks for the quick reply! Awesome proc/strace one-liners by the
way. I have attached the grepped strace output.

I'm seeing some EBUSY's on sdc, but only when it's opening the device,
not the partition. Seems to love urandom too.

-Charles

[Ricardo M. Correia]

Hi Charles,

On Sex, 2008-10-03 at 10:25 -0500, GenPFault wrote:
> I'm seeing some EBUSY's on sdc, but only when it's opening the device,
> not the partition. Seems to love urandom too.

The output looks normal, so it seems the problem is not when opening the
devices. It seems that another system call is failing.

Can you send me all the /tmp/zfs_debug.* files so that I can analyze
them?

Thanks,
Ricardo

[Ricardo M. Correia]

On Sex, 2008-10-03 at 16:45 +0100, Ricardo M. Correia wrote:
> Can you send me all the /tmp/zfs_debug.* files so that I can analyze
> them?

Also, can you do the following:

$ cd $ZFS_FUSE_SRC/src/cmd/zdb
$ scons -u debug=2
$ (as root) ./zdb -p /dev/disk/by-id -e tank debug=on

The output of the last command will greatly help to pinpoint the
problem.

Thanks,
Ricardo

[GenPFault]

Sure thing. Attached.

The first zdb invocation complained about multiple matching pools, so
I tried the guid(?) which generated more output.

I don't think I mentioned it before but I was using either 0.5.0 or
trunk before the zdb test; now I'm definitely using trunk.

-Charles

[Ricardo M. Correia]

On Sex, 2008-10-03 at 11:24 -0500, GenPFault wrote:
> The first zdb invocation complained about multiple matching pools, so
> I tried the guid(?) which generated more output.

Thanks, that one was much more helpful.

> I don't think I mentioned it before but I was using either 0.5.0 or
> trunk before the zdb test; now I'm definitely using trunk.

You've ran into a strange problem. It seems ZFS is getting an error
about a missing device when reading a log record.

AFAICT, this may be happening because you have a missing device and ZFS
is not being able to reconstruct the block from parity.
This could also happen if you had a slog device in your pool and now
it's missing (although this doesn't seem likely from what I read in your
original post).

Since ZFS is unable to replay the log, it refuses to import the pool.

Unfortunately it's not very easy to determine if this is a bug or not
without a more detailed investigation.

Anyway, if you want to try to force ZFS into importing your pool, you
can try modifying the file $ZFS_FUSE_SRC/src/lib/libzpool/zil.c, inside
function zil_check_log_chain(), around line 568-570 you should see this
piece of code:

error = zil_read_log_block(zilog, &blk, &abuf);
if (error)
break;

You can change that piece of code into this:

(void) zil_read_log_block(zilog, &blk, &abuf);
error = 0;

You can then recompile and reinstall with "scons" and "scons install",
and you should be able to import your pool.

However, I recommend that you change that piece of code back as it were
and recompile/reinstall after importing your pool, because ignoring
errors when replaying the log is generally not a good idea.

HTH,
Ricardo

[Ricardo M. Correia]

Wait, I made a mistake.

You should change this:

On Sex, 2008-10-03 at 19:04 +0100, Ricardo M. Correia wrote:
> error = zil_read_log_block(zilog, &blk, &abuf);
> if (error)
> break;

Into this:

> error = zil_read_log_block(zilog, &blk, &abuf);

> if (error) {
> error = 0;
> break;
> }

(Otherwise you'll probably get another error).

Cheers,
Ricardo

[GenPFault]

The second snippet seems to have done the trick. Import worked
without complaint and a zfs mount -a -o ro seems to show all my
filesystems/data. Thank you very, very much for your help!

Also, while searching over lunch I came across this upstream bug:
http://bugs.opensolaris.org/view_bug.do?bug_id=6736213

The symptoms look similar to the problem I was experiencing, although
guid import did not work in my case.

-Charles

[Matt B]

Hi, I'm in a very similar situation to this. I'm currently using
0.5.0, and one disk in a four disk raidZ array failed. I exported it
and removed the disk, and upon rebooting, 'zpool import -a' and 'zpool
import filez' tell me:
cannot import 'filez': one or more devices is currently unavailable

zpool import gives:
pool: filez
id: 2058135307045458040

state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.
The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:

filez DEGRADED
raidz1 DEGRADED
sda1 ONLINE
sdc1 UNAVAIL cannot open
hdc1 ONLINE
hda1 ONLINE

I'm pretty sure this is the same situation, I've got a raidZ with one
disk missing, which should be importable, but zpool import refuses to.
I've tried importing from /dev/disk/by-id and i've tried the patch
posted but neither of these works.
Can someone give me some strace/zdb commands that could help diagnose
the problem?

Thanks
Matt

[Ricardo M. Correia]

Hi Matt,

On Ter, 2008-11-11 at 01:13 -0800, Matt B wrote:
> I'm pretty sure this is the same situation, I've got a raidZ with one
> disk missing, which should be importable, but zpool import refuses to.
> I've tried importing from /dev/disk/by-id and i've tried the patch
> posted but neither of these works.
> Can someone give me some strace/zdb commands that could help diagnose
> the problem?

Ok, I think this will help:

1) Compile zdb in debug mode:

$ cd $ZFS_FUSE_SRC/src/cmd/zdb
$ scons -u debug=2

2) Try to import the pool with zdb and with debug enabled:

$ (as root) ./zdb -p /dev/disk/by-id -e tank debug=on

Can you tell me the output of that?

Thanks,
Ricardo

[Matt B]

Output is:
kernel_init: physmem = 254909 pages (0.97GB)
cv_timedwait: thread -1245578320 is at cv_timedwait at 331.67 with
delta 1.00 secs
cv_timedwait: thread -1253971024 is at cv_timedwait at 331.68 with
delta 1.00 secs
hdr_recl: hdr_recl called
cv_timedwait: thread -1245578320 exited cv_timedwait at 331.99 (rem =
0.68)
cv_timedwait: thread -1245578320 is at cv_timedwait at 331.99 with
delta 1.00 secs
spa_load: spa_load(): error 22
zdb: can't open filez: Invalid argument



On Nov 12, 3:18 am, "Ricardo M. Correia" <Ricardo.M.Corr...@Sun.COM>
wrote:

[Ricardo M. Correia]

On Ter, 2008-11-11 at 11:30 -0800, Matt B wrote:
> Output is:
> kernel_init: physmem = 254909 pages (0.97GB)
> cv_timedwait: thread -1245578320 is at cv_timedwait at 331.67 with
> delta 1.00 secs
> cv_timedwait: thread -1253971024 is at cv_timedwait at 331.68 with
> delta 1.00 secs
> hdr_recl: hdr_recl called
> cv_timedwait: thread -1245578320 exited cv_timedwait at 331.99 (rem =
> 0.68)
> cv_timedwait: thread -1245578320 is at cv_timedwait at 331.99 with
> delta 1.00 secs
> spa_load: spa_load(): error 22
> zdb: can't open filez: Invalid argument

Thanks for that.
Unfortunately, it didn't help much :(

If you can do the following steps, it will help a lot:

$ cd $ZFS_FUSE/src/cmd/zdb
$ scons -u debug=3
$ touch TRACE
$ mkdir trace
$ ./zdb -p /dev/disk/by-id -e <pool-name>
$ ./trace-parse.py ./zdb
$ tar -jcvf trace-data.tar.bz2 trace/
$ rm -rf TRACE trace

Of course, I would need you to send me the trace-data.tar.bz2 file for
analysis..

Thanks,
Ricardo

[Matt B]

Never mind, I downloaded the latest version of Nevada and booted up
from that DVD. The copy of zpool on there could import the array with
no problems, and now back in Linux I can import the array fine. Maybe
this is an upstream bug fixed by b101?
Thanks so much for your help anyway.

Matt

On Nov 12, 6:41 am, "Ricardo M. Correia" <Ricardo.M.Corr...@Sun.COM>
wrote: